Data transfer procedures

Data from LRS2 and VIRUS are transferred using the code pivot.

If you need to transfer guider or acam images over to the TACC try the following:

• login to ldas or vdas as yourself
• su hetdex (use the password known on the board)
• find the directory that you want to transfer, for example /hetdata/data/20160604/gc1 or /hetdata/data/20160603/acm
• scp with -r to maverick.tacc.utexas.edu:/work/03946/hetdex/maverick/YYYYMMDD/ as hetdex

  Example:
  scp -r  /hetdata/data/20160603/acm hetdex@maverick.tacc.utexas.edu:/work/03946/hetdex/maverick/20160603/
  

Troubleshooting

If the data won't appear in the /hetdata/data directory then probably the permissions are not set properly or the route from the instrument to the disk is broken. As hetdex on the instrument of interest, e.g. lrs2, do

ls -l /hetdata/data

to check the permissions it should look like

drwxrwxr-x   8 hetdex     hetdex     8 Jun 11 02:51 20160611/
drwxrwxr-x   7 hetdex     hetdex     7 Jun 12 03:06 20160612/
drwxrwxr-x   8 hetdex     hetdex     8 Jun 13 02:11 20160613/
drwxrwxr-x   8 hetdex     hetdex     8 Jun 14 21:59 20160614/
drwxrwxr-x   8 hetdex     hetdex     8 Jun 15 00:06 20160615/
drwxrwxr-x   8 hetdex     hetdex     8 Jun 16 23:50 20160616/

OR

drwxrwxr-x   8 guider     staff      8 Jun 17 01:03 20160617/
drwxrwxr-x   4 guider     staff      4 Jun 18 00:24 20160618/
drwxrwxr-x   8 guider     staff      8 Jun 19 05:56 20160619/
drwxrwxr-x   8 guider     staff      8 Jun 20 01:15 20160620/
drwxrwxr-x  10 guider     staff     10 Jun 21 09:17 20160621/

If you can not see /hetdata/data then the NSF mount is down so call IT support.

If it has the wrong permissions (missing g+w) or group (staff or hetdex) then reset this with

sudo chown hetdex.hetdex /hetdata/data/20160622
sudo chmod g+w /hetdata/data/20160622

now transfer the files over that are still waiting. For example:

/home/hetdex/code/het/trunk/camra/testing/proxy-pivot-lrs2-data.sh /mnt/camra_ramdisk/20160620/lrs2/lrs2*/exp0*/lrs2/*.fits

This moves the data to /home/data (for LRS2), /hetdata/data/ and TACC.

It is important to realize that you must be logged in as user=hetdex on the machine vdas (or lrs2). You are denied permission to ssh to hetdex. Hence _{after you hav logged in as yourself (i.e. ssh -X sco@vdas) you then must switch to the hetdex account (i.e. su - hetdex). Here is a complete example for performing both pivot steps for a set of VIRUS data:}

[sco@vdas virustmp]$ ls /var/log/vdas_pivot.log
[sco@vdas virustmp]$ tail /var/log/vdas_pivot.log
rsync: failed to set times on "/hetdata/data/.": Operation not permitted (1)
rsync: recv_generator: mkdir "/hetdata/data/20160624" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
2016-06-24T21:02:54.011 ERROR [tcs_proxy.cpp       :process             :  85] - Command exited 5888.

Note: the previous step created /opt/data/20160624

[sco@vdas ~]$ su - hetdex
Password:
[hetdex@vdas ~]$ /home/hetdex/code/het/trunk/camra/testing/proxy-pivot-virus-data.sh /mnt/camra_ramdisk/20160624/virus/virus*/exp0*/virus/*.fits
Authenticated with partial success.

The following have been reloaded with a version change:
  1) intel/14.0.1.106 => intel/15.0.3  2) mvapich2/2.0b => mvapich2/2.1

[hetdex@vdas ~]$ df
Now we verify that the ramdisk is cleared:
tmpfs                     8388608          0      8388608   0% /mnt/camra_ramdisk

After the manual proxy-pivot the same images were transferred to TACC. However, the more critical point is that the ramdisk was cleared. We have recently (mid- and late- Jun2016) had many problems with the ramdisk filling and causing a cascade of further problems. In the last step above I show the df command and output (for the ramdisk only) that verifies that the ramdisk is cleared.

Problems with PIVOT

If the problem is not with the directories then it could be that the PIVOT instance is not running. You can check to see if PIVOT is running with a chksys command on the appropriate machine, ie. lrs2 or vdas. Before stopping or restarting anything check the PIVOT log on the appropriate computer, ie. on the machine lrs2 it would be /var/log/ldas_pivot.log. Look for anything strange or at least document the last dozen or so lines in a PR.

You can restart the PIVOT routine with

sudo service ldas_pivot restart or sudo service vdas_pivot restart

Monitor how it comes up in the /var/log/ pivot log on the appropriate machine.

Managing PI access

At the present time all post-WFU data is stored on the TACC. At the present time for a PI to gain access they have to get an account on the TACC portal and then contact either Karl Gebhardt or Matthew Shetrone with their login name and we will link their account to the project. Data locations on the TACC is currently: /work/03946/hetdex/maverick/YYYYMMDD where YYYYMMDD is the directory where the data is stored in UT date.

This is all spelled out for the PI at: ​hydra help pages under Phase III