= Data transfer procedures =

Data from LRS2 and VIRUS are transferred using the code pivot.

If you need to transfer guider or acam images over to the TACC try the following:

* login to ldas or vdas as yourself
* su hetdex (use the password known on the board)
* find the directory that you want to transfer, for example /hetdata/data/20160604/gc1 or /hetdata/data/20160603/acm
* scp with -r to maverick.tacc.utexas.edu:/work/03946/hetdex/maverick/YYYYMMDD/ as hetdex
{{{
Example:
scp -r  /hetdata/data/20160603/acm hetdex@maverick.tacc.utexas.edu:/work/03946/hetdex/maverick/20160603/
}}}

== Troubleshooting == #trouble

If the data won't appear in the /hetdata/data directory then probably the permissions are not set properly or the route from the instrument to the disk is broken.   As hetdex on the instrument of interest, e.g. lrs2, do 

{{{
ls -l /hetdata/data
}}}

to check the permissions it should look like 

{{{
drwxrwxr-x   8 hetdex     hetdex     8 Jun 11 02:51 20160611/
drwxrwxr-x   7 hetdex     hetdex     7 Jun 12 03:06 20160612/
drwxrwxr-x   8 hetdex     hetdex     8 Jun 13 02:11 20160613/
drwxrwxr-x   8 hetdex     hetdex     8 Jun 14 21:59 20160614/
drwxrwxr-x   8 hetdex     hetdex     8 Jun 15 00:06 20160615/
drwxrwxr-x   8 hetdex     hetdex     8 Jun 16 23:50 20160616/

OR

drwxrwxr-x   8 guider     staff      8 Jun 17 01:03 20160617/
drwxrwxr-x   4 guider     staff      4 Jun 18 00:24 20160618/
drwxrwxr-x   8 guider     staff      8 Jun 19 05:56 20160619/
drwxrwxr-x   8 guider     staff      8 Jun 20 01:15 20160620/
drwxrwxr-x  10 guider     staff     10 Jun 21 09:17 20160621/
}}}

If you can not see /hetdata/data then the NSF mount is down so call IT support.

If it has the wrong permissions (missing g+w) or group (staff or hetdex) then reset this with

{{{
sudo chown hetdex.hetdex /hetdata/data/20160622
sudo chmod g+w /hetdata/data/20160622
}}}

now transfer the files over that are still waiting.  For example:

{{{

/home/hetdex/code/het/trunk/camra/testing/proxy-pivot-lrs2-data.sh /mnt/camra_ramdisk/20160620/lrs2/lrs2*/exp0*/lrs2/*.fits
}}}

This moves the data to /home/data (for LRS2), /hetdata/data/ and  TACC.

It is important to realize that  you must be logged in as user=hetdex
on the machine vdas (or lrs2). You are denied permission to ssh to hetdex. Hence,, after you 
hav logged in as yourself (i.e. ssh -X sco@vdas) you then must switch to the hetdex account 
(i.e. su - hetdex). Here is a complete example for performing both pivot steps for a set of VIRUS 
data:
{{{
[sco@vdas virustmp]$ ls /var/log/vdas_pivot.log
[sco@vdas virustmp]$ tail /var/log/vdas_pivot.log
rsync: failed to set times on "/hetdata/data/.": Operation not permitted (1)
rsync: recv_generator: mkdir "/hetdata/data/20160624" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
2016-06-24T21:02:54.011 ERROR [tcs_proxy.cpp       :process             :  85] - Command exited 5888.

Note: the previous step created /opt/data/20160624

[sco@vdas ~]$ su - hetdex
Password:
[hetdex@vdas ~]$ /home/hetdex/code/het/trunk/camra/testing/proxy-pivot-virus-data.sh /mnt/camra_ramdisk/20160624/virus/virus*/exp0*/virus/*.fits
Authenticated with partial success.

The following have been reloaded with a version change:
  1) intel/14.0.1.106 => intel/15.0.3  2) mvapich2/2.0b => mvapich2/2.1

[hetdex@vdas ~]$ df
Now we verify that the ramdisk is cleared:
tmpfs                     8388608          0      8388608   0% /mnt/camra_ramdisk

}}}
After the manual proxy-pivot the same images were 
transferred to TACC. However, the more critical point is that the 
ramdisk was cleared. We have recently (mid- and late- Jun2016) had many problems with the 
ramdisk filling and causing a cascade of further problems. In the last step above I show the df command and 
output (for the ramdisk only) that verifies that the ramdisk is cleared. 

=== Problems with PIVOT ===


If the problem is not with the directories then it could be that the PIVOT instance is not running.  
You can check to see if PIVOT is running with a {{{chksys}}} command on the appropriate machine, ie. lrs2 or vdas.  Before stopping or restarting anything check the PIVOT log on the appropriate computer, ie. on the machine lrs2 it would be /var/log/ldas_pivot.log.   Look for anything strange or at least document the last dozen or so lines in a PR.

You can restart the PIVOT routine with 

{{{sudo service ldas_pivot restart   or   sudo service vdas_pivot restart }}}

Monitor how it comes up in the /var/log/ pivot log on the appropriate machine.



== Managing PI access ==

At the present time all post-WFU data is stored on the TACC.   At the present time for a PI to gain access they have to get an account on the TACC portal and then contact either Karl Gebhardt or Matthew Shetrone with their login name and we will link their account to the project.     Data locations on the TACC is currently:   /work/03946/hetdex/maverick/YYYYMMDD where YYYYMMDD is the directory where the data is stored in UT date.

This is all spelled out for the PI at:  [http://hydra.as.utexas.edu/?a=help&h=56 hydra help pages under Phase III]