Data transfer procedures

Data from LRS2 and VIRUS are transferred using the code pivot. This script moves data off the RAMDISK to a local disk and then to /hetdata and finally to the TACC.

As of 2022, data transfer is more robust and reliable than it was in the past, with additional details on this page, but if you have questions please ask Steven J and Sergey for help.

Troubleshooting

If the data won't appear in the /hetdata/data directory then pivot is likely down, the permissions are not set properly, or the route from the instrument to the disk is broken. The first check would be to see if pivot is down.

Problems with PIVOT

If the problem is not with the directories then it could be that the PIVOT instance is not running or its socket has been taken. You can check to see if PIVOT is running with a chksys command on the appropriate machine, ie. lrs2 or vdas. Before stopping or restarting anything check the PIVOT log on the appropriate computer, ie. on the machine lrs2 it would be /var/log/ldas_pivot.log. Look for anything strange or at least document the last dozen or so lines in a PR.

You can restart the PIVOT routine with the pull down menu of the RA launcher for the instrument of interest.

Monitor how it comes up in the /var/log/ pivot log on the appropriate machine. Note that this normally needs to be done before the instrument server comes up, but give it a try and see if you have any luck.

Now check to see if there is data left on the RAMDISK with a df command.

To get the older data moving again you will need to do the following as hetdex on the machine of interest (e.g. vdas): data_check virus. But note this takes a long time to get that data going so be patient (perhaps 10 minutes or longer).

If you still have problems getting PIVOT to come up then try restarting tcsNameD from the TO launcher.

Permissions problems

As hetdex on the instrument of interest, e.g. lrs2, do

ls -l /hetdata/data

to check the permissions it should look like

drwxrwxr-x   8 hetdex     hetdex     8 Jun 11 02:51 20160611/
drwxrwxr-x   7 hetdex     hetdex     7 Jun 12 03:06 20160612/
drwxrwxr-x   8 hetdex     hetdex     8 Jun 13 02:11 20160613/
drwxrwxr-x   8 hetdex     hetdex     8 Jun 14 21:59 20160614/
drwxrwxr-x   8 hetdex     hetdex     8 Jun 15 00:06 20160615/
drwxrwxr-x   8 hetdex     hetdex     8 Jun 16 23:50 20160616/

If you can not see /hetdata/data then the NSF mount is down so call IT support.

If it has the wrong permissions (missing g+w) or group (staff or hetdex) then reset this with

sudo chown hetdex.hetdex /hetdata/data/20160622
sudo chmod g+w /hetdata/data/20160622

now transfer the files over that are still waiting. See instructions for each instrument under trouble shooting for that instrument.

Managing PI access

At the present time all post-WFU data is stored on the TACC. At the present time for a PI to gain access they have to get an account on the TACC portal and then contact either Karl Gebhardt or Greg Zeimann with their login name and we will link their account to the project. Data locations on the TACC is currently: /work/03946/hetdex/maverick/YYYYMMDD where YYYYMMDD is the directory where the data is stored in UT date.

This is all spelled out for the PI at: ​hydra help pages under Phase III

File locations

This information is current as of Dec 14, 2019, and comes from an email from Jim:

• The data in /hetdata/data/YYYYmmdd is transferred only to TACC and is not duplicated at the astronomy department

• The data in /hetdata/data/ImageQuality is not duplicated anywhere.

• The data in /opt/het/hetdex is currently replicated on cerberus. Following the transfer of gringotts file systems to hetnas1 these files will be replicated on hetnas2. Neither cerberus nor hetnas2 have files systems that are exported to other machines.