VIRUS Operating Procedures

The VIRUS instrument is controlled from mcs which issues commands to vdas.

Startup

Starting the Servers

The camra_server and camra_monitor should be running at all times on vdas, thus no action from the RA should be required.

If you want to check on these servers try:

<user>@vdas ~]$ ps -ef | grep -i camra
hetdex    8817     1  3 19:27 ?        00:00:02 /home/hetdex/code/het/trunk/camra/camra_server -c /home/hetdex/code/het/trunk/camra/testing/vdas-het.conf
hetdex    8847     1  0 19:27 ?        00:00:00 python /home/hetdex/code/het/trunk/camra/testing/vdas_monitor_temperatures.py
hetdex    8867  8407  0 19:28 pts/2    00:00:00 grep -i camra 

Below are the Linux init scripts are used to start the data acquisition server and temperature monitor on vdas. If needed this should be run on vdas as hetdex.

sudo service vdas start
sudo service vdas_monitor start

Setting up monitoring

It is advisable to watch the vdas.log located on the vdas machine in /var/log/vdas.log. This can be done with

tail -f /var/log/vdas.log

It is useful to pull some logging output to the RA console to be able to follow what is going on with the instruments. The easiest way is using jrf's monitor command:

On another terminal, start a monitor that shows tcs and pfip activity (note the different filter, we're excluding debug messages here):

monitor -v -T -P --key-filter 'log_[^d].*' --log-print

e

Calibrations

Calibration Script

A prototype calibration script is available that can calibrate VIRUS. This script attempts to expose all known supported FCU configurations as well as darks and biases. The console output of the script provides a log of the calibration process so it is recommended to capture the output to a log file via tee.

For CSH shell

$HET_SRC_ROOT/integration/cal.py -i virus -l lamp1a lamp2a -c ldls -o 2 | & tee -a virus-cal.201612DD.log

• TBD: We'll develop a way to do LRS2+VIRUS darks and biases as one set.
• Notice in the example above we have set Observation = 100.

NB

• in case of any problem try to get hardware status of instruments. if got problem try to kill and start camra_server on lrs2
• full path for cal.py is $HET_SRC_ROOT/integration/cal.py
• adjust number of line and continuum lamps (L and C) and obs number as required

The above will expose the lamp1a lamp1b ldls (Hg Cd ldls) as observation 1 (and subsequent numbers) and take 11 darks and biases at the end. This is the standard calibration sequence for VIRUS.

Calling the script with --help produces more information:

$HET_SRC_ROOT/integration/cal.py --help

The list of lamp combinations is spelled out in a table at the beginning of the script. That table also sets the exposure times and warmup times for the lamps.

Use the --dry-run option to see what the script would do given some options without actually executing any commands on the hardware.

Twilight

Twilights should be taken with type "twi". If you use GC2 with the B filter and a 0.1 second exposure when you get a count of 4,500 ADU then you are ready to take a 15 second twilight:

syscmd -v -V 'expose( seconds=15, x_binning=2, y_binning=1, type="twi", object="twilight", observation=1, exposure=1 )'

Taking Science Exposures

It is important to tell lrs2 that it is going to have control of the PFIP shutter. This is done by executing the following on lrs2 (logged in as yourself):

syscmd -V -v 'enable_shutter()' #!bash There are many ways to take an exposure, the simplest uses Python to issue the expose command.

syscmd -v -V 'expose( seconds=20, x_binning=2, y_binning=1, type="sci", object="Test", observation=1234, exposure=1 )'

or

$HET_SRC_ROOT/camra/testing/virus.py 'expose( seconds=20, x_binning=2, y_binning=1, type="sci", object="Test", observation=1234, exposure=1  )'

or

$HET_SRC_ROOT/integration/hetdex-dither.py -e 360 -O GOODSN_west -o 17

The exposure command will return once the readout has started. The output FITS files will be available 40-50 seconds after that, depending on binning and disk I/O. The return string of the expose command provides the output directory for that exposure.

The exposure time is in the "seconds" parameter (minimum of 15).

Observation and exposure determine the directories under which the data are grouped. An example output directory structure might be

/hetdata/data/NWD/virus/virus0000023/exp01/
                               /exp02/

So here two exposures are grouped under the observation number 23 taken on 20160309.

The combination of observation/exposure must be unique for the UTC date. The agreed upon types are "sci", "cmp", "zro", "flt", "drk", "tst".

End of Night Procedures

The Servers

Please leave the servers and monitors up at the end of the night so that they can check on the health of the instrument. You can see if they are running the following from vdas logged in as yourself:

hetdex@lrs2 ~]$ ps -ef | grep -i vdas
hetdex     9246      1  0 Apr10 ?        00:00:13 /home/hetdex/code/het/trunk/proxy/tcs_proxy --proxy-route ipc:///tmp/vdas_pivot.ipc --proxy-command /home/hetdex/code/het/trunk/camra/testing/proxy-pivot-het-data.sh
xymon      9275      1  0 Apr10 ?        00:00:00 /home/xymon/bin/xymonlaunch --config=/home/xymon/etc/clientlaunch.cfg --log=/home/xymon/logs/clientlaunch.log --pidfile=/home/xymon/logs/clientlaunch.vdas.pid
hetdex    10809      1  0 Apr10 ?        00:09:45 /home/hetdex/code/het/trunk/camra/camra_server -c /home/hetdex/code/het/trunk/camra/etc/vdas.conf
xymon     44106      1  0 01:18 ?        00:00:00 sh -c vmstat 300 2 1>/home/xymon/tmp/xymon_vmstat.vdas.44020 2>&1; mv /home/xymon/tmp/xymon_vmstat.vdas.44020 /home/xymon/tmp/xymon_vmstat.vdas
shetrone  44172  43977  0 01:23 pts/3    00:00:00 grep -i vdas

Transfer of data

At the present time all data transfers are automated and no interaction from the RA is required.

Troubleshooting

Checking Hardware Status

syscmd -v -l 'get_hardware_status( update_state="true" )'

syscmd -v -V 'get_hardware_status( update_state="true" )'

The status may be queried mid-exposure by omitting the update_state parameter.

Restarting the Controllers and Muxes

When CAMRA crashes, it is likely due to a controller going offline. That will not be seen in the log messages displayed from the monitor and, in the case described here, if it happens in the middle of an exposure it can leave a connected client script hanging waiting for the response to a command (i.e. expose, get_ccd_temp, etc). The *only* place you can see what is going on in this case is by tailing the /var/log/vdas.log file on vdas. You will see that the log ends with a string that includes "CArc" and "exception" in the case of a hardware failure.

When there is a problem with the hardware, there should be no attempt to bring vdas back up without power cycling the controllers. I don't think this has to include the muxes but here is a prescription for how to restart vdas from scratch:

1. From your account on vdas: If running, stop vdas_monitor sudo service vdas_monitor stop
   
2. From your account on vdas: If running, stop vdas sudo service vdas stop
   
3. On the APC GUI Power off hardware controllers (First Controller 1:2 then Controller 1:1)
   
4. On the APC GUI Power off hardware VIRUS muxes
   
5. Do a sudo halt -h 0 or sudo poweroff for vdas
   
6. Power down the Magma power distribution box
   
7. Wait 15 - 30 minutes.
8. On the APC GUI Power on hardware controllers (First Controller 1:2 then Controller 1:1) wait 10 seconds between each.
   
9. On the APC GUI Power on hardware muxes starting with Mux0 ending with Mux4; wait 5 seconds between each.
   
10. Power on the Magma power distribution box; wait 1 minute after powering
    
11. Power on vdas computer with power button on upper left of machine. You can watch the system power up on VNC.
    
12. From your account on vdas: Start vdas sudo service vdas start
    
13. Watch the /var/log/vdas.log for messages about Accepting commands tail -f /var/log/vdas.log
    
14. Very quickly take a test frame $HET_SRC_ROOT/integration/cal.py -i virus -b 1 -d 0 -o 5555
15. Take a test bias frame with VIRUS to look for bias frames with 0 value (which are not warmed)
    
16. From your account on vdas: Start vdas_monitor sudo service vdas_monitor start
    

NOTE: Even if you see a frame with an all 0 value make sure that one is not currently warmed for repumping. If not then try just restarting the server ( vdas) and see if it fixes that. If you see errors in the vdas.log of any kind but the server is running then try restarting the server only and see if that fixes the problem.

Changing the CCD Temperatures

syscmd -V -v 'set_ccd_temp( ccd=XXXXX, temp=-110 )'

To set the temperature for all CCDs in the array, specify ccd=0 in the command parameters

NOTE: The commanded setpoint is reset to the configured settings when vdas is restarted. These are located in ldas.conf and vdas.conf

To change an individual CCD away from the global configured settings look at /home/hetdex/code/het/trunk/camra/etc/spectro-NNN.conf

Recovering the dither position

If you are observing with the dither script and you have a crash you can check the dither position through the TCS GUI in the "Acquisition Camera Status" tab near the bottom. You can move the dither position with the following command:

syscmd -P -v 'AdjustDither(pos=1)'

Additional Help

Help for the complete set of CAMRA commands can be obtained by issuing a 'help()' command to the CAMRA subsystem.

syscmd -v -l 'help()'