VIRUS Operating Procedures

• Calibrations
• Observing
• VIRUS Ion Pumps
• Parallel
• VIRUS Health Check
• VIRUS Quicklook with Remedy
• End of the night
• Troubleshooting
• Annex

The VIRUS instrument is usually controlled from zeus which issues commands to vdas.

Startup

Starting the Servers

The camra_server and data transfer (called pivot) should be running at all times on vdas, thus no action from the RA should be required.

If you want to check on these servers try:

<user>@vdas ~]$ chksys
hetdex    8817     1  3 19:27 ?        00:00:02 /home/hetdex/code/het/trunk/camra/camra_server -c /home/hetdex/code/het/trunk/camra/testing/vdas-het.conf
hetdex    8847     1  0 19:27 ?        00:00:00 python /home/hetdex/code/het/trunk/camra/testing/vdas_monitor_temperatures.py
hetdex    8867  8407  0 19:28 pts/2    00:00:00 grep -i camra 

If you need to restart the virus server or pivot server use the RAlauncher under Operations and Camera Servers.

Setting up monitoring

It is advisable to watch the virus_server.log located on the vdas machine in /var/log/tcs_logs/virus/virus_server.log. This can be done with

tail -f /var/log/tcs_logs/virus/virus_server.log

It is useful to pull some logging output to the RA console to be able to follow what is going on with the instruments. The easiest way is using jrf's monitor command:

On another terminal, start a monitor that shows tcs and pfip activity (note the different filter, we're excluding debug messages here):

monitor --verbose -T -P --key-filter 'log_[^d].*' --log-print

At the start of RA OPS the RA should turn off the V309_IONPumpGC1 0.15999 GC1 -0.00001 GC2 -0.15999 GC2 0.00000 WFS1 0.00002 WFS1 0.15997 WFS2 -0.00001 WFS2 -0.16000

on the APC gui.

Notes on File numbering and engineering

See this page for current numbering schemes.

Calibrations

Calibration Script

The basic set of cals for VIRUS is:

• Sometime during the night take Cd-A, Hg, LDLS, darks, and biases:

cal virus -l ldls_long cd-a hg -B - about 19 minutes

• To get LRS2 and VIRUS biases in parallel use

cal virus -pb -b 11 -B - about 11 minutes

• To get LRS2 and VIRUS darks in parallel use

cal virus -pd -de 360 -d 3 -B - about 22 minutes

OR cal lrs2 -pb -b 11 -pd -de 360 -d 3 -B parallel biases and darks together - 33 minutes

NB

• in case of any problem try to get hardware status of instruments.

Calling the script with --help produces more information:

cal virus --help

The list of lamp combination is spelled out in a table at the beginning of the script. That table also sets the exposure times and warmup times for the lamps.

Use the --dry-run option to see what the script would do given some options without actually executing any commands on the hardware.

This page contains the latest data on measuring the time each calibration dataset requires.

Stopping Calibrations

If you wish to stop the calibrations elegantly you should use the command sc There is a help for sc with the command sc -h. If you want to stop cal.py in an exposure immediately use the command sc -i.

Special calibrations

1. 1-2x/night VIRUS line (Cd and Hg) lamp cals
   • during night can be acquired by gcals, great if closed for clouds during HETDEX
   • at the end of the night by eon_tasks

2. Weekly VIRUS darks for Karl/Greg
   • Darks

Twilight

See Overview of start of night procedures

Taking Science Exposures

We currently use iexp which handles all of the complexity for us.

saving VIRUS bias images with vjpg.py

SVN location: ​svn+ssh://cetus.het.astronomy.utexas.edu/svn/astronomy/miscutils/vjpg.py

Starting in August 2020 we have been using a python code to generate JPG images from VIRUS biases in a consistent format. It will produce two images when it runs - attach the one with the am/pm ending to your emails, like: /hetdata/data/20200827/virus/virus0000027/bias_20200827am.jpg. This code defaults to the most recent exposure, but you can specify -utdate and/or -obs ID number if necessary:

[stevenj@zeus ~]$ vjpg.py -h
usage: vjpg.py [-h] [-utdate UTDATE] [-obs OBS]

Save JPG image of raw VIRUS bias for specific observation

optional arguments:
  -h, --help      show this help message and exit
  -utdate UTDATE  data date. default current UT date
  -obs OBS        observation number. default latest

VIRUS Ion Pumps

Starting in mid-2022, the VIRUS Ion Pump hardware is being run remotely via Phillip and Sam in Austin. Locally we can see whether or not the ion pumps are turned on in a bias image:

At the command line we can access and control the ion pumps. As of October 2022 we are authorized to turn them off if they are on during science operations. See below for examples:

Check the status of the ion pumps:

[stevenj@zeus ~]$ vip status -H lrs2a
All ion pumps are off since 2022-10-28 15:18:49Z

Turn off the ion pumps:

[stevenj@zeus ~]$ vip off -H lrs2a

VIRUS Health Check

The VIRUS Health Check (vhc) can be run on vdas as astronomer on data located at /hetdata/data/. When run it performs a series of checks on the VIRUS frames and generates a summary html file. The latest html file is always located in the directory where vhc is run (usually /home/mcs/astronomer/vhcrun). The older html summary files are located observation directory, e.g. /hetdata/data/20170415/virus/virus0000001.

A readme.txt can be found in the /home/mcs/astronomer/vhcrun directory. The full set of help pages can be found at ​https://vhc.readthedocs.io/en/latest/

PIVOT currently runs vhc for each virus exposure that is executed and saved in the /hetdata/data/ directory. You can point your browser to any of these summary html files either browsing for it using cntrl-O in firefox or by typing in the file location directly e.g. file:///hetdata/data/20170513/virus/virus0000005/flat_recap_0.html.

To run vhc manually you can follow the following procedure for running this is:

• Login in as astronomer to vdas: ssh -Y astronomer@vdas
• cd vhcrun
• Example call: vhc -c vhc_settings.cfg -f fplane.txt /hetdata/data/20170415/virus/virus0000001
• To view the result, open vhc_current.html via: firefox /home/mcs/astronomer/vhcrun/vhc_current.html &
• leave the browser open and simply reload the same vhc_current.html after each time you run the check.

To get a nice summary of the errors try viruserr

Example: viruserr This will show vhc failed tests for current utdate latest observation

For specific utdate and observation use command viruserr -utdate YYYYMMDD -obs xxxxxxx

To monitor vhc output while taking data, try viruserr_watch or monitor the output of autodisplay

Display one VIRUS exposure

The alias ds9v was in use on the shared astronomer account on MCS. Now that we have separate user accounts, this functionality has been replaced by a small bash script also named ds9v, which runs the following command:

[stevenj@zeus ~]$ which ds9v
/usr/local/bin/ds9v
[stevenj@zeus ~]$ more /usr/local/bin/ds9v
ds9 -view layout vertical -view minmax yes -zscale -zoom 0.15 -view buttons no -width 1750 -height 1000 -tile mode 
grid -tile grid layout 16 20 -lock frame image -lock scale yes -lock colorbar yes *.fits &

Display one IFU of VIRUS data

The alias virusview was in use on the shared astronomer account on MCS. Now that we have separate user accounts, you can add this alias to your own .bashrc or .cshrc file. The command is:

/usr/local/bin/ds9 -tile row -view vertgraph -view minmax yes -geometry 600x1200 -zscale -zoom 0.2 *_!:1LU* -rotate 180 *_!:1LL* -rotate 180 *_!:1RL* -rotate 180 *_!:1RU* -rotate 180

and it is to be run in a directory like:

/hetdata/data/20230312/virus/virus0000014/exp01/virus

and requires as a first argument the three-digit IFU ID, like "052". For example: virusview 052

VIRUS Quicklook with Remedy

The code Remedy is a python script that can be run on mcs or vdas to reduce a single visit and turn a single ifu into an image. This can be useful for looking to see if an object is centered properly on the IFU and it is used with the Gravity Wave experiment to look for the optical transient (OT).

The source code can be downloaded from: ​https://github.com/grzeimann/Remedy

and it is currently living in: /home/mcs/astronomer/bin/Remedy/

As the documentation explains, you need an fplane file and a *.h5 calibrations file (in Remedy/CALS/).

-- Fplane is sym-linked to /opt/het/hetdex/etc/Virus/vhc_config/fplane/fplane.txt (but later is called explicitly, so this doesn't matter)

-- The latest *h5 cal file is from TACC: /work/03730/gregz/maverick/test_cal_20190412.h5

When I run it as astronomer on mcs for a particular IFU, this is the command: (note that this will produce output files where ever you run it)

python ~/bin/Remedy/quick_reduction.py 20190412 18 47 ~/bin/Remedy/CALS/test_cal_20190412.h5 --sky_ifuslot 46 --fplane_file /opt/het/hetdex/etc/Virus/vhc_config/fplane/fplane.txt --rootdir /hetdata/data/

That example shows the first GW trigger on HET from the night of 20190412 UT, using observation #18, analyzing IFU 47, and using 46 as sky.

It produces a single image of that IFU called: 20190412_0000020_047.fits and also a full data cube 20190412_0000020_047_cube.fits , which is fun to scroll through. Best to run this in a clean directory and not in the data directories...

Compare to an archival DSS image with "ds9gw" code: /home/mcs/astronomer/bin/ds9gw as follows:

ds9gw 20190412_0000020_047.fits

This displays the IFU image in ds9 and then loads a DSS archival image (can do manually under: Analysis, Image Server, DSS changing the size to 2 arcmin in size and then doing a Frame - Match - Frame - WCS) and shows the SDSS DR9 sources which have g<21.5 and their g filter magnitudes.

If your observations are not within the SDSS footprint, use this command instead:

ds9gw_noSDSS 20190412_0000020_047.fits

It can also be run on TACC (lonestar6) with:

idev -N 1 -m 120 -p development
python3 /work/03730/gregz/maverick/Remedy/quick_reduction.py 20230905 22 37 /work/03730/gregz/maverick/output/20230301_20230401.h5 -nd 8 -fp /work/03730/gregz/maverick/fplaneall.txt -nD -qs -mc > log_20230905_11_37.txt 2>&1

End of Night Procedures

V309 Ion Pump

The VIRUS unit 309 formerly had its own ion pump controlled by the APC switch, but as of summer 2022 this ion pump has been removed and replaced with a new Ion Pump as described in the Ion Pump section.

Transfer of data

At the present time all data transfers are automated and no interaction from the RA is required.

Troubleshooting

Checking Hardware Status

syscmd -v -l 'get_hardware_status( update_state="true" )'

syscmd -v -V 'get_hardware_status( update_state="true" )'

The status may be queried mid-exposure by omitting the update_state parameter.

Order for VIRUS trouble shooting

As of late Dec2019 we have decided that VIRUS server restarts are to be avoided as much as possible. This stresses the controllers, and there are more benign ways to fix problems associated with cases where numerous VIRUS units (e.g., more than 3) appear to yield disrupted images in bias images. In late Dec2019 it was decided that the primary users of VIRUS consider less than 20% of the available units in a "bad" state is tolerable and preferable to a restart.

1. Take an exposure of the qth and then take a bias. Bad units may be "fixed" in the bias image after an qth exposure. To do this you can use the following single command line:

cal virus -B -l qth -L 1 -b -1 -o 6060

The "-1" argument to the "-b" flag insures that the bias is taken as the second calibration.

Or you can run it without waiting for warmup (if you know the LDLS are on and warmed up) with:

vlcal -i virus -l ldls_long -L 1 -o 6060 -b -1 -skip_warmup -B

2. Run vreload (more details below) on remaining bad units. Take an LDLS exposure followed by a bias to assess the results.

3. Run a VIRUS server restart. Use the RA launcher to Stop and Restart the VIRUS server in the Camra Server > VIRUS Server menu. Then take LDLS+bias exposures to assess.

4. Run a VIRUS server restart and also cycle power on multiplexers (muxes) and controllers following the details included below.

A discussion was begun in late Dec2019 on VIRUS restarts. It emerged that a full restart is hard on the controllers, and we want to try every means of correction prior to that before actually performing step 4. Even step 3, a simple server restart, is apparently stressful for the controllers, and this should be avoided as much as possible.

vreload: Resetting just one controller

Phillip gave us this tool and its name, vreload, means 'load the VIRUS Digital to Analog Converters.

There are a number of flags and options; two of them are:

1. -v makes the command verbose, reporting the voltage values that will be loaded into the controller
   
2. -n reports what will be loaded into the controller, but does not load the controller. This is basically a test to see if the configuration file (Jim, it's the spectro-nnn.conf file) can be found and read for the spectrograph specified on the command line. The values in the configuration file are what get loaded into the controller.
   

Here are the details for the argument <spectrograph> as three example command lines for spectrograph V320:

/opt/het/hetdex/bin/vreload V320 -cv -s 0.25

/opt/het/hetdex/bin/vreload S053L -cv -s 0.25

/opt/het/hetdex/bin/vreload V320R -cv -s 0.25

You can reload both the left and right CCDs (LL, LU, RL, and RU as in the first example), the left CCD only (LL and LU as in the second example), and the right CCD only (RL and RU as in the third example).

vrall is a nice command which will parse the VHC output and generate a list of vreload commands which you can copy/paste and run. Below is an example:

[stevenj@zeus ~]$ vrall
ssh astronomer@vdas "/opt/het/hetdex/bin/vreload -vcbs 0.25 V317L" 
ssh astronomer@vdas "/opt/het/hetdex/bin/vreload -vcbs 0.25 V317R" 
ssh astronomer@vdas "/opt/het/hetdex/bin/vreload -vcbs 0.25 V315L" 
ssh astronomer@vdas "/opt/het/hetdex/bin/vreload -vcbs 0.25 V315R" 
ssh astronomer@vdas "/opt/het/hetdex/bin/vreload -vcbs 0.25 V313R" 
ssh astronomer@vdas "/opt/het/hetdex/bin/vreload -vcbs 0.25 V413L" 

Some more comments:

1. this can be done as astronomer@vdas, and does not require the user to become hetdex@vdas like we used to think
   
2. from the command line: /opt/het/hetdex/bin/vreload <spectrograph> <-cv>
   
3. note that this uses either the V number or the S number
   
4. VIRUS must be inactive when the command is executed. That is, it can't be doing anything except waiting for a command, including integrating, reading out, or reporting temperatures anywhere
   
5. command load_vdacs cannot work on some types of controller problems, for example, a blown A-to-D converter. The command can be run without an issue, it just fix the problem in that case
   
6. should a VIRUS stop working 'during the night', I think you should try this command once (or twice if you feel really lucky)
   
7. you can't do harm with this command if used as specified (and probably even if used randomly)
   
8. no, it's not in svn, and won't be for a bit
   

Restarting all of the Controllers and Muxes

Here is a prescription for how to restart the controllers and multiplexers (muxes):

Recommended procedures

Procedure 1

• stop [virus|lrs]Server
• power off controllers & muxes using vpwr off
• wait about a minute
• power on muxes & controllers using vpwr on
• (re)start [virus|lrs2]Server
• monitor server logs
• verify the virus|lrs2_server has restarted properly and is "accepting commands"
  • wait 30s after "accepting commands" before attempting to read temperatures
• try procedure 1 again if functionality is not restored
• try procedure two if this fails twice

Procedure 2

Try this directly if the error is "readout in progress". Try this twice then call Jim if you can not restart system.

• stop [virus|lrs]Server (if it is not already stopped)
• power off controllers & muxes using vpwr off
• shutdown vdas with shutdown -h now (will probably have to long-press power button to turn vdas off)
• shutdown both magma boxes (in UER)
• wait about a minute
• power on magma boxes first
• power on vdas second; wait for vdas to fully load (check KVM)
• power on muxes & controllers using vpwr on
• (re)start [virus|lrs2]Server
• monitor logs
• verify the virus|lrs2_server has restarted properly and is "accepting commands"
  • wait 30s after "accepting commands" before attempting to read temperatures
• try procedure one if server does not come up the first time
• try procedure two again
• call Jim

Data Transfer problem

• check that virusDataTransfer service are running (through RA launcher)
  It should report something like : virusDataTransfer (pid 36304) is running...
• if not restart it through RA launcher (and it should copy all data)

If the ramdisk fills up (probably because pivot is not running) then try the following:

• login to vdas as hetdex
• run dt_fix virus

Changing the CCD Temperatures

In 2022 only Herman and Jim change VIRUS temperatures - contact them if you feel a temperature needs to be changed.

Usually you should make temperature changes through the TCS gui under VIRUS Environment tab look under the Warm-Up/Cool-Down sub-tab. Enter the VIRUS spectrograph ("V") number under Spec. Hit update temps to see what the current set points are. Enter the new set points under the Target Temps section.

If you have to do this through the command line the command is.

syscmd -V -v 'set_ccd_temp( ccd=XXXXX, temp=-110 )'

To set the temperature for all CCDs in the array, specify ccd=0 in the command parameters

NOTE: The commanded setpoint is reset to the configured settings when vdas is restarted. These are located in ldas.conf and vdas.conf

To change an individual CCD away from the global configured settings look at /home/hetdex/code/het/trunk/camra/etc/spectro-NNN.conf

Hardware Changes

If one of the electronics day staff changes a controller a change must be made to the configuration files. For both LRS2 and VIRUS this must be done by either Herman or Jim so call them at home at any hour needed to get the instrument running.

Manual fplane update

Automatic generation of fplane file runs daily at 22:30UT. In case if this did not work or there is a need of manually fplane update it can be done loging into vdas as hetdex and by running:
update_fplane -c -o /opt/het/hetdex/etc/Virus/vhc_config/fplane/fplane.txt

Recovering the dither position

If you are observing with the dither script and you have a crash you can check the dither position through the TCS GUI in the "Acquisition Camera Status" tab near the bottom. You can move the dither position with the following command:

syscmd -P -v 'AdjustDither(pos=1)'

Additional Help

Help for the complete set of CAMRA commands can be obtained $HET_SRC_ROOT/scripts/by issuing a 'help()' command to the CAMRA subsystem.

syscmd -v -l 'help()'

VIRUS annex photo

Very rarely it may be necessary to visit one of the VIRUS enclosure lower annexes to power cycle some of the hardware. Shown below is a photograph of the lower annex on VIRUS enclosure side 1 (with LRS2).

Doing Parallel Science with VIRUS as primary

To get help on this command try vlexp --help for an example with the virus as primary

vlexp -i virus -t sci -texp 360 -pobj HF35_W

NOTE: This is a little non-standand and we generally never have VIRUS with primary.

To stop and readout both the LRS2 and VIRUS do syscmd -l 'abort_exposure()' and syscmd -V 'abort_exposure()' quickly back to back.

Standard vlexp behavior is to run parallel exposures if exposure time more then 300 seconds To disable that use keyword "-np"