= VIRUS Operating Procedures = * [wiki:HetProcedures/RA/virus#cals Calibrations ] * [wiki:HetProcedures/RA/virus#obs Observing ] * [wiki:HetProcedures/RA/virus#IP VIRUS Ion Pumps ] * [wiki:HetProcedures/RA/virus#parallel Parallel ] * [wiki:HetProcedures/RA/virus#vhc VIRUS Health Check ] * [wiki:HetProcedures/RA/virus#remedy VIRUS Quicklook with Remedy ] * [wiki:HetProcedures/RA/virus#park End of the night ] * [wiki:HetProcedures/RA/virus#PR Troubleshooting ] * [wiki:HetProcedures/RA/virus#annex Annex ] [[br]] The VIRUS instrument is usually controlled from zeus which issues commands to vdas. == Startup == #Startup === Starting the Servers === The camra_server and data transfer (called pivot) should be running at all times on vdas, thus no action from the RA should be required. If you want to check on these servers try: {{{ @vdas ~]$ chksys hetdex 8817 1 3 19:27 ? 00:00:02 /home/hetdex/code/het/trunk/camra/camra_server -c /home/hetdex/code/het/trunk/camra/testing/vdas-het.conf hetdex 8847 1 0 19:27 ? 00:00:00 python /home/hetdex/code/het/trunk/camra/testing/vdas_monitor_temperatures.py hetdex 8867 8407 0 19:28 pts/2 00:00:00 grep -i camra }}} If you need to restart the virus server or pivot server use the RAlauncher under Operations and Camera Servers. === Setting up monitoring It is advisable to watch the virus_server.log located on the vdas machine in /var/log/tcs_logs/virus/virus_server.log. This can be done with {{{ tail -f /var/log/tcs_logs/virus/virus_server.log }}} It is useful to pull some logging output to the RA console to be able to follow what is going on with the instruments. The easiest way is using jrf's monitor command: On another terminal, start a monitor that shows tcs and pfip activity (note the different filter, we're excluding debug messages here): {{{ #!div style="font-size: 80%" {{{#!bash monitor --verbose -T -P --key-filter 'log_[^d].*' --log-print }}} }}} At the start of RA OPS the RA should turn off the V309_IONPumpGC1 0.15999 GC1 -0.00001 GC2 -0.15999 GC2 0.00000 WFS1 0.00002 WFS1 0.15997 WFS2 -0.00001 WFS2 -0.16000 on the APC gui. == Notes on File numbering and engineering == #engnum See [wiki:RA/observation_numbering this page] for current numbering schemes. == Calibrations == #cals === Calibration Script The basic set of cals for VIRUS is: * Sometime during the night take Cd-A, Hg, LDLS, darks, and biases: {{{ cal virus -l ldls_long cd-a hg -B }}} - about 19 minutes * To get LRS2 and VIRUS biases in parallel use {{{ cal virus -pb -b 11 -B}}} - about 11 minutes * To get LRS2 and VIRUS darks in parallel use {{{ cal virus -pd -de 360 -d 3 -B}}} - about 22 minutes OR {{{cal lrs2 -pb -b 11 -pd -de 360 -d 3 -B}}} parallel biases and darks together - 33 minutes NB - in case of any problem try to get hardware status of instruments. Calling the script with --help produces more information: {{{ cal virus --help }}} The list of lamp combination is spelled out in a table at the beginning of the script. That table also sets the exposure times and warmup times for the lamps. Use the --dry-run option to see what the script would do given some options without actually executing any commands on the hardware. [wiki:HetProcedures/RA/cal_timing This page] contains the latest data on measuring the time each calibration dataset requires. ==== Stopping Calibrations ==== If you wish to stop the calibrations elegantly you should use the command {{{sc}}} There is a help for sc with the command {{{sc -h}}}. If you want to stop cal.py in an exposure immediately use the command {{{sc -i}}}. ==== Special calibrations ==== 1. 1-2x/night VIRUS line (Cd and Hg) lamp cals * during night can be acquired by {{{gcals}}}, great if closed for clouds during HETDEX * at the end of the night by {{{eon_tasks}}} 2. Weekly VIRUS darks for !Karl/Greg * [wiki:NightOperations/SciencePlans/longcals#virus Darks] === Twilight === See [wiki:HetProcedures/RA/overview#startup Overview of start of night procedures] == Taking Science Exposures == #obs We currently use {{{iexp}}} which handles all of the complexity for us. == saving VIRUS bias images with {{{vjpg.py}}} == #vjpg SVN location: svn+ssh://cetus.het.astronomy.utexas.edu/svn/astronomy/miscutils/vjpg.py Starting in August 2020 we have been using a python code to generate JPG images from VIRUS biases in a consistent format. It will produce two images when it runs - attach the one with the am/pm ending to your emails, like: {{{/hetdata/data/20200827/virus/virus0000027/bias_20200827am.jpg}}}. This code defaults to the most recent exposure, but you can specify {{{-utdate}}} and/or {{{-obs}}} ID number if necessary: {{{ [stevenj@zeus ~]$ vjpg.py -h usage: vjpg.py [-h] [-utdate UTDATE] [-obs OBS] Save JPG image of raw VIRUS bias for specific observation optional arguments: -h, --help show this help message and exit -utdate UTDATE data date. default current UT date -obs OBS observation number. default latest }}} == VIRUS Ion Pumps == #IP Starting in mid-2022, the VIRUS Ion Pump hardware is being run remotely via Phillip and Sam in Austin. Locally we can see whether or not the ion pumps are turned on in a bias image: [[Image(bias_20220822pm_two_IP_on_lo2.jpg, 500px)]] At the command line we can access and control the ion pumps. As of October 2022 we are authorized to turn them off if they are on during science operations. See below for examples: Check the status of the ion pumps: {{{ [stevenj@zeus ~]$ vip status -H lrs2a All ion pumps are off since 2022-10-28 15:18:49Z }}} Turn off the ion pumps: {{{ [stevenj@zeus ~]$ vip off -H lrs2a }}} == VIRUS Health Check == #vhc The VIRUS Health Check (vhc) can be run on vdas as astronomer on data located at /hetdata/data/. When run it performs a series of checks on the VIRUS frames and generates a summary html file. The latest html file is always located in the directory where vhc is run (usually /home/mcs/astronomer/vhcrun). The older html summary files are located observation directory, e.g. /hetdata/data/20170415/virus/virus0000001. A readme.txt can be found in the /home/mcs/astronomer/vhcrun directory. The full set of help pages can be found at [https://vhc.readthedocs.io/en/latest/] PIVOT currently runs vhc for each virus exposure that is executed and saved in the /hetdata/data/ directory. You can point your browser to any of these summary html files either browsing for it using cntrl-O in firefox or by typing in the file location directly e.g. {{{file:///hetdata/data/20170513/virus/virus0000005/flat_recap_0.html}}}. To run vhc manually you can follow the following procedure for running this is: * Login in as astronomer to vdas: ssh -Y astronomer@vdas * cd vhcrun * Example call: vhc -c vhc_settings.cfg -f fplane.txt /hetdata/data/20170415/virus/virus0000001 * To view the result, open vhc_current.html via: firefox /home/mcs/astronomer/vhcrun/vhc_current.html & * leave the browser open and simply reload the same vhc_current.html after each time you run the check. '''To get a nice summary of the errors try viruserr ''' Example: {{{viruserr}}} This will show vhc failed tests for current utdate latest observation [[br]] For specific utdate and observation use command {{{viruserr -utdate YYYYMMDD -obs xxxxxxx}}} '''To monitor vhc output while taking data, try {{{viruserr_watch}}} or monitor the output of [wiki:RA/autodisplay autodisplay]''' == Display one VIRUS exposure == #ds9v The alias {{{ds9v}}} was in use on the shared astronomer account on MCS. Now that we have separate user accounts, this functionality has been replaced by a small bash script also named {{{ds9v}}}, which runs the following command: {{{ [stevenj@zeus ~]$ which ds9v /usr/local/bin/ds9v [stevenj@zeus ~]$ more /usr/local/bin/ds9v ds9 -view layout vertical -view minmax yes -zscale -zoom 0.15 -view buttons no -width 1750 -height 1000 -tile mode grid -tile grid layout 16 20 -lock frame image -lock scale yes -lock colorbar yes *.fits & }}} == Display one IFU of VIRUS data == #virusview The alias {{{virusview}}} was in use on the shared astronomer account on MCS. Now that we have separate user accounts, you can add this alias to your own .bashrc or .cshrc file. The command is: {{{ /usr/local/bin/ds9 -tile row -view vertgraph -view minmax yes -geometry 600x1200 -zscale -zoom 0.2 *_!:1LU* -rotate 180 *_!:1LL* -rotate 180 *_!:1RL* -rotate 180 *_!:1RU* -rotate 180 }}} and it is to be run in a directory like: {{{ /hetdata/data/20230312/virus/virus0000014/exp01/virus }}} and requires as a first argument the three-digit IFU ID, like "052". For example: {{{virusview 052}}} == VIRUS Quicklook with Remedy == #remedy The code Remedy is a python script that can be run on mcs or vdas to reduce a single visit and turn a single ifu into an image. This can be useful for looking to see if an object is centered properly on the IFU and it is used with the Gravity Wave experiment to look for the optical transient (OT). The source code can be downloaded from: https://github.com/grzeimann/Remedy and it is currently living in: /home/mcs/astronomer/bin/Remedy/ As the documentation explains, you need an fplane file and a *.h5 calibrations file (in Remedy/CALS/). -- Fplane is sym-linked to {{{ /opt/het/hetdex/etc/Virus/vhc_config/fplane/fplane.txt }}} (but later is called explicitly, so this doesn't matter) -- The latest *h5 cal file is from TACC: /work/03730/gregz/maverick/test_cal_20190412.h5 When I run it as astronomer on mcs for a particular IFU, this is the command: (note that this will produce output files where ever you run it) {{{ python ~/bin/Remedy/quick_reduction.py 20190412 18 47 ~/bin/Remedy/CALS/test_cal_20190412.h5 --sky_ifuslot 46 --fplane_file /opt/het/hetdex/etc/Virus/vhc_config/fplane/fplane.txt --rootdir /hetdata/data/ }}} That example shows the first GW trigger on HET from the night of 20190412 UT, using observation #18, analyzing IFU 47, and using 46 as sky. It produces a single image of that IFU called: {{{ 20190412_0000020_047.fits }}} and also a full data cube {{{ 20190412_0000020_047_cube.fits }}}, which is fun to scroll through. Best to run this in a clean directory and not in the data directories... Compare to an archival DSS image with "ds9gw" code: /home/mcs/astronomer/bin/ds9gw as follows: {{{ ds9gw 20190412_0000020_047.fits }}} This displays the IFU image in ds9 and then loads a DSS archival image (can do manually under: Analysis, Image Server, DSS changing the size to 2 arcmin in size and then doing a Frame - Match - Frame - WCS) and shows the SDSS DR9 sources which have g<21.5 and their g filter magnitudes. If your observations are not within the SDSS footprint, use this command instead: {{{ ds9gw_noSDSS 20190412_0000020_047.fits }}} [[br]] It can also be run on TACC (lonestar6) with: {{{ idev -N 1 -m 120 -p development python3 /work/03730/gregz/maverick/Remedy/quick_reduction.py 20230905 22 37 /work/03730/gregz/maverick/output/20230301_20230401.h5 -nd 8 -fp /work/03730/gregz/maverick/fplaneall.txt -nD -qs -mc > log_20230905_11_37.txt 2>&1 }}} == End of Night Procedures == #park === V309 Ion Pump === The VIRUS unit 309 formerly had its own ion pump controlled by the APC switch, but as of summer 2022 this ion pump has been removed and replaced with a new Ion Pump as described in the Ion Pump section. === Transfer of data === At the present time all data transfers are automated and no interaction from the RA is required. = Troubleshooting = #PR === Checking Hardware Status === {{{ #!div style="font-size: 80%" {{{#!bash syscmd -v -l 'get_hardware_status( update_state="true" )' }}} }}} {{{ #!div style="font-size: 80%" {{{#!bash syscmd -v -V 'get_hardware_status( update_state="true" )' }}} }}} The status may be queried mid-exposure by omitting the update_state parameter. === Order for VIRUS trouble shooting === As of late Dec2019 we have decided that VIRUS server restarts are to be avoided as much as possible. This stresses the controllers, and there are more benign ways to fix problems associated with cases where numerous VIRUS units (e.g., more than 3) appear to yield disrupted images in bias images. In late Dec2019 it was decided that the primary users of VIRUS consider less than 20% of the available units in a "bad" state is tolerable and preferable to a restart. 1. Take an exposure of the qth and then take a bias. Bad units may be "fixed" in the bias image after an qth exposure. To do this you can use the following single command line: {{{cal virus -B -l qth -L 1 -b -1 -o 6060}}} The "-1" argument to the "-b" flag insures that the bias is taken as the second calibration. Or you can run it without waiting for warmup (if you know the LDLS are on and warmed up) with: {{{vlcal -i virus -l ldls_long -L 1 -o 6060 -b -1 -skip_warmup -B}}} 2. Run {{{vreload}}} ([wiki:RA/virus#vreload more details below]) on remaining bad units. Take an LDLS exposure followed by a bias to assess the results. 3. Run a VIRUS server restart. Use the RA launcher to Stop and Restart the VIRUS server in the Camra Server > VIRUS Server menu. Then take LDLS+bias exposures to assess. 4. Run a VIRUS server restart and also cycle power on multiplexers (muxes) and controllers following the [wiki:RA/virus#restart details included below]. A discussion was begun in late Dec2019 on VIRUS restarts. It emerged that a full restart is hard on the controllers, and we want to try every means of correction prior to that before actually performing step 4. Even step 3, a simple server restart, is apparently stressful for the controllers, and this should be avoided as much as possible. === {{{vreload}}}: Resetting just one controller === #vreload Phillip gave us this tool and its name, '''vreload''', means 'load the VIRUS Digital to Analog Converters. There are a number of flags and options; two of them are:[[BR]] 1. -v makes the command verbose, reporting the voltage values that will be loaded into the controller[[BR]] 2. -n reports what will be loaded into the controller, but does not load the controller. This is basically a test to see if the configuration file (Jim, it's the spectro-nnn.conf file) can be found and read for the spectrograph specified on the command line. The values in the configuration file are what get loaded into the controller.[[BR]] Here are the details for the argument as three example command lines for spectrograph V320: {{{/opt/het/hetdex/bin/vreload V320 -cv -s 0.25 }}} {{{/opt/het/hetdex/bin/vreload S053L -cv -s 0.25 }}} {{{/opt/het/hetdex/bin/vreload V320R -cv -s 0.25 }}} You can reload both the left and right CCDs (LL, LU, RL, and RU as in the first example), the left CCD only (LL and LU as in the second example), and the right CCD only (RL and RU as in the third example). '''vrall''' is a nice command which will parse the VHC output and generate a list of {{{vreload}}} commands which you can copy/paste and run. Below is an example: {{{ [stevenj@zeus ~]$ vrall ssh astronomer@vdas "/opt/het/hetdex/bin/vreload -vcbs 0.25 V317L" ssh astronomer@vdas "/opt/het/hetdex/bin/vreload -vcbs 0.25 V317R" ssh astronomer@vdas "/opt/het/hetdex/bin/vreload -vcbs 0.25 V315L" ssh astronomer@vdas "/opt/het/hetdex/bin/vreload -vcbs 0.25 V315R" ssh astronomer@vdas "/opt/het/hetdex/bin/vreload -vcbs 0.25 V313R" ssh astronomer@vdas "/opt/het/hetdex/bin/vreload -vcbs 0.25 V413L" }}} Some more comments: 1. this can be done as astronomer@vdas, and does not require the user to become hetdex@vdas like we used to think [[BR]] 2. from the command line: {{{/opt/het/hetdex/bin/vreload <-cv>}}} [[BR]] 3. note that this uses either the V number or the S number[[BR]] 4. VIRUS must be inactive when the command is executed. That is, it can't be doing anything except waiting for a command, including integrating, reading out, or reporting temperatures anywhere[[BR]] 5. command {{{load_vdacs}}} cannot work on some types of controller problems, for example, a blown A-to-D converter. The command can be run without an issue, it just fix the problem in that case[[BR]] 6. should a VIRUS stop working 'during the night', I think you should try this command once (or twice if you feel really lucky)[[BR]] 7. you can't do harm with this command if used as specified (and probably even if used randomly)[[BR]] 8. no, it's not in svn, and won't be for a bit[[BR]] === Restarting all of the Controllers and Muxes === #restart Here is a prescription for how to restart the controllers and multiplexers (muxes): ==== Recommended procedures ==== ==== Procedure 1 ==== #p1 * stop [virus|lrs]Server * power off controllers & muxes using {{{vpwr off}}} * wait about a minute * power on muxes & controllers using {{{vpwr on}}} * (re)start [virus|lrs2]Server * monitor server logs * verify the virus|lrs2_server has restarted properly and is "accepting commands" * '''wait 30s after "accepting commands" before attempting to read temperatures''' * try procedure 1 again if functionality is not restored * try procedure two if this fails twice ==== Procedure 2 ==== #p2 Try this directly if the error is "readout in progress". Try this twice then call Jim if you can not restart system. * stop [virus|lrs]Server (if it is not already stopped) * power off controllers & muxes using {{{vpwr off}}} * shutdown vdas with {{{shutdown -h now}}} (will probably have to long-press power button to turn vdas off) * shutdown both magma boxes (in UER) * wait about a minute * power on magma boxes first * power on vdas second; wait for vdas to fully load (check KVM) * power on muxes & controllers using {{{vpwr on}}} * (re)start [virus|lrs2]Server * monitor logs * verify the virus|lrs2_server has restarted properly and is "accepting commands" * '''wait 30s after "accepting commands" before attempting to read temperatures''' * try procedure one if server does not come up the first time * try procedure two again * call Jim === Data Transfer problem === - check that virusDataTransfer service are running (through RA launcher) [[br]] It should report something like : {{{ virusDataTransfer (pid 36304) is running... }}} - if not restart it through RA launcher (and it should copy all data) If the ramdisk fills up (probably because pivot is not running) then try the following: - login to vdas as hetdex - run {{{dt_fix virus}}} === Changing the CCD Temperatures === **In 2022 only Herman and Jim change VIRUS temperatures - contact them if you feel a temperature needs to be changed.** Usually you should make temperature changes through the TCS gui under VIRUS Environment tab look under the Warm-!Up/Cool-Down sub-tab. Enter the VIRUS spectrograph ("V") number under Spec. Hit update temps to see what the current set points are. Enter the new set points under the Target Temps section. If you have to do this through the command line the command is. {{{syscmd -V -v 'set_ccd_temp( ccd=XXXXX, temp=-110 )'}}} To set the temperature for all CCDs in the array, specify ccd=0 in the command parameters NOTE: The commanded {{{setpoint}}} is reset to the configured settings when vdas is restarted. These are located in ldas.conf and vdas.conf To change an individual CCD away from the global configured settings look at /home/hetdex/code/het/trunk/camra/etc/spectro-NNN.conf === Hardware Changes === If one of the electronics day staff changes a controller a change must be made to the configuration files. For both LRS2 and VIRUS this must be done by either Herman or Jim so call them at home at any hour needed to get the instrument running. === Manual fplane update === Automatic generation of fplane file runs daily at 22:30UT. In case if this did not work or there is a need of manually fplane update it can be done loging into vdas as hetdex and by running:[[br]] {{{ update_fplane -c -o /opt/het/hetdex/etc/Virus/vhc_config/fplane/fplane.txt }}} === Recovering the dither position === If you are observing with the dither script and you have a crash you can check the dither position through the TCS GUI in the "Acquisition Camera Status" tab near the bottom. You can move the dither position with the following command: {{{ syscmd -P -v 'AdjustDither(pos=1)' }}} === Additional Help === Help for the complete set of CAMRA commands can be obtained $HET_SRC_ROOT/scripts/by issuing a 'help()' command to the CAMRA subsystem. {{{ #!div style="font-size: 80%" {{{#!bash syscmd -v -l 'help()' }}} }}} === VIRUS annex photo === #annex Very rarely it may be necessary to visit one of the VIRUS enclosure lower annexes to power cycle some of the hardware. Shown below is a photograph of the lower annex on VIRUS enclosure side 1 (with LRS2). [[Image(virus_lower_annex_lo.jpg, 500px)]] == Doing Parallel Science with VIRUS as primary == #parallel To get help on this command try {{{ vlexp --help }}} for an example with the virus as primary {{{ vlexp -i virus -t sci -texp 360 -pobj HF35_W }}} NOTE: This is a little non-standand and we generally never have VIRUS with primary. To stop and readout both the LRS2 and VIRUS do {{{ syscmd -l 'abort_exposure()' }}} and {{{ syscmd -V 'abort_exposure()' }}} quickly back to back. Standard vlexp behavior is to run parallel exposures if exposure time more then 300 seconds To disable that use keyword "-np"