LRS2 Operating Procedures

• LRS2 nightly calibrations
• LRS2 setup and monitoring
• LRS2 Ion Pump and Vacuum Gauge
• LRS2 Taking science exposures
• LRS2 parallel data
• LRS2 End of night procedures
• LRS2 Troubleshooting tips

Calibrations

Calibration Script

A calibration script is available for VIRUS and LRS2. This script attempts to expose all known supported FCU configurations as well as darks and biases. Almost all of the time the command below is what we should do for the standard cals. Here is the standard calibrations commands with times:

• standard set of lamp cals [26 minutes]

cal lrs2 -l qth ldls_long cd-a fear hg -B

• LRS2 and VIRUS biases in parallel [11 minutes]

cal lrs2 -pb -b 11 -B

• LRS2 and VIRUS darks in parallel [22 minutes]

cal lrs2 -pd -de 360 -d 3 -B

• LRS2 and VIRUS, darks and biases in parallel [33 minutes]

cal lrs2 -pb -b 11 -pd -de 360 -d 3 -B

NB: in case of any problem try to get hardware status of instruments.

Calling the script with --help produces more information:

cal lrs2 --help

The list of lamp combinations is spelled out in a table at the beginning of the script. That table also sets the exposure times and warmup times for the lamps.

Use the --dry-run option to see what the script would do given some options without actually executing any commands on the hardware.

For a standard (default) nightly calibration set, the directory structure for your images will be something like:

lrs20000002    Hg     red,blue x3 each
lrs20000003    Cd     blue x3
lrs20000004    FeAr   red x3
lrs20000005    Qth    red x5
lrs20000006    ldls   blue x3
lrs20000007    bias   exp01-11
lrs20000008    dark   exp01-03

You should use lds or [wiki:HetProcedures/RA/autodisplay autodisplay] to inspect at least one image per set if you are taking these in person.

This page contains the latest data on measuring the time each calibration dataset requires.

Stopping Calibrations

If you wish to stop the calibrations elegantly you should use the command sc There is a help for sc with the command sc -h. If you want to stop cal.py in an exposure immediately use the command sc -i but be warned that if you run this during readout the instrument server will crash.

Special calibrations

Special long LRS2 cals to be taken monthly during poor weather

LRS2 setup and monitoring logs

There is a computer called "lrs2" which runs the LRS2 server.

Starting the Servers

The camra_server and data transfer (called pivot) should be running at all times on lrs2, thus no action from the RA should be required.

If you want to check on these servers try:

<user>@vdas ~]$ chksys
hetdex   24118     1  0 Sep06 ?        00:00:49 /opt/het/hetdex/bin/lrs2_server --named-route tcp://192.168.66.99:30000 -c /opt/het/hetdex/etc/conf/lrs2_server.conf
hetdex   20051     1  0 Sep06 ?        00:00:04 /opt/het/hetdex/bin/tcs_proxy --proxy-route tcp://127.0.0.1:39999 --proxy-command /opt/het/hetdex/bin/proxy-pivot-lrs2-data

If you do need to restart the lrs2 data acquisition server and the temperature monitor please follow the LRS2 Start Camra Starting LRS2/VDAS software and monitor procedure.

Setting up monitoring

It is advisable to watch the lrs2_server.log located on the lrs2 machine in /var/log/tcs_logs/lrs2/lrs2_server.log. This can be done with

tail -f /var/log/tcs_logs/lrs2/lrs2_server.log

On another terminal, in a different directory, start a monitor that shows tcs and pfip activity:

monitor --verbose -T -P --key-filter 'log_[^d].*' --log-print

(Note the different filter because of excluding debug messages.)

Ion Pump and Vacuum Gauge

The easiest way to start and stop the LRS2 IP and VG is with ipvg on and ipvg off (remember to use the {-V} flag if you want to control the VIRUS V309 IP as well). Normally it will be turned off during OPS to prepare for the night, and turned on at the end of the night.

Troubleshooting

Restarting the Vacuum Gauge

Most problems with the LRS2 IP/VG can be fixed with the following procedure. If you start the VG and get a very slow response back and perhaps the error:

[astronomer@mcs ~]$ syscmd -l 'enable_pressure_gauges()'
Traceback (most recent call last):
  File "/opt/het/hetdex/bin/syscmd", line 163, in <module>
    result = eval( 'cli.' + command )
  File "<string>", line 1, in <module>
  File "<string>", line 3, in enable_pressure_gauges_method
pytcs.error: remote exception: Failed to connect to the pressure gauge '192.168.66.151:8000'

and in the lrs2 logs:

Failed to connect to the pressure gauge '192.168.66.151:8000'

the following will usually fix it:

1. Make sure the B&R IPs and B&R VG controllers are powered on in the APC tab of the TCS GUI.

2. Run ipvg on

3. Run the following:

   apcCmd off LRS2R-VacGaugeCntrlr
   apcCmd off LRS2B-VacGaugeCntrlr
   sleep 30
   apcCmd on LRS2R-VacGaugeCntrlr
   apcCmd on LRS2B-VacGaugeCntrlr
   sleep 120
   syscmd -l 'get_cryo_pressure( cntl=0, update_state="true" )'
   

4. If you are still unable to get pressure readouts, try this procedure:
   • stop LRS2 server with RA Launcher
   • power off LRS2 VG controllers in APC tab of TCS GUI (2: B & R)
   • wait 30 seconds
   • restart LRS2 server with RA Launcher
   • after LRS2 server log says "Accepting Commands", wait ~30 seconds
   • power on LRS2 VG controllers (B & R)
   • run ipvg on
   • Hopefully that gives you pressure readings with the temperatures. If not, call Jim.

Manual procedures

To turn off the IONPump just hit the APC green light for those two IONPumps to turn them red.

To disable the two VG you must disable them from the command line with

syscmd -l 'disable_pressure_gauges()'

The VG controllers should always remain powered on. You will not see any feed back in the lrs2logs and a normal response to the command line looks like:

[stevenj@zeus ~]$ syscmd -l 'disable_pressure_gauges()'
{u'__ack': u'false',
 u'__async': u'false',
 u'__data': u'false',
 u'__data_time': u'1484447502.964383517',
 u'__done': u'true',
 u'__effective_id': 338,
 u'__error': u'false',
 u'__handler': u'disable_pressure_gauges',
 u'__origin': u'tcsutils.pyc_lrs215227.mcs.15227.0327b23c6->tcp://192.168.66.15:31300',
 u'__original_id': 2,
 u'__pid': 2835,
 u'__wire_time': u'1484447502.964384730'}

A 60 sec dark frame is really the only way to check this. If you fail to turn the IP/VG off our cal script will identify the problem and prevent you from taking observations since any exposures will have extra light evident in the dark frames and science frames. The image below shows the state of the APC LRS2 column with the IP powered off and ready for science observations.

Taking Science Exposures

Use command iexp. Here is the logic:

• Note that LRS2-B has ifu designation of 056, and LRS2-R has ifu designation of 066.
• RA runs commands from target_setup which generates the setup commands, including go_next and the iexp command
• RA runs the iexp command from target_setup, confirms correct exposure/etc, and presses Enter. iexp will wait for setup_complete event to start exposure
• TO sets up on target saving ACQ images throughout
• Once finished, TO clicks "Setup complete" then iexp starts immediately
• iexp
  • retract ACQ (will be done by OTT)
  • start saving images on probes and wfs's (will be done by OTT)
  • start exposure
• at the end of exposure iexp
  • stop saving probes images (will be done by OTT)
  • set exp time for wavefront sensors to 5s (will be done by OTT)
  • cancel setup
  • insert ACQ (will be done by OTT)

At this point system is ready for new commands from target_setup

Observation and exposure determine the directories under which the data are grouped. An example output directory structure might be

/hetdata/data/NWD/lrs2/lrs20000023/exp01/
                                  /exp02/

So here two exposures are grouped under the observation number 23 taken on 20160309.

The combination of observation/exposure must be unique for the UTC date. The agreed upon types are "sci", "cmp", "zro", "flt", "drk", "tst".

To stop and readout an exposure use the command syscmd -l 'abort_exposure()'

Doing Parallel Science with LRS as primary

To get help on this command try vlexp --help for an example with the LRS2 as primary

vlexp -i lrs2 -t sci -texp 600 -pobj Myobject2 -B

NOTE: There will be no dithering for this type of exposure.

To stop and readout both the LRS2 and VIRUS do syscmd -l 'abort_exposure()' and syscmd -V 'abort_exposure()' quickly back to back.

End of Night Procedures

Use ipvg on -V to turn on the LRS2 VG and LRS2 IP (and V309 IP) and look at the LRS2 log to make sure that the VG connected. Wait 10 minutes and then take a pressure reading and record that into the RA logs. The APC tab in the TCS GUI should look like the following:

At the present time all data transfers are automated and no interaction from the RA is required.

Troubleshooting

Checking Hardware Status

syscmd -l 'get_hardware_status( update_state="true" )'

The status may be queried mid-exposure by omitting the update_state parameter.

Typical problems

• For a constant value in an amp you only need to restart the LRS2 server (service), without cycling power to anything.
• For a readout that looks like hash or huge noise you need to restart the server (service) and the controller (using APC tab in TCS GUI)
• For repeating data you will need to restart the controllers and muxes
• For an error in the /var/log/tcs_logs/lrs2/lrs2_server.log that includes an error with "MUX"; you need to restart the server (service), controller (APC tab) and mux (APC tab).

Restarting the Controllers and Muxes

When CAMRA crashes, it is likely due to a controller going offline. That will not be seen in the log messages displayed from the monitor and, in the case described here, if it happens in the middle of an exposure it can leave a connected client script hanging waiting for the response to a command (i.e. expose, get_ccd_temp, etc). The *only* place you can see what is going on in this case is by tailing the /var/log/tcs_logs/lrs2/lrs2_server.log file on lrs2. You will see that the log ends with a string that includes "CArc" and "exception" in the case of a hardware failure.

When there is a problem with the hardware, there should be no attempt to bring lrs2 back up without power cycling the controllers. I don't think this has to include the muxes but here is a prescription for how to restart lrs2 from scratch:

1. From the RA launcher choose: Operations and Camra Servers and LRS2 Server select Stop
   
2. On the APC GUI Power off hardware LRS2 controller
   
3. Turn off the LRS2MUX on the APC GUI
   
4. On the APC GUI Power on LRS2 controller
   

5. Wait 30 seconds then on the APC GUI Power on LRS2MUX
   

6. From the RA launcher choose: Operations and Camra Servers and LRS2 Server select Restart.
7. Watch the /var/log/tcs_logs/lrs2/lrs2_server.log for messages about Accepting commands tail -f /var/log/tcs_logs/lrs2/lrs2_server.log
   
8. then take a test exposure cal lrs2 -b 1 -d 0 -o 5555
9. Check for 0 bias frames with a test lrs2 exposure

Data transfer problem, data left on ramdisk?

• check that Lrs2 DataTransfer? service are running (through RA launcher)
  

If it is stopped, Restart it!

• if data still remains on the RAMdisk, then:
  • as hetdex on lrs2 execute dt_fix lrs2 this will start moving data off the ramdisk but might take 10 minutes.

If the ramdisk fills up (probably because pivot is not running) then try the following:

• stop lrs2Server
• wait 1 minute
• start lrs2Pivot
• check the lrs2_pivot log to make sure it is running properly (use RAlauncher)
• start lrs2Server
• as hetdex on lrs2 execute dt_fix lrs2 this will start moving data off the ramdisk but might take 10 minutes.

Get help on ldas command

• The syscmd mode can be used to display help information on ldas commands. Here are some examples:

  syscmd -l 'help("abort_exposure")'
  syscmd -l 'help("wait_for_readout")'
  syscmd -l 'help("wait_for_write")'
  syscmd -l 'help("wait_for_shutter_close")'
  

• Suppose you only remember part of the command name:

  syscmd -l 'help()' | grep wait
  

• Suppose you want to be notified when the shutter closes:

  syscmd -l 'wait_for_shutter_close(observation=8001, exposure=1  )'
  beep
  beep
  beep
  

The command will not return until the shutter closes. at which point in the example above you'll hear three beeps.

Hardware Changes

If one of the electronics day staff changes a controller a change must be made to the configuration files. For both LRS2 and VIRUS this must be done by either Herman or Jim so call them at home at any hour needed to get the instrument running.

Additional Help

Help for the complete set of CAMRA commands can be obtained by issuing a 'help()' command to the CAMRA subsystem.

syscmd -l 'help()'

Need to add:

• -BR option target_setup for LRS2 SPC and science targets
  • common practice of not waiting and checking
• lds description and svn location
• Steve O's lrs2stars code