LRS2 Operating Procedures

The LRS2 instrument is controlled from mcs which issues commands to lrs2.

Startup

Ion Pump and Vacuum Gauge

From the RA launcher under Engineering bring up the APC gui.

At the start of the night turn off the Lrs2B-IONPump, Lrs2R-IONPump, and Lrs2-VacuumGage?. If you fail to turn off these you will have extra light evident in the dark frames and science frames. The image below shows the state of the APC LRS2 column when done.

Starting the Servers

The camra_server and camra_monitor should be running at all times on lrs2, thus no action from the RA should be required.

If you want to check on these servers try the following from your own home directory on lrs2:

hetdex@lrs2 ~]$ ps -ef | grep -i camra
hetdex    8564     1  0 Mar30 ?        00:00:26 /home/hetdex/code/het/trunk/proxy/tcs_proxy --proxy-route ipc:///tmp/ldas_pivot.ipc --proxy-command /home/hetdex/code/het/trunk/camra/testing/proxy-pivot-het-data.sh
hetdex   36468     1  1 19:54 ?        00:01:26 /home/hetdex/code/het/trunk/camra/camra_server -c /home/hetdex/code/het/trunk/camra/etc/ldas.conf
hetdex   38012     1  0 20:41 ?        00:00:00 python /home/hetdex/code/het/trunk/camra/testing/ldas_monitor_temperatures.py
shetrone 40367 40182  0 21:51 pts/0    00:00:00 grep -i ldas

If you do need to restart the lrs data acquisition server and the temperature monitor please follow the LRS2 Start Camra Starting LRS2/VDAS software and monitor procedure.

Setting up monitoring

It is advisable to watch the ldas.log located on the lrs2 machine in /var/log/ldas.log. This can be done with

tail -f /var/log/ldas.log

On another terminal, in a different directory, preferably /home/mcs/astronomer (to avoid mixed calls to the db), start a monitor that shows tcs and pfip activity:

monitor -v -T -P --key-filter 'log_[^d].*' --log-print

(Note the different filter because of excluding debug messages.)

Calibrations

Calibration Script

A prototype calibration script is available that can calibrate VIRUS and LRS2. This script attempts to expose all known supported FCU configurations as well as darks and biases. The console output of the script provides a log of the calibration process so it is recommended to capture the output to a log file via tee. Run this from /home/mcs/astronomer/lrstmp and at the end of the night scp the lrs-cal.YYYYMMDD.log to the lrs2 nightly working directory /opt/data/YYYYMMDD/.

Almost all of the time the command below is what we should do for the standard cals. For the standard cals at the end of the night change the observation number (the -o) rounding up to the next 100. For example if the last science exposure was -o 287 then use -o 301 for the end of night calibrations.

$HET_SRC_ROOT/integration/cal.py -i lrs2 -l lamp1a lamp2a lamp3a -c lamp5b ldls -o 2 |& tee -a lrs-cal.201612DD.log

NB

• in case of any problem try to get hardware status of instruments. if got problem try to kill and start camra_server on lrs2
• full path for cal.py is $HET_SRC_ROOT/integration/cal.py
• adjust number of line and continuum lamps (L and C) and obs number as required

The above will expose the lamp1a lamp1b lamp5b ldls (Hg Cd Qth ldls) as observation 1 (and subsequent numbers) and take 11 biases but 0 darks at the end. This is the standard calibration sequence for LRS2.

Calling the script with --help produces more information:

   $HET_SRC_ROOT/integration/cal.py --help

The list of lamp combinations is spelled out in a table at the beginning of the script. That table also sets the exposure times and warmup times for the lamps.

Use the --dry-run option to see what the script would do given some options without actually executing any commands on the hardware.

For a standard (default) calibration set, the directory structure for your images will be something like:

lrs20000002    Hg     red,blue x3 each
lrs20000003    Cd     blue x3
lrs20000004    FeAr   red x3
lrs20000005    Qt     red x5
lrs20000006    ldls   blue x3
lrs20000007    bias   exp00-10

You should use lrs2view to inspect at least one image per set.

enable_shutter

Twilight

Twilights should be taken with type "twi". If you use GC2 with the B filter and a 0.1 second exposure when you get a count of 30,000 ADU then you are ready to take a 15 second twilight:

syscmd -v -l 'expose( seconds=15, x_binning=1, y_binning=1, type="twi", object="twilight", observation=1, exposure=1 )'

Get help on ldas command

• The syscmd mode can be used to display help information on ldas commands. Here are some examples:

  syscmd -l 'help("abort_exposure")'
  syscmd -l 'help("wait_for_readout")'
  syscmd -l 'help("wait_for_write")'
  syscmd -l 'help("wait_for_shutter_close")'
  

• Suppose you only remember part of the command name:

  syscmd -l 'help()' | grep wait
  

Generic Setup of LRS2 Observation

• Record properties of the target (name, Ra, Dec, Az, ....)
• Run lrs2g to get the initial setup:

  lrs2g 12:38:12.76 +63:55:49.76 13.3 gc2 B
  

• Record Ra,Dec of predicted IHMP and issue ltraj command:

  ltraj 12:38:30.490 +63:57:37.35 E 150
  syscmd -T -v 'load_trajectory( id=150, ra=12.641802778, dec=63.960375000, equinox=2000.0, dir="EAST" )'
  

• We make the initial large guide probe move move using output from lrs2g:

  syscmd -T -v 'Guider2_set_position( x_ang=-0.02311, y_ang=0.16403) '
  syscmd -T -v 'pfip_move_probes()'
  

• Watch tcs monitor until the "correct for structure" is made, then move the guide probes:

  rdsets 12:38:59.409 +64:07:12.14 12:38:59.409 +64:07:12.14
  syscmd -T 'Guider2_set_position( ra=12.649835833, dec=64.120038889, equinox=2000.0, trajectory="current" )'
  syscmd -T 'pfip_move_probes()'
  

• Take LRS2 spectrum, commanding from the /home/mcs/astronomer/lrstmp directory:

  $ syscmd -v -l 'expose( seconds=180, x_binning=1, y_binning=1, type="sci", object="Star2", observation=1001, exposure=1 )'
  

• View images with lrs2view (must be done on lrs2):

  cd /hetdata/data/NWD/lrs2/lrs20001001/exp01/lrs2     ! to get to data directory
  lrs2view 56       ! to view LRS2-B images
  lrs2view 66       ! to view LRS2-R images
  

• Determine offsets needed and apply them:

  $ syscmd -T 'offset_trajectory(dx_ang=0, dy_ang=+1.0, adjust_probes="true")'
  

• To move the star DOWN between banks in lrs2view, we apply a POSITIVE change in dy_ang. The star moves down in altitude.
• To move the peak of a star UP within a bank in lrs2view, we apply a POSITIVE change in dx_ang. The star increases in Az.
• To move from LRS2-B to LRS2-R, we use offset_trajectory:

  $ syscmd -T 'offset_trajectory(dx_ang=-101, dy_ang=4, adjust_probes="true")'
  

  For offsets:
  B2R: syscmd -T 'offset_trajectory(dx_ang=-101, dy_ang=+4, adjust_probes="true")'
  R2B: syscmd -T 'offset_trajectory(dx_ang=+101, dy_ang=-4, adjust_probes="true")'
  syscmd -T 'pfip_clear_positions()'
  

• Note that OFTEN before executing the above offset_trajectory command it can be EXTREMELY useful to clear the old probe positions by executing the following:

  syscmd -T 'pfip_clear_positions()'
  

In some cases, if you have not cleared the old positions, a probe that you are no longer using will leave the annulus when you make your trajectory offset and cause the procedure (for the probe you are using) to fail.

• You may want to create a temporary htopx2 list of LRS2 targets and ​you can read about this here.

Taking Science Exposures

A target must be placed on either the lrs2b or lrs2r markers for the lrs2 to see them. Once the object is on the correct marker and guiding has been achieved on the guide probe be sure to have the TO retract the acquisition camera. It is critical that you log which unit of lrs2 the target is on, since asking for an lrs2 exposure will read out both the lrs2b and lrs2r. It is also important to note the observation number that is currently being used. Do not over-write the beginning of the night calibrations. Perhaps starting the science numbering with 1000 and the start of the night calibration numbers with 100 will avoid this problem?

Be sure to have the TO remove the retract the ACQ

It is important to tell lrs2 that it is going to have control of the PFIP shutter. This is done by executing the following on lrs2 (logged in as yourself):

syscmd -l -v 'enable_shutter()'

There are many ways to take an exposure; the simplest uses Python to issue the expose command, preferably from the /home/mcs/astronomer/lrstmp directory.

# LRS2:
syscmd -v -l 'expose( seconds=20, x_binning=1, y_binning=1, type="sci", object="mrk110", observation=1234, exposure=1 )'

or

$HET_SRC_ROOT/camra/testing/lrs2.py  'expose( seconds=20, x_binning=1, y_binning=1, type="sci", object="mrk110", observation=1234, exposure=1 )'

The exposure command will return a prompt once the readout has started. The output FITS files will be available 40-50 seconds after that, depending on binning and disk I/O. The return string of the expose command provides the output directory for that exposure.

The exposure time is in the "seconds" parameter (minimum of 15).

Observation and exposure determine the directories under which the data are grouped. An example output directory structure might be

/hetdata/data/NWD/lrs2/lrs20000023/exp01/
                             /exp02/

So here two exposures are grouped under the observation number 23 taken on 20160309.

The combination of observation/exposure must be unique for the UTC date. The agreed upon types are "sci", "cmp", "zro", "flt", "drk", "tst".

End of Night Procedures

Ion Pump and Vacuum Gauge

At the present time turning on the ion pump can cause the lrs2 controller to die. To avoid this take down the lrs2 servers, take down the controller and then turn on the ion pump. The procedures is given below:

login to lrs2 and turn off the ldas_monitor and ldas.

sudo service ldas_monitor stop
sudo service ldas stop

From the RA launcher under Engineering bring up the APC gui.

Turn off the LRS2Controller. The image below shows the state of the APC LRS2TDK before you take this action.

Wait about 1 minute then turn on the Lrs2Benable_shutter-IONPump, Lrs2R-IONPump, and Lrs2-VacuumGage?. The image below shows the state of the APC LRS2 column when done.

Wait about 1 minute then turn the LRS2Controller power back on through the APC gui. The image below shows the state of the APC LRS2 column when done.

wait about 1 minute then on lrs2 and turn on the ldas.

sudo service ldas start

Wait until the ldas monitor returns "Accepting commands" and then VERY QUICKLY TAKE A TEST EXPOSURE

$HET_SRC_ROOT/integration/cal.py -i lrs2 -b 1 -d 0 -o 5555

turn on the ldas_monitor

sudo service ldas_monitor start

Record the final LRS2 temperatures into the RA log.

Transfer of data

At the present time all data transfers are automated and no interaction from the RA is required.

Troubleshooting

Checking Hardware Status

syscmd -v -l 'get_hardware_status( update_state="true" )'

syscmd -v -V 'get_hardware_status( update_state="true" )'

The status may be queried mid-exposure by omitting the update_state parameter.

Typical problems

• For a constant value in an amp you only need to restart the service.
• For a readout that looks like hash or huge noise you need to restart the service and the controller
• For an error in the /var/log/ldas.log that includes an error with "MUX"; you need to restart the service, controller and mux.

Restarting the Controllers and Muxes

When CAMRA crashes, it is likely due to a controller going offline. That will not be seen in the log messages displayed from the monitor and, in the case described here, if it happens in the middle of an exposure it can leave a connected client script hanging waiting for the response to a command (i.e. expose, get_ccd_temp, etc). The *only* place you can see what is going on in this case is by tailing the /var/log/vdas.log file on vdas. You will see that the log ends with a string that includes "CArc" and "exception" in the case of a hardware failure.

When there is a problem with the hardware, there should be no attempt to bring vdas back up without power cycling the controllers. I don't think this has to include the muxes but here is a prescription for how to restart vdas from scratch:

1. From your account on lrs2: If running, stop ldas_monitor sudo service ldas_monitor stop
   
2. From your account on lrs2: If running, stop ldas sudo service ldas stop
   
3. On the APC GUI Power off hardware LRS2 controller
   
4. Turn off the LRS2MUX on the APC GUI
   
5. On the APC GUI Power on LRS2 controller
   
6. Wait 30 seconds then on the APC GUI Power on LRS2MUX
   
7. From your account on lrs2: Start ldas sudo service ldas start
   
8. Watch the /var/log/ldas.log for messages about Accepting commands tail -f /var/log/ldas.log
   
9. then VERY QUICKLY TAKE A TEST EXPOSURE $HET_SRC_ROOT/integration/cal.py -i lrs2 -b 1 -d 0 -o 5555
10. From your account on lrs2: Start ldas_monitor sudo service ldas_monitor start
    
11. Check for 0 bias frames with a test lrs2 exposure

Pivot script problem

If pivot is not running and ldas_pivot.log have message like:

Failed to bind to tcp://127.0.0.1:39999: Address already in use

use following recipe

• stop ldas
• wait 5-10 minutes
• start ldas_pivot
• start ldas

Additional Help

Help for the complete set of CAMRA commands can be obtained by issuing a 'help()' command to the CAMRA subsystem.

syscmd -v -l 'help()'