= LRS2 Operating Procedures = * [wiki:HetProcedures/RA/lrs2#cals LRS2 nightly calibrations] * [wiki:HetProcedures/RA/lrs2#setup LRS2 setup and monitoring] * [wiki:HetProcedures/RA/lrs2#ipvg LRS2 Ion Pump and Vacuum Gauge] * [wiki:HetProcedures/RA/lrs2#obs LRS2 Taking science exposures] * [wiki:HetProcedures/RA/lrs2#parallel LRS2 parallel data] * [wiki:HetProcedures/RA/lrs2#park LRS2 End of night procedures] * [wiki:HetProcedures/RA/lrs2#PR LRS2 Troubleshooting tips] [[br]] == Calibrations == #cals === Calibration Script A calibration script is available for VIRUS and LRS2. This script attempts to expose all known supported FCU configurations as well as darks and biases. Almost all of the time the command below is what we should do for the standard cals. Here is the standard calibrations commands with times: * standard set of lamp cals [26 minutes] {{{cal lrs2 -l qth ldls_long cd-a fear hg -B}}} * LRS2 and VIRUS biases in parallel [11 minutes] {{{cal lrs2 -pb -b 11 -B}}} * LRS2 and VIRUS darks in parallel [22 minutes] {{{ cal lrs2 -pd -de 360 -d 3 -B}}} * LRS2 and VIRUS, darks and biases in parallel [33 minutes] {{{cal lrs2 -pb -b 11 -pd -de 360 -d 3 -B}}} NB: in case of any problem try to get hardware status of instruments. Calling the script with --help produces more information: {{{cal lrs2 --help}}} The list of lamp combinations is spelled out in a table at the beginning of the script. That table also sets the exposure times and warmup times for the lamps. Use the {{{--dry-run}}} option to see what the script would do given some options without actually executing any commands on the hardware. For a standard (default) nightly calibration set, the directory structure for your images will be something like: {{{ lrs20000002 Hg red,blue x3 each lrs20000003 Cd blue x3 lrs20000004 FeAr red x3 lrs20000005 Qth red x5 lrs20000006 ldls blue x3 lrs20000007 bias exp01-11 lrs20000008 dark exp01-03 }}} You should use {{{lds}}} or {{{[wiki:HetProcedures/RA/autodisplay autodisplay]}}} to inspect at least one image per set if you are taking these in person. [wiki:HetProcedures/RA/cal_timing This page] contains the latest data on measuring the time each calibration dataset requires. ==== Stopping Calibrations ==== If you wish to stop the calibrations elegantly you should use the command {{{sc}}} There is a help for sc with the command {{{sc -h}}}. If you want to stop cal.py in an exposure immediately use the command {{{sc -i}}} but **be warned that if you run this during readout the instrument server will crash**. ==== Special calibrations ==== [wiki:NightOperations/SciencePlans/longcals#lrs2 Special long LRS2 cals to be taken monthly during poor weather] [[br]] == LRS2 setup and monitoring logs == #setup There is a computer called "lrs2" which runs the LRS2 server. === Starting the Servers === The camra_server and [wiki:HetProcedures/RA/DT data transfer] (called pivot) should be running at all times on lrs2, thus no action from the RA should be required. If you want to check on these servers try: {{{ @vdas ~]$ chksys hetdex 24118 1 0 Sep06 ? 00:00:49 /opt/het/hetdex/bin/lrs2_server --named-route tcp://192.168.66.99:30000 -c /opt/het/hetdex/etc/conf/lrs2_server.conf hetdex 20051 1 0 Sep06 ? 00:00:04 /opt/het/hetdex/bin/tcs_proxy --proxy-route tcp://127.0.0.1:39999 --proxy-command /opt/het/hetdex/bin/proxy-pivot-lrs2-data }}} If you do need to restart the lrs2 data acquisition server and the temperature monitor please follow the [wiki:StartCamra LRS2 Start Camra] Starting LRS2/VDAS software and monitor procedure. === Setting up monitoring === It is advisable to watch the lrs2_server.log located on the lrs2 machine in {{{/var/log/tcs_logs/lrs2/lrs2_server.log}}}. This can be done with {{{ tail -f /var/log/tcs_logs/lrs2/lrs2_server.log }}} On another terminal, in a different directory, start a monitor that shows tcs and pfip activity: {{{ monitor --verbose -T -P --key-filter 'log_[^d].*' --log-print }}} (Note the different filter because of excluding debug messages.) [[br]] == Ion Pump and Vacuum Gauge == #ipvg The easiest way to start and stop the LRS2 IP and VG is with {{{ipvg on}}} and {{{ipvg off}}} (remember to use the {-V} flag if you want to control the VIRUS V309 IP as well). Normally it will be turned off during OPS to prepare for the night, and turned on at the end of the night. === Troubleshooting === ==== Restarting the Vacuum Gauge ==== Most problems with the LRS2 IP/VG can be fixed with the following procedure. If you start the VG and get a very slow response back and perhaps the error: {{{ [astronomer@mcs ~]$ syscmd -l 'enable_pressure_gauges()' Traceback (most recent call last): File "/opt/het/hetdex/bin/syscmd", line 163, in result = eval( 'cli.' + command ) File "", line 1, in File "", line 3, in enable_pressure_gauges_method pytcs.error: remote exception: Failed to connect to the pressure gauge '192.168.66.151:8000' }}} and in the lrs2 logs: {{{ Failed to connect to the pressure gauge '192.168.66.151:8000' }}} the following will usually fix it: 1. Make sure the B&R IPs and B&R VG controllers are powered on in the APC tab of the TCS GUI. 2. Run {{{ipvg on}}} 3. Run the following: {{{ apcCmd off LRS2R-VacGaugeCntrlr apcCmd off LRS2B-VacGaugeCntrlr sleep 30 apcCmd on LRS2R-VacGaugeCntrlr apcCmd on LRS2B-VacGaugeCntrlr sleep 120 syscmd -l 'get_cryo_pressure( cntl=0, update_state="true" )' }}} 4. If you are still unable to get pressure readouts, try this procedure: * stop LRS2 server with RA Launcher * power off LRS2 VG controllers in APC tab of TCS GUI (2: B & R) * wait 30 seconds * restart LRS2 server with RA Launcher * after LRS2 server log says "Accepting Commands", wait ~30 seconds * power on LRS2 VG controllers (B & R) * run {{{ipvg on}}} * Hopefully that gives you pressure readings with the temperatures. If not, call Jim. ==== Manual procedures ==== To turn off the IONPump just hit the APC green light for those two IONPumps to turn them red. To disable the two VG you must disable them from the command line with {{{ syscmd -l 'disable_pressure_gauges()' }}} The VG controllers should always remain powered on. You will not see any feed back in the lrs2logs and a normal response to the command line looks like: {{{ [stevenj@zeus ~]$ syscmd -l 'disable_pressure_gauges()' {u'__ack': u'false', u'__async': u'false', u'__data': u'false', u'__data_time': u'1484447502.964383517', u'__done': u'true', u'__effective_id': 338, u'__error': u'false', u'__handler': u'disable_pressure_gauges', u'__origin': u'tcsutils.pyc_lrs215227.mcs.15227.0327b23c6->tcp://192.168.66.15:31300', u'__original_id': 2, u'__pid': 2835, u'__wire_time': u'1484447502.964384730'} }}} A 60 sec dark frame is really the only way to check this. If you fail to turn the IP/VG off our {{{cal}}} script will identify the problem and prevent you from taking observations since any exposures will have extra light evident in the dark frames and science frames. The image below shows the state of the APC LRS2 column with the IP powered off and ready for science observations. [[Image(APCLRS2startofnight.png)]] [[br]] == Taking Science Exposures == #obs Use command {{{iexp}}}. Here is the logic: * Note that LRS2-B has ifu designation of 056, and LRS2-R has ifu designation of 066. * RA runs commands from {{{target_setup}}} which generates the setup commands, including {{{go_next}}} and the {{{iexp}}} command * RA runs the {{{iexp}}} command from {{{target_setup}}}, confirms correct exposure/etc, and presses Enter. {{{iexp}}} will wait for setup_complete event to start exposure * TO sets up on target saving ACQ images throughout * Once finished, TO clicks "Setup complete" then iexp starts immediately * iexp * retract ACQ (will be done by OTT) * start saving images on probes and wfs's (will be done by OTT) * start exposure * at the end of exposure iexp * stop saving probes images (will be done by OTT) * set exp time for wavefront sensors to 5s (will be done by OTT) * cancel setup * insert ACQ (will be done by OTT) At this point system is ready for new commands from {{{target_setup}}} Observation and exposure determine the directories under which the data are grouped. An example output directory structure might be {{{ /hetdata/data/NWD/lrs2/lrs20000023/exp01/ /exp02/ }}} So here two exposures are grouped under the observation number 23 taken on 20160309. The combination of observation/exposure must be unique for the UTC date. The agreed upon types are "sci", "cmp", "zro", "flt", "drk", "tst". To stop and readout an exposure use the command {{{syscmd -l 'abort_exposure()'}}} [[br]] == Doing Parallel Science with LRS as primary == #parallel To get help on this command try {{{ vlexp --help }}} for an example with the LRS2 as primary {{{ vlexp -i lrs2 -t sci -texp 600 -pobj Myobject2 -B }}} NOTE: There will be no dithering for this type of exposure. To stop and readout both the LRS2 and VIRUS do {{{ syscmd -l 'abort_exposure()' }}} and {{{ syscmd -V 'abort_exposure()' }}} quickly back to back. [[br]] == End of Night Procedures == #park Use {{{ipvg on -V}}} to turn on the LRS2 VG and LRS2 IP (and V309 IP) and look at the LRS2 log to make sure that the VG connected. Wait 10 minutes and then take a pressure reading and record that into the RA logs. The APC tab in the TCS GUI should look like the following: [[Image(APCLRS2endofnight.png)]] At the present time all data transfers are automated and no interaction from the RA is required. [[br]] == Troubleshooting == #PR === Checking Hardware Status === {{{ #!div style="font-size: 80%" {{{#!bash syscmd -l 'get_hardware_status( update_state="true" )' }}} }}} The status may be queried mid-exposure by omitting the update_state parameter. === Typical problems === * For a constant value in an amp you only need to restart the LRS2 server (service), without cycling power to anything. * For a readout that looks like hash or huge noise you need to restart the server (service) and the controller (using APC tab in TCS GUI) * For repeating data you will need to restart the controllers and muxes * For an error in the /var/log/tcs_logs/lrs2/lrs2_server.log that includes an error with "MUX"; you need to restart the server (service), controller (APC tab) and mux (APC tab). === Restarting the Controllers and Muxes === When CAMRA crashes, it is likely due to a controller going offline. That will not be seen in the log messages displayed from the monitor and, in the case described here, if it happens in the middle of an exposure it can leave a connected client script hanging waiting for the response to a command (i.e. expose, get_ccd_temp, etc). The *only* place you can see what is going on in this case is by tailing the /var/log/tcs_logs/lrs2/lrs2_server.log file on lrs2. You will see that the log ends with a string that includes "CArc" and "exception" in the case of a hardware failure. When there is a problem with the hardware, there should be no attempt to bring lrs2 back up without power cycling the controllers. I don't think this has to include the muxes but here is a prescription for how to restart lrs2 from scratch: 1. From the RA launcher choose: '''Operations''' and '''Camra Servers''' and '''LRS2 Server''' select '''Stop'''[[BR]] 2. On the APC GUI Power off hardware LRS2 controller [[BR]] 3. Turn off the '''LRS2MUX''' on the APC GUI [[BR]] 4. On the APC GUI Power on LRS2 controller [[BR]] [[Image(LRS2TDK.png)]] 5. Wait 30 seconds then on the APC GUI Power on LRS2MUX [[BR]] [[Image(LRS2Mux.png)]] 6. From the RA launcher choose: '''Operations''' and '''Camra Servers''' and '''LRS2 Server''' select '''Restart'''. 7. Watch the /var/log/tcs_logs/lrs2/lrs2_server.log for messages about Accepting commands {{{ tail -f /var/log/tcs_logs/lrs2/lrs2_server.log }}} [[BR]] 8. then take a test exposure {{{ cal lrs2 -b 1 -d 0 -o 5555 }}} 9. Check for 0 bias frames with a test lrs2 exposure === Data transfer problem, data left on ramdisk? === - check that Lrs2 DataTransfer service are running (through RA launcher) [[br]] If it is stopped, Restart it! - if data still remains on the RAMdisk, then: - as hetdex on lrs2 execute {{{dt_fix lrs2}}} this will start moving data off the ramdisk but might take 10 minutes. If the ramdisk fills up (probably because pivot is not running) then try the following: - stop lrs2Server - wait 1 minute - start lrs2Pivot - check the lrs2_pivot log to make sure it is running properly (use RAlauncher) - start lrs2Server - as hetdex on lrs2 execute {{{dt_fix lrs2}}} this will start moving data off the ramdisk but might take 10 minutes. [[br]] == Get help on ldas command == * The syscmd mode can be used to display help information on ldas commands. Here are some examples: {{{ syscmd -l 'help("abort_exposure")' syscmd -l 'help("wait_for_readout")' syscmd -l 'help("wait_for_write")' syscmd -l 'help("wait_for_shutter_close")' }}} * Suppose you only remember part of the command name: {{{ syscmd -l 'help()' | grep wait }}} * Suppose you want to be notified when the shutter closes: {{{ syscmd -l 'wait_for_shutter_close(observation=8001, exposure=1 )' beep beep beep }}} The command will not return until the shutter closes. at which point in the example above you'll hear three beeps. === Hardware Changes === If one of the electronics day staff changes a controller a change must be made to the configuration files. For both LRS2 and VIRUS this must be done by either Herman or Jim so call them at home at any hour needed to get the instrument running. === Additional Help === Help for the complete set of CAMRA commands can be obtained by issuing a 'help()' command to the CAMRA subsystem. {{{ #!div style="font-size: 80%" {{{#!bash syscmd -l 'help()' }}} }}} [[br]] Need to add: * -BR option target_setup for LRS2 SPC and science targets * common practice of not waiting and checking * lds description and svn location * Steve O's lrs2stars code