| Version 253 (modified by stevenj, 22 months ago) (diff) |
|---|
HPF
You can find the HPF team's website with their manual at http://psuastro.github.io/HPF/Operators-Manual/ and another useful site is their exposure time calculator
HPF Observing
Suggested terminals setup
Easiest method to start up RA screen session is to
- ssh -X hpf@hpf ( and make the window pretty large )
- run rasession in the SSH session
This will bring up the usual four-terminal screen session that the RAs have used. If you find rasession dead or in unusable state here is the fix:
- ssh hpf@hpf
- run command screen -X -S RA quit
- run command rasession
Most important note: never run exit or press ctrl-C/ctrl-D in these screens as it is likely to crash the HPF instrument. Disconnect from this by closing the xterm/gnome-terminal window externally - do not type "exit" or anything like that, ever.
The layout is shown below along with the commands used to connect to each screen (note top left is not connected to any screen session). More here about HPF Screens
|----------------------|---------------------| | - HPF terminal - | - HET client - | | ssh -X hpf@hpf | ssh -X hpf@hpf | | | screen -x TIMS | |--------------------------------------------| | - HPF monitor - | - HPF Scipts - | | ssh -X hpf@hpf |ssh -X hpf@hpf | | screen -x hpf |screen -x sc | |----------------------|---------------------|
Further rasession troubleshooting tips can be found here.
In the rare event that these screens sessions are not running (or you accidentally "exit"ed out of the screen session), you can create them with the following command, e.g., for the Scripts screen:
screen -S scripts
and then connect to them as usual.
If you have a problem where the screen looks badly sized, like this:
you can press ctrl-a F to "fit" the screen to the current window. (note that is a capital F, so the sequence is Ctrl-a Shift-F)
Instrument Control
Here are the old instrument control commands. Many of which are valid but we have new wrappers that are easier to use and more elegant.
To get to the HPF machine: ssh -X hpf@hpf (the password is on the white board in the control room)
We can communicate with HPF via screens; to see more about screens see our Screens Summary Page. To bring up the HPF monitor use:
(HPFpython) [hpf@hpfserver ~]$ screen -x hpf
That screen is simply for monitoring the state of the instrument. To end the screen session close the xterm/tab or type <ctrl>-a, d. Do not type a <ctrl> d
You may want another ssh window to make commands to the instrument ssh -X hpf@hpf . In this window you will control the lamps, the comparison lamp shutters, exposure times etc... grab the scripts screen session screen -x scripts session so that any failures can be diagnosed by the HPF team.
Data is stored in /HPFData/LinuxGigE/YYYYMMDD/hpf from other machines, e.g. mcs, or /home/hpf/HPFData/LinuxGigE/YYYYMMDD/hpf on the hpf machine, and copied to /hetdata/data/YYYYMMMDD/hpf/ as well. Scripts for changing the instrument settings are located in /home/hpf/Scripts/InstConfig
Controlling the Laser Frequency Comb (lfc) and etalon lamps
As of 2020 this has all been automated into iexp and into cal but these notes are retained. For most modern usage, this will help:
[stevenj@zeus ~]$ cal hpf -h
usage: hpfcal [-h] [-lfc [N] | -et [N]] [-et_warmup] [-et_off]
[-evAC | -evA | -evC | -evAC_nolfc | -evA_nolfc | -mn | -mn_nolfc]
[-mode {direct,hpf}] [--dry-run]
HPF calibrations
optional arguments:
-h, --help show this help message and exit
-lfc [N] Run <N> LFC calibrations. Default is 1
-et [N] Run <N> Etalon calibrations. Default is 1
-et_warmup Turn on and warmup UNeS Etalon lamp
-et_off Turn off UNeS Etalon lamp
-evAC Run evening AC calibrations.
-evA Run evening A calibrations.
-evC Run evening C calibrations.
-evAC_nolfc Run evening AC calibrations. LFC is down.
-evA_nolfc Run evening A calibrations. LFC is down.
-mn Run morning calibrations.
-mn_nolfc Run morning calibrations. LFC is down.
-mode {direct,hpf} Operation mode. "direct" or "hpf". Default is "hpf"
--dry-run Print actions without executing commands on the hardware
For more background, keep reading...
Some targets, usually the brighter ones and the precision radial velocity targets, will want to use the lfc to get the best possible radial velocities. The lfc remains on at all times but to include it in the observation you need to open its internal shutter. To turn it off and on use the following commands:
lfc_open
lfc_close
In some cases, the laser comb may be down and we must use the following:
etalon_open
etalon_close
Note that each of these command sets are executed in the Scripts window. The open command will usually take a few seconds to complete and you will see a variety of messages coming to the window.
After PartA cals are done, the etalon lamp is turned off. It can be turned back on with: ./hpf_TurnOn.sh Etalon
That takes 10 minutes to warm up.
Then, throughout the nights, I try to take the etalon cals at least once an hour: ./hpf_takeEtalonHRCalExposure.sh 3
Doing 3 of those takes about 9 minutes. You can run: ./hpf_takeEtalonHRCalExposure.sh 1 to run a single ramp (takes 3 minutes).
For targets which request LFC, I think our approach right now is to:
-keep cal_closed
-run etalon cals before observing the target (while slewing/setting up)
-run etalon cals after the target
At the end of the night turn off the Etalon, or let the morning cals do it for you ./hpf_TurnOff.sh Etalon
Low-level observing commands
The command to execute any basic observation is hpframp targetname #reads_per_ramp #ramps [obs#]. Each read is 10.5 seconds so a 30 read observation would be 315 seconds. There is a 2 read reset at the beginning of the exposure that adds to the overhead. You can include a optional observing number, but if the software has not been restarted and you do not include this observing number it will use the next observing number. If you restart the software the count starts again at 1 and the next target will be included in directory 0001 even if there is data in that directory, however, due to the time stamp no data will be overwritten.
Example:
hpframp AD_Leo 30 2
You can set the type to Eng so that they are not science exposures and sent to the TACC by doing a tims detect pyhxrg:SetObstype:Eng
Log information on the HPF machine
/home/hpf/HPFics/log
HPF logs are not kept indefinitely, but some recent logs exist in the above path. There are logs for each system(?)
- Calib
- Detect
- Enviro
- Het
- tims_cli
- tims_server
Science observing
Our scripts run all of our science/engineering frames and cals in the screen -x scripts screen session so that any failures can be diagnosed by the HPF team.
Currently iexp starts our HPF exposures for us, but these notes are still useful if you ever need to start exposures manually. The command (as hpf@hpf) to execute any basic observation is hpfseconds targetname exposuretime #exposures [obs#]. If the target name has a space we should use double quotes around the name, e.g. "please do not use spaces in target names" The exposure time is in seconds. You can include a optional observing number, but if the software has not been restarted and you do not include this observing number it will use the next observing number. If you restart the software the count starts again at 1 and the next target will be included in directory 0001 even if there is data in that directory, however, due to the time stamp no data will be overwritten.
Examples:
hpfseconds GJ_3470 315 5 - 5 exposures of 315 seconds each.
hpfseconds GJ_3470 315 5 100 - 5 exposures of 315 seconds each to be saved in directory 0100.
You can look at any spectrum using showhpfspec YYYYMMDD obs#. If you leave off the date and observation number it will show the last exposure.
Examples:
showhpfspec
showhpfspec 20180428 2
You can also look at slope images which have sqrt(N) less noise than the "showhpfspec" quicklook images. Use commands like:
(HPFpython) [hpf@hpfserver ~]$ createslopehpf 20200619 0002
Aborting normal science exposures
An exposure in progress can not be stopped. Beware.
To stop after the current exposure (before the next ramp/exposure starts) you will need another hpf xterm (i.e., ssh hpf@hpf and run the command:
sc
If you need to stop one of the observing scripts, like PartA, then try a <ctrl>-C in the Scripts screen session ("screen:sc"). The scripts are smart enough to catch the <ctrl>-C and it will stop the script at the end of the current exposure (which you can monitor from the HPF monitor screen) you should be able to resume working with HPF.
Standard stars
This is now automated, but the details are retained below:
For telluric standards you should not observe anything brighter than V=4.5 (our lists have been culled to remove these and emission line objects). If you find that any telluric is in a visual binary with a brighter star please please feel free to remove that from the list and let astronomer@… know so that we can edit the online list.
For exposure times please use:
- 315 sec exposure (30 reads) for anything fainter than V = 5.8
- 105 sec exposure (10 reads) for anything between 4.5 and 5.8 in V
Be sure to edit the night report to change the Data Type to "telluric"
NB Telluric standards must be taken with LFC closed
Quick-look data and SNR
Starting in late March 2020, Goldilocks (Greg Zeimann's HPF data reduction pipeline) has been running in near real time. Details are here: https://github.com/grzeimann/Goldilocks
Goldilocks runs automatically on each exposure after the next exposure starts. It saves quick-look HPF data to /hetdata/data/<UTDATE>/hpf/Goldilocks/. Among more quantitative output files, it also saves a PNG snapshot of order 19 (~1um). Order 19 is used to determine the continuum signal-to-noise ratio, which gets saved to the NR as "SNR", asynchronously.
You can run Goldilocks manually on VDAS with something like: python ~astronomer/Goldilocks/mountain_hpf_reduction.py 20200402 7 for the 0058 observation taken on UT date 20200402. This can take a few minutes to run, and will produce three files in whatever directory you run it: Goldilocks_20200402T023958_v1.0_0008.fits Goldilocks_20200402T023958_v1.0_0008.spectra.fits Goldilocks_20200402T023958_v1.0_0008.spectra.png
Monitoring HPF exposures for saturation
Starting in 2022 we have a new tool to monitor real-time predictions for the final counts expected in an exposure in progress: hpf_sat_check.py It runs on the workstations and produces output like the example below where it is run during a twilight exposure. I like to leave it running in a terminal all night while I am observing. When it detects a new HPF exposure in progress it measures each read's FITS file to calculate:
- O18flux - the raw pixel counts in a small region near the center of Order 18
- pedestal - the raw pixel counts in an unilluminated area of the detector
- pixcnt - the difference of O18flux - pedestal
- Predicted total - extrapolated total counts expected, where 65k is saturation
[stevenj@zeus ~]$ hpf_sat_check.py
checking HPF data for saturation in this UT date folder:20220108
detected latest HPF exposure: 20220108 obs #0001 requesting 28 total reads
Readnum / Nreads O18flux pedestal pixcnt Predicted total Object
1 / 28 18934.0 18604.0 845.0 37200.8 twilight
2 / 28 19728.0 18617.0 1373.0 37534.2 twilight
3 / 28 20807.0 18628.0 2179.0 38965.3 twilight
4 / 28 21572.0 18637.0 2935.0 39182.0 twilight
5 / 28 22354.0 18635.0 3719.0 39461.4 twilight
6 / 28 23085.0 18668.0 4417.0 39280.7 twilight
7 / 28 23848.0 18674.0 5174.0 39370.0 twilight
8 / 28 24565.0 18683.0 5882.0 39270.0 twilight
9 / 28 25289.0 18707.0 6582.0 39184.3 twilight
10 / 28 25983.0 18711.0 7272.0 39072.6 twilight
11 / 28 26645.0 18704.0 7941.0 38917.5 twilight
12 / 28 27302.0 18685.0 8617.0 38791.3 twilight
13 / 28 27935.0 18687.0 9248.0 38605.8 twilight
14 / 28 28585.0 18728.0 9857.0 38442.0 twilight
15 / 28 29182.0 18685.0 10497.0 38279.4 twilight
16 / 28 29790.0 18720.0 11070.0 38092.5 twilight
17 / 28 30377.0 18726.0 11651.0 37915.9 twilight
18 / 28 30961.0 18725.0 12236.0 37758.8 twilight
19 / 28 31519.0 18732.0 12787.0 37576.0 twilight
20 / 28 32079.0 18743.0 13336.0 37413.4 twilight
21 / 28 32586.0 18730.0 13856.0 37204.7 twilight
22 / 28 33082.0 18719.0 14363.0 36999.2 twilight
23 / 28 33587.0 18738.0 14849.0 36815.0 twilight
24 / 28 34030.0 18747.0 15283.0 36577.2 twilight
25 / 28 34476.0 18761.0 15715.0 36361.8 twilight
26 / 28 34925.0 18742.0 16183.0 36169.8 twilight
27 / 28 35368.0 18775.0 16593.0 35982.6 twilight
28 / 28 35735.0 18748.0 16987.0 35735.0 twilight
If it predicts total counts above 55,000 it will play an announcement to warn the RA on duty that the HPF exposure might be close to saturation and further attention is required.
As an example, here is what saturation looks like with a too-long exposure on a too-bright star (20211201, Observation 26):
And here is an example of the type of persistence we see in a dark exposure after a saturation event - this time the saturation was the LFC. Note the faint ghostly persistent signal from the LFC despite this being a dark exposure:
Restarting TIMS client connection the HET, done daily in OPS
As of 2021, this is accomplished with the RA OPS webpage interface (see this section for more detail). The following manual descriptions are retained in case they are useful:
- Find the 'HET Client' window in the 'TIMS' Screen session by doing a CTRL-a n (multiple times, until the window title says 'HET Client'). If a TIMS screen session is not open open one in an xterm with (HPFpython) [hpf@hpfserver ~]$ screen -x TIMS
- In the HPF Scripts window or a new hpf xterm execute : tims het shutdown . This will stop the dictionary python stuff in the 'HET Client' window.
- If you don't have a command prompt in the HET Client window, CTRL-c the client. If you do end up doing this you might want to run another tims het shutdown just to make sure everything is down.
- Run the command 'timshet' in the HET Client screen window timshet
Note that this is the only TIMS client startup command that will not run with the normal startup syntax (e.g. python -m TIMS.clients.tims_het). This is because the library environment needs to be changed to access the het system libraries. We only want to do this for the HET client, so I've wrapped the normal startup command in a bash script that lives in ~/bin/ .
HPF twilight sky flats (evening)
These are not required every night, but are desired when possible. Only take HPF twilights if the sky is completely clear. Brightly lit clouds can cause HPF to saturate.
In mid-2021, the old approach of "2 minutes after sunset" seemed to consistently yield too-faint spectra. We now recommend starting the HPF twilight exactly at the sunset time listed in the Night Report.
Run htwi on zeus, exactly at sunset.
Monitor closely on hpf@hpf either with showhpfspec (slower) or ds9h (faster). If saturation is imminent, close the PFIP shutter manually
HPF Darks and checking for saturation
If you need to take a dark exposure (i.e., to check for evidence of persistence), this is the procedure (in the CalScripts folder). First, make sure PFIP shutter is closed. Then, run this command to take a 5 minute (30 read) dark exposure:
./hpf_setup_ToTakeDark.sh
./hpf_expose.sh Dark 30 1 &
Then you can view the slope image (higher S/N than showhpfspec quicklook) for this with:
createslopehpf 20200619 0002
Note that taking dark exposures does not improve persistence any faster, just shows you where it is.
HPF cals in the evening and morning
Currently we run our science frames and cals in the screen -x scripts session so that any failures can be diagnosed by the HPF team.
Notes on special commands controlling cals:
- When you want to pause any script before the next new set of hpf exposures start:
- pausehpf: pause what is currently running, before the next HPF exposure sequence starts
- playhpf: continue the paused exposure.
- When you want to skip the rest of the exposure in the current ongoing multiple frame exposure sequence.
- sc: This command is to discard all the remaining exposure in a multiple exposure command. It is meant for use in a normal science observation. If used during a cal script, it will discard the remaining exposures inside a set of ongoing exposures, and continue on to the next set of cals exposures. Note: It will not stop the rest of the cal script.
- When you want to abort the morning or evening cals, Ctrl+C is the way to abort the script. Ctrl+C will trigger the wrap up scripts, and it should cleanly abort the cals scripts. (With the caveat that any ongoing exposure which has already started will not stop until that is completed.)
- When you want to pause the cals before the next usage which requires FCU unit.
- pausecal : Is the command to pause the cals before it stars a next set of observations before using the FCU head. Historically this command was made to pause the cals temporally for anybody on side to switch on the lights in the dome, or use fcu head for something else, etc. Note that it may be several minutes after issuing the "pausecal" command before it takes effect (i.e., once it gets to the next part of the script which uses the FCU head) and it is safe to, say, turn on the dome lights.
- playcal : This command will resume the cal scripts by taking control of FCU again.
Evening Calibrations
The evening 'A' calibration sequence includes the LFC and Etalon frames (and flats) that are necessary to start the nightly drift calibration. The 'C' calibrations contain the UNe lamps, which are only present as an emergency backup source in case the LFC is offline. Taking the 'A' sequence is critical; skipping the 'C' sequence if you are crunched for time is OK; that sequence has been timed to fit within a typical stack time.
- Nightly evening cals: PartA (43 minutes) and PartC (45 minutes).
- Typically the Ops RA will start the PartA (sometimes A+C) cals in the afternoon, with:
cd /home/hpf/Scripts/InstConfig/CalScripts ./hpf_evening_cals_PartA.sh
- If you don't have the 44m minutes for the full PartA cals just run the
./matt_shortflats.sh 22 minutes
or
./matt_wave.sh 14 minutes
depending on how long you have before sunset. The most critical cals are the wavelength (wave) cals.
- If you don't have the 44m minutes for the full PartA cals just run the
- If necessary, the Part "C" cals can be run during other activities, as they are purely internal cals. As long as we don't get in a situation where the dome lights are on and the PFIP shutter is open, it's ok. Part "C" can be run during VIRUS/LRS2 twilight spectra, during VIRUS/LRS2 lamp cals, during stacking, or even during LRS2/VIRUS science observations. You can run the "C" cals (43min) with:
cd /home/hpf/Scripts/InstConfig/CalScripts ./hpf_evening_cals_PartC.sh
- You can tell if the cals have finished by confirming that the command prompt has returned in the HPF command window. If the job is still in execution, we'll see a sequence of integers being written to this window. Also, if the script is still executing, we'll see update message appearing in the HPF monitor window.
- Combining A+C cals takes 1h 28m, and is uaully initiated at the end of the afternoon OPS if dome stays closed.
- IMPORTANT NOTE on order of A,C cals:
- Part C cals must be run AFTER Part A for lamp warmup/shutdown reasons. When Part A is run, it leaves the UNe lamp on. This lamp is needed by Part C, and Part C turns it off when it finishes. However, if Part C is not run after Part A, you must turn off the lamp manually with:
/home/hpf/Scripts/InstConfig/CalScripts/hpf_TurnOff.sh UNeS
- Similarly, if you are running Part C without first running Part A, you must turn on the UNe lamp via:
/home/hpf/Scripts/InstConfig/CalScripts/hpf_TurnOn.sh UNeS sleep 600
- Part C cals must be run AFTER Part A for lamp warmup/shutdown reasons. When Part A is run, it leaves the UNe lamp on. This lamp is needed by Part C, and Part C turns it off when it finishes. However, if Part C is not run after Part A, you must turn off the lamp manually with:
- Additional LFC wavelength cals throughout the night:
We have an additional script (hpf_takeLFCHRCalExposure.sh) which that will take a LFC exposure through the calibration fiber (~2 minutes to complete). This should not interfere with any other telescope operations, as it is entirely internal to HPF. If you find yourself in a situation similar to last night, you can run this stand alone script a few times while a different primary operation is happening (e.g. LRS2 or VIRUS exposure, slew, or other telescope setup). That will at least provide a few LFC reference points to tie the drift calculation to.
The script accepts an optional argument - N - where N is the number of ramps you wish to take. e.g. ./hpf_takeLFCHRCalExposure.sh 3
You can also use the following on Zeus to run this: cal hpf -l 3
Morning
- The night RA will start a morning calibration for HPF at the end of the night. This will take about 45m to complete the portion requiring the FCUhead and a dark dome. An additional 1h12m with the pfip shutter closed is required to complete the full calibration script run.
[hpf@hpfserver CalScripts] ./hpf_morning_cals.sh
If you took data for UT21-1-002 which require extremely high S/N for their targets, use the alternative morning cals script which acquires an extra 15 quartz lamps and takes 20 minutes longer. It requires 80min with FCUhead in place and dark dome, and 90min with the PFIP shutter closed:
[hpf@hpfserver CalScripts] ./hpf_morning_cals_15xQuartz.sh
When the morning cal script is run it produces a text file to list the expected completion times like:
[stevenj@zeus ~]$ cat /home/mcs/astronomer/Desktop/morning.txt 15:13 UT: HPF morning cals started 16:13 UT: HPF finished with dark dome 17:23 UT: HPF finished with PFIP shutter
As of January 2020, you can also specify that the "Dome lights off" message should remain on the sign outside the door to the dome, in case you have scheduled more (non-HPF) calibrations to run in the morning (e.g., LRS2 or VIRUS cals which did not get taken during the night). To leave the cal sign on, use:
[hpf@hpfserver CalScripts] ./hpf_morning_cals.sh leave_signs_on
Then run the usual cal script on zeus/vdas with the -eon flag so that it turns off the signs after it has finished.
This page contains the latest data on measuring the time each calibration dataset requires.
Stopping Cals
If you need to stop a cal script then you can type <ctrl>-C in the scripts window which will stop the script but will NOT stop the current exposure. Watch the HPF monitor screen to see when the instrument stops the exposure and goes idle.
Trouble shooting and restarting
Arrow-ing up to see what happened
If you missed something that has scrolled off the top of the screen and you want to see what it was you can do a <ctrl>-a followed by an <esc> to get into "copy" mode. Then use the arrows to move the cursor up to get to the message of interest. To leaves copy mode hit two <esc> in a row. To see the full set of screens commands please see our Screens Summary Page.
Observing if the LFC is down
If the LFC is down, we usually go into "low precision" mode for HPF. PIs are required to specify Y or N to RVHIGHPREC as a keyword in their TSL files. If we do not have the LFC or have had a thermal stability problem, we should not observe targets with RVHIGHPREC=Y that night. If the LFC is unavailable we must use the Etalon lamp as a calibrator instead. Most of this is automated in our iexp and cal scripts, and will read the status of the LFC from the RA Night Report drop-down menu.
- Calibrations without LFC
- evening cals: cal hpf -evAC_nolfc
- or use these manual commands in the Scripts screen:
- ./hpf_evening_cals_PartA_withoutLFC.sh 53 minutes'''
- ./hpf_evening_cals_PartC.sh no change
- ./hpf_evening_cals_PartAC_withoutLFC.sh xx min
- morning cals: cal hpf -mn_nolfc
- or use these manual commands in the Scripts screen:
- ./hpf_morning_cals_withoutLFC.sh 34 min dark dome; 1 h 15 min PFIP shutter closed
- evening cals: cal hpf -evAC_nolfc
NOTE: you must turn on the Etalon lamp since the A+C evening cals automatically power it down. You can do that on zeus with:
cal hpf -et_warmup
or by running this in the HPF scripts screen:
./hpf_TurnOn.sh Etalon
Fits header cards/keywords and metadata issues
If you need to manually change the FITS header keywords on existing data (i.e., if the metadata was missing), follow this example to edit the raw data and then inform Sergey and Greg to re-copy that night's data.
1. ssh hpf@hpf 2. cd /home/hpf/HPFData/LinuxGigE/20201014/hpf/0010/fits/ 3. /mnt/het/hetdex/bin/headfits -m -k QIDX -t int -v 6709 *.fits 4. /mnt/het/hetdex/bin/headfits -m -k QPROG -v UT20-1-999 *.fits 5. /mnt/het/hetdex/bin/headfits -m -k QRA -v '01:02:03.40' *.fits 6. /mnt/het/hetdex/bin/headfits -m -k QDEC -v '+45:34:23.1' *.fits 7. /mnt/het/hetdex/bin/headfits -m -k QEQUINOX -t float -v 2000 *.fits 8. /mnt/het/hetdex/bin/headfits -m -k QPMRA -t float -v 308.227 *.fits 9. /mnt/het/hetdex/bin/headfits -m -k QPMDEC -t float -v -608.695 *.fits 10. /mnt/het/hetdex/bin/headfits -m -k QEPOCH -t float -v 2015.5 *.fits We must fix all exposures and all values: QIDX, QPROG, QRA, QDEC, QEQUNIOX, QPMRA, QPMDEC, QEPOCH You can extract these values from the queue database with this command if you want, replacing the IDX number for the target you want: [stevenj@zeus ~]$ mysql -hnossy htopx -e "select idx,plan,ra,declination,equinox,pmra,pmdec,pmepoch from plandb_latest_cur where idx=4000"
Note that HPF data are transferred to PSU each day at 16:00 UT, so as long as these errors are corrected before then we can recover from this problem. If we do not identify it in time, it is likely that that night's data will have to be rejected and re-observed
*Please add a comment in the RA Night Report to the observations where you manually added the Q* metadata to be sure that the PI is aware of this manual intervention. While it can successfully reproduce all of the missing Q* metadata, we need to inform the PIs in case we have not fully and accurately replaced the Q* metadata.
LFC OSA power-cycling
While it is primarily a TO duty (in early 2021), the instructions for power-cycling the Optical Spectrum Analyzer on the LFC are here: HetProcedures/RA/LFC
Restart Client when it stop working
If see 'Error' in hpf session of client is not operation normally you can try to restart client
If you get the QSocketNotifier: in the detector client window you can try restarting it but if it comes back then the instrument is down and you should call Chad.
Here is restarting procedure: Starting and Stopping the Client
Problem with LFC
If part A script hang on message:
"WARNING: Setting LFC flux to High .." or "Not yet set to high mode"
means absence of communication to LFC computer. It possible to abort script with Ctrl-C and email Chad, Joe and Connor for help.
Until problem fixed it is best to assume that LFC is down and use Etalon
The etalon is brighter than the LFC so if you take an exposure longer than 150 seconds with the etalon open you will likely saturate the detector. Thus, do not use the etalon simultaneously during an observation unless the exposures are less than 150 seconds. Instead, taken an single etalon before and after each HPF observation.
Detector client
If you accidentally send a "ctrl-C" in the "HPF Detect Client" screen, you should immediately press the up-arrow and re-run the command to restart the detector client. This restart will reset the observation number, so you will also need to reset the obsnum to the most recent value with tims Detect pyhxrg:SetObsnum:NNN command, where NNN is the last object number that was taken (it does a +1 at the beginning of the exposure sequence).
Also you should notify Chad because this kind of detector client restart can cause a small temperature transient.
Fixing the calibration connection
If the etalon turn off command hangs or any of the cal lamps try the following:
tims Calib shutdown in any hpf window with a prompt
In the TIMS HET window do a cntrl-a n about five times to move to screen 2 which is the TIMS Calib screen. Note that we must hit cntl-a then hit the n (for next srccen). This full sequence is repeated to go to the next screen. The screen we usually operate in has the heading name "Screen 4: HPF HET Client". To fix the probelm of Q-data missing from the HPF headers (for example) we need the screen that is labeled "Screen 3: HPF TCS Relay". The screen we start from is named "Screen 4: HPF HET Client". The screen we want to end up at is It should have a prompt. If it does not you may need to do a few cntrl-c to kill any hanging processes.
The TCS Relay screen should look something like this:
In the TIMS Calib screen you should do an up arrow and see the python -m TIMS.clients.tims_calib command which restarted the Calib client. Wait a few minutes for it to finish.
Take a quick lfc image and perhaps a quick sky exposure to make sure all of the shutters and baffles are working.
Fixing the loss of tcs info in hpf headers
See our main page here about fixing TCS data for all instruments
Listening to the fiber scrambler
As of Feb 2019, you can use the "RA" knob on the audio box (under the UT clock) to control the volume of the audio feed from HPF calibration enclosure which plays on the speaker under the RA desk. You can also connect to this audio feed with VLC media player on MCS via:
cvlc rtp://@:4444 --sout-rtp-caching 40
(drop the leading "c" if you want a UI)
Or use web browser http://192.168.66.90/xstream
Problem with RASESSION
In case if you have a problem with "rasession" you can kill it and recreate.
- ssh hpf@hpf
- run command screen -X -S RA quit
- run command rasession
Stopping and starting the client (rarely used!)
restarting the the HPF client
Note that the HPF client is restarted automatically every day 16:45 UT just after the LN2 refill. The act of restarting the HPF makes a small change to the instrument temperature which impacts RV stability.
restart_detectorclient is called daily as part of the crontab at 16:45 UT to restart the HPF client daily. It can also be called on demand (but we very rarely do this).
There are two executable scripts:
1) start_hpfdetectorInfloop.sh - this is running in the 'hpf' screen session, and takes care of automatically restarting the client after it is shutdown
2) restart_detectorclient - can be called from any hpf terminal (but preferably the 'scripts' session. This will trigger a shutdown sequence. After the shutdown, start_hpfdetectorInfloop.sh will automatically take care of bringing things back up.
Check that the observation number is correct with pyhxrg:GetObsnum
if it is not then use the following command to set it: tims Detect pyhxrg:SetObsnum:nn where nn is the number from previous GetObsnum command
Dealing with problems with the Screen session getting stuck or frozen
0) Remember that screen can be blocking if it gets into its 'copy mode' (ctrl-a ESC), and probably several other ways to prompt this state too. If it is blocking you loose the ability to communicate with the process running inside of it.
1) Pause any scripts that are running - you don't want to get things running and then have a script send a bunch of queued up commands in quick succession before you've been able to check functionality.
2) Try to connect to the screen session:
screen -x hpf
3) If this loads like normal and you see the detector client running, then the problem is not the screen session. Proceed with troubleshooting re. detector crash/non-responsiveness
4) If you do not get a normal detector client, but instead just see a cursor and the process looks hung, try giving it *one* ctrl-c. This may unstick the screen session by killing whatever process in it had hung. Then the detector client software should show up like normal.
How to stop the client (very rarely used!!)
If you need to completely power off the detector and prevent a restart, there are two options:
- tims detect pyhxrg:GetObsnum
- touch ~/stopdetectorloop
- tims detect pyhxrg:PowerDown
- tims detect shutdown
This will create a file in $HOME called stopdetectorloop. If that file exists, the infinite loop will not continue following the normal powerdown/shutdown sequence.
- There is a 10 s pause between completing the power down and auto-
restarting. You can press CTRL+c in the 'hpf' screen session during that time to abort the script.
You should see something like
Configured HxRG Sending Cli15380: Detect Cli15380 PowerDown:OK Resetting Asic for JADE1 ... Success Powering Down Analog Supply Voltages: VDDA ... Success Powering Down Digital Supply Voltages: VDD3p3 ... Success VDD2p5 ... Success VDDIO ... Success VSSIO ... Success Powering Down Voltage Regulators: VDDIO ... Success VDD2p5 ... Success VDD2p5 ... Success Powering Down Analog Main: 5V ... Success De-Initializing Hal Powered Down.
Shutting down this client.
Stopping scheduled loops:
...lakeshore336
Shutting down subclients
...pyhxrg
......Failed
...lakeshore336
......Success
Sending Cli16420: Detect Cli1642 * tims detect pyhxrg:SetObstype:Sci0 message:Detect_shuttingdown
Shutting down...
Connection lost to: lakeshore336
Reason : [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.
]
Closing connection to server
Client stopped.
When you restart the system you should check to see if it has the correct observation number (assuming you have not rolled through 0 UT).
Attachments (4)
- hpf_screen_size.jpg (48.7 KB) - added by stevenj 2 years ago.
- hpf_tcs_relay.jpg (208.6 KB) - added by stevenj 23 months ago.
- hpf_saturation_20211201_26.png (68.5 KB) - added by stevenj 23 months ago.
- hpf_dark.jpg (168.1 KB) - added by stevenj 23 months ago.
Download all attachments as: .zip
