Creating a system monitor
A monitor can be run from inside the TCS gui or from the command line in htcs, but to avoid cross purposes with TO, run it preferably from /home/mcs/astronomer.
In the following example on is run for the TCS (note the -T) but it excludes the debug commands (note the d):
monitor --verbose -T --log-print --key-filter='log_[^d].*'
In the following example you can get the WFS seeing estimate from pas:
monitor --verbose -p --key-filter='metrology_data' --source='WFS*' | grep '\"time\|seeing\|\.metrology'
Log files
Perhaps more useful than the db are the log files. These can be accessed during the night and include the current night a usually about a week of older logs. There are a number of different logs on different machines:
- TCS logs are located on htcs in /var/log/tcs_logs/subsystem_name/
- for older logs, see /opt/het/hetdex/logs/Mon/Console
- TCS_GUI logs are located in /opt/het/hetdex/logs/console/tcsGui
- VIRUS and LRS logs are located on vdas and lrs2 in /var/log/tcs_logs/instrument_name/
- Htopx logs are located in /data1/nossy/www/html/noss/htopx3/log/htopx.log
The TCS logs are split into the different sub-systems: legacy, pfip, pas, tcs, and tracker. There are several ways one might use these. Note that to run this command you must be logged into htcs (e.g. ssh -X sco@htcs).
- For example to look at the pfip probe motions to check for the "move complete" before making any further moves. This would be done as tail -F tcs_server.log | grep pfip
- One can also look for a "go_next" or "load_trajectory" inside of the tcs_server logs.
- Be sure to be logged into htcs.
One could look at the TCS GUI logs to see what buttons were pushed to get an idea if a problem was a human error or a software problem. The htopx_logs are useful for looking to see when/if targets were incremented.
During the summer of 2021 we began logging in PR5364 the occurrence of problems with moving the WFS2 probe. It was necessary to know the time stamp for when the probe timed out in these failure cases. To find this one case in the enormous pfip log is laborious at best. A simple way to find this in the pfip server log is to use something like:
[sco@htcs ~]$ tail -50 /var/log/tcs_logs/pfip/pfipServer.log |grep 'timed out'
The resulting output will include something like:
2021-10-01T22:10:23.967 ERROR [PFIPHandlers.cpp :process : 23] - handler - MoveProbes handler failed (Probe move timed out)
Nightly Databases
Nightly databases can be found in /opt/het/hetdex/logs/Mon/YYYY/MM/ where YYYY and MM are the year and month. Data files are written at around 18 UT and are off by one UT date. For example /opt/het/hetdex/logs/Mon/2017/05/20170513T180005.db is the database for the night of 20170514.
You can extract a text version of the db with the python code: db2log.py
Plotter
Here is an example of plotting guider1 metrology x vs. y: tcs_plotter --named-route "tcp://192.168.66.99:30000" --plotter-route "" --endpoint-system tcs --system tcs --source Guider1 --key measurement -x 'focal_plane.dx_asec' -y 'focal_plane.dy_asec' --scatter 1024
Here is another example tcs_plotter --named-route "tcp://192.168.66.99:30000" --plotter-route "" --endpoint-system tcs --system tcs --source root --key ra_dec -x 'ra' -y 'dec'
Other useful monitors
Looking at the DMI and WFS data:
monitor --verbose -T --source-filter 'WFS*|DMI' --key-filter measurement
working with the htopx and night report databases
There are many databases here. some of the more useful ones are:
- plandb_latest_cur (this holds the most recent update for any record in the current queue)
- plandb (holds all changes and version so allows you to look at history)
- plandb_latest_new (this is for the new trimester)
- nightreport_ra_targets
- tacs
- ephemerides
- nightreport_ra_programs
It is useful to know what fields can be queried with a mysql query in these databases. To get a list try: mysql -hnossy htopx -e "describe plandb_latest_cur" where the .... is the password
Recent (early 2021) example queries to get a feel for how to access information in the databases
This produces a list of targets from a given night, and some basic details about them:
[astronomer@vdas ~]$ mysql -hnossy noss -e "select \ nightreport_ra_targets_id, target, file, \ start, exp_start, exp, end, program, \ accept FROM nightreport_ra_targets where datatype='sci' \ and target!='parallel' and date='2020-04-20' order by start " +---------------------------+---------------+-----------------+-------+-----------+-------+-------+-------------+--------+ | nightreport_ra_targets_id | target | file | start | exp_start | exp | end | program | accept | +---------------------------+---------------+-----------------+-------+-----------+-------+-------+-------------+--------+ | 335463 | TOI-1728 | 0017 | 02:14 | 02:21 | 969.1 | 02:37 | UT20-2-007 | A | | 335464 | TOI-1728 | 0018 | 02:37 | 02:38 | 969.1 | 02:54 | UT20-2-007 | A | | 335468 | DEXsp1361 | virus0000013_01 | 02:54 | 02:59 | 360 | 03:05 | HET20-2-100 | A | | 335470 | DEXsp1361 | virus0000013_02 | 03:05 | 03:06 | 360 | 03:12 | HET20-2-100 | A | | 335471 | DEXsp1361 | virus0000013_03 | 03:12 | 03:13 | 360 | 03:19 | HET20-2-100 | A | ....
This will show all spectrophotometric and telluric standards observed after 1 Jan 2018:
[astronomer@vdas ~]$ mysql -hnossy noss -e "select \ nightreport_ra_targets_id, queue_target_id, night_idx, \ date, time_add, time_mod, file, target, \ start, exp_start, exp, end, program, datatype, \ setup, accept FROM nightreport_ra_targets \ where (datatype='spc' or datatype='tel') and \ target!='parallel' and date>'2018-01-01' " +---------------------------+-----------------+-----------+------------+---------------------+---------------------+-----------------+-----------------------------+-------+-----------+-------+-------+-------------+----------+--------+--------+ | nightreport_ra_targets_id | queue_target_id | night_idx | date | time_add | time_mod | file | target | start | exp_start | exp | end | program | datatype | setup | accept | +---------------------------+-----------------+-----------+------------+---------------------+---------------------+-----------------+-----------------------------+-------+-----------+-------+-------+-------------+----------+--------+--------+ | 185465 | 0 | 109 | 2018-01-02 | 2018-01-02 10:00:25 | 2018-01-02 10:17:25 | lrs20000017_01 | FEIGE_66 | 09:52 | 10:00 | 90 | 10:02 | | spc | LRS2-B | A | | 185466 | 0 | 110 | 2018-01-02 | 2018-01-02 10:06:55 | 2018-01-02 10:17:55 | lrs20000018_01 | FEIGE_66 | 10:02 | 10:06 | 90 | 10:08 | | spc | LRS2-R | A | | 186081 | 0 | 88 | 2018-01-05 | 2018-01-05 07:23:15 | 2020-10-09 15:35:16 | lrs20000010_01 | G191B2B_066_W | 06:58 | 07:23 | 96.5 | 07:25 | | spc | LRS2 | A | | 186082 | 0 | 89 | 2018-01-05 | 2018-01-05 07:30:18 | 2020-10-09 15:34:59 | lrs20000011_01 | G191B2B_056_W | 07:25 | 07:30 | 97.1 | 07:32 | | spc | LRS2 | H | ....
This will show all fields available for the RA NR database:
[astronomer@vdas ~]$ mysql -hnossy noss -e "describe nightreport_ra_targets"
These queries work on the TO night report, first showing which TO was on duty each night of April 2020:
[astronomer@vdas ~]$ mysql -hnossy nightreport_to -e "select date,operator from nightreport_to WHERE (date>='2020-04-01' AND date<='2020-04-30')" +------------+-------------------+ | date | operator | +------------+-------------------+ | 2020-04-01 | Cassie Crowe | | 2020-04-02 | Cassie Crowe | | 2020-04-03 | Cassie Crowe | ....
Or to show which RA is marked as overlapped in the month of April 2020:
[astronomer@vdas ~]$ mysql -hnossy nightreport_to -e "select date,overlapra from nightreport_to WHERE (date>='2020-04-01' AND date<='2020-04-30')" +------------+-------------------+ | date | overlapra | +------------+-------------------+ | 2020-04-01 | | | 2020-04-02 | | | 2020-04-03 | | | 2020-04-04 | Steven Janowiecki | ....
Or to use the RA night report to display which RA was on duty and whether we were observing high RV precisions HPF targets for 10 nights in April 2021:
[astronomer@vdas ~]$ mysql -hnossy noss -e "select date,ra,conf_hpf_rvhighprec from nightreport_ra where date>='2021-04-11' and date<='2021-04-19'" +------------+-------------------+---------------------+ | date | ra | conf_hpf_rvhighprec | +------------+-------------------+---------------------+ | 2021-04-11 | Sergey Rostopchin | y | | 2021-04-12 | Sergey Rostopchin | y | | 2021-04-13 | Justen Pautzke | n | | 2021-04-14 | Justen Pautzke | y | | 2021-04-15 | Justen Pautzke | y | | 2021-04-16 | Stephen Odewahn | y | | 2021-04-17 | Stephen Odewahn | n | | 2021-04-18 | Stephen Odewahn | y | | 2021-04-19 | Stephen Odewahn | y | +------------+-------------------+---------------------+
It also may be useful to query the real-time probe data (IQ) that is used to create the giq/gtrans/gskyb graphs. Try the following queries:
[stevenj@zeus ~]$ mysql -hnossy noss -e "describe probes_realtime_data"
+--------------+-----------------------------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-----------------------------------------+------+-----+---------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| obsdate | date | NO | MUL | NULL | |
| probe | enum('Guider1','Guider2','WFS1','WFS2') | YES | | NULL | |
| timestamp | datetime(6) | NO | | NULL | |
| iq | float | NO | | -1 | |
| lohobliq | float | NO | | -1 | |
| hobliq | float | NO | | -1 | |
| transparency | float | NO | | -1 | |
| pupil_illum | float | NO | | -1 | |
| skymag | float | NO | | -1 | |
| filter | enum('','Clear','B','g`','i`','r`') | NO | | NULL | |
| guide_corr | float | NO | | -99 | |
+--------------+-----------------------------------------+------+-----+---------+----------------+
[stevenj@zeus ~]$ mysql -hnossy noss -e "select obsdate,timestamp,probe,iq from probes_realtime_data where obsdate='2022-11-08'"
+------------+----------------------------+---------+----------+
| obsdate | timestamp | probe | iq |
+------------+----------------------------+---------+----------+
| 2022-11-08 | 2022-11-07 21:26:43.452605 | Guider2 | 0.676208 |
| 2022-11-08 | 2022-11-07 21:26:43.452605 | Guider2 | -1 |
| 2022-11-08 | 2022-11-07 21:26:43.456097 | Guider1 | 0.604858 |
| 2022-11-08 | 2022-11-07 21:26:43.456097 | Guider1 | -1 |
| 2022-11-08 | 2022-11-07 21:26:49.382614 | Guider1 | 0.605058 |
| 2022-11-08 | 2022-11-07 21:26:49.386115 | Guider2 | 0.817397 |
| 2022-11-08 | 2022-11-07 21:26:49.382614 | Guider1 | -1 |
| 2022-11-08 | 2022-11-07 21:26:49.386115 | Guider2 | -1 |
| 2022-11-08 | 2022-11-07 21:26:54.374506 | WFS2 | 2.16653 |
| 2022-11-08 | 2022-11-07 21:26:59.302576 | WFS1 | 2.2619 |
| 2022-11-08 | 2022-11-07 21:30:02.138808 | Guider1 | 0.598671 |
| 2022-11-08 | 2022-11-07 21:30:02.138808 | Guider1 | -1 |
| 2022-11-08 | 2022-11-07 21:30:02.149524 | Guider2 | -1 |
| 2022-11-08 | 2022-11-07 21:30:08.133083 | Guider2 | 0.723475 |
| 2022-11-08 | 2022-11-07 21:30:08.133083 | Guider2 | -1 |
| 2022-11-08 | 2022-11-07 21:30:08.144753 | Guider1 | 0.598711 |
| 2022-11-08 | 2022-11-07 21:30:08.144753 | Guider1 | -1 |
| 2022-11-08 | 2022-11-07 21:30:13.055517 | WFS2 | 2.21865 |
| 2022-11-08 | 2022-11-07 21:30:14.144439 | Guider1 | 0.598894 |
| 2022-11-08 | 2022-11-07 21:30:14.144439 | Guider1 | -1 |
| 2022-11-08 | 2022-11-07 21:30:14.152561 | Guider2 | -1 |
| 2022-11-08 | 2022-11-07 21:30:14.152561 | Guider2 | 0.627922 |
....
