wiki:HetProcedures/RA/dailychecks

Here are some things that need to be done regularly for science operations.

Currently most of these are done by the Science Operation Manager, although some of that may be flexible in the future as our operational model evolves.


Daily checks

Before the 8:15am morning meeting:

  1. Check the ​NR Reader for last night
    • read PRs, OPS/night comments, respond/reply appropriately
    • also read the night RA's morning email summary
  1. Check the ​time accounting graph for last night
    • check that we have an activity entry at 18-deg evening twilight
    • check that there is no gap between stacking and the first target
    • check that all non-science activity entries have comments (e.g., idle/engineering)
    • check that all targets are consistent with the activity at that time (sci vs eng)
    • check for any un-used time/gaps and correct errors (or email RA if necessary)
    • eliminate "queue holes" that can be rolled into setup if OH remains <= 10min
    • add and increment targets to the -000 ENG program if appropriate to charge time for operational engineering activity entries
    • check for any other weird issues or obvious errors and take appropriate action
    • consider running nrcheck.py on zeus to catch any obvious issues
  1. Check the ​RA Night Report on nossy
    • check that all 'A/B' targets were incremented
    • check that no rejected targets were incremented
    • check for any crazy overheads
    • check that all setups with OH>10min have comments
    • check for any other weird issues or obvious errors
      • missing program IDs or other metadata?
      • acceptance/rejection errors or issues?
    • consider looking at the gcplot output from last night
      • in /data1/archive/gcplot/gcplot_YYYYMMDD_targets.pdf
      • verify all targets were observed within their requested IQ/skysb/skytrans limits
      • verify all track choices seem sane
  1. Update (manually) exposure times on all OCD-extended-exposures
    • effective 21-2, all OCD observations are charged at the on-sky exposure time, not the nominal/requested
    • use htopx to edit the value of exp on all accepted VIRUS targets observed with OCD that are not 1080s (DEX, DFW, G09, NEP/TESLA, GAMA, SSA, etc)
  1. Check the Daily Site Status report, the Daily Telemetry Report, the SAMS status report, etc via emails
    • look for anything crazy/weird
    • check that plot of pointing corrections looks sane

Join the 8:15am morning meeting whenever possible to contribute as needed to any PR-related discussions and field any operational questions about morning calibrations and that day's work plans.

After the "new data" emails go out at 10:30am CST / 11:30am CDT

  1. Check each of these programs' % Used at each priority ​here
    • For programs which include time shared/joined between multiple programs, log in to Hydra and access the PI-view for up-to-date totals (may need to add yourself as co-I to see it)
    • If a program has <0.2h remaining at any priority and still has targets active (i.e., <100% in the % Complete row), email the PI to let them know that allocation has been completed and they should defer their active targets.
      • Judgement can be used here w/r/t time balancing -- e.g., if Munich is lagging behind their partner share and a Munich program is poised to modestly exceed their allocation, I generally take no action.

Generally pay attention to OPS/night emails, address problems raised, answer questions, help improve operations and efficiency.


Weekly checks

  • review FCU lamp flux levels using LRS2 cals over the past week
    • generated weekly by plot_cals.py --email
    • nightly data collected when a cron job runs checkcals3.py
      • as of Oct 2023: only LRS2 cals are checked, would like to add HPF and VIRUS too
  • evaluate latest week of ACQ stability test data
    • generated weekly by stability_test.py acq --meta
    • (this is run manually by the RA at OPS each day to generate data)
  • evaluate weekly HPF pedestal level graph
    • generated by checkhbias.py --analyze
    • (a cron job runs checkhbias.py every day to collect data)
  • check program monitor outputs (prog_mon.py) and alert any PIs with issues
    • a cron job runs prog_mon.py on Monday mornings and emails the outputs
  • approve timesheets in Workday and update timeoff spreadsheet (usually Monday nights)
    • notify any night staff members who have more than 2 previous weeks of unsubmitted time sheets (usually Tuesday mornings)
    • more on timesheets here: NightOperations/Policies
  • collect and summarize project updates in ongoing night staff project reports (maybe ~monthly?)
    • share these summaries with Jim at the software meeting and Herman as well
  • attend Tuesday software meeting and share updates/input on night staff projects or other science operations issues
  • Organize colloquium-viewing at AL lower level meeting room
    • Monday: send email reminder if there is a talk
    • Monday: confirm with Karen at AL about cookies (usually order 3-5)
    • Tuesday: come at ~3:20 to set up screen, projector, computer
    • In July or August, talk to the AL manager about Colloquium dates for the fall semester and make reservations for the lower level meeting room
    • In December, talk to the AL manager about Colloquium dates for the spring semester likewise
  • Organize weekly operations staff meeting
    • Tuesday: request agenda items
    • Wednesday: distribute agenda
    • Thursday: run meeting from 3:30-4:30pm CDT / 3:00-4:00pm CST
    • Friday: distribute notes from meeting
  • Also, when changes/activities come up, notify the Visitor Center's staff as appropriate. Their tours to the GTAG normally include looking at the HET, so if we are doing maintenance or have dome lights off, it's very nice to keep them updated.


Bi-weekly checks

  • provide science operations report for bi-weekly telecon report; give feedback on other topics in report as needed.
  • attend the bi-weekly telecons: give science operations report, answer questions, give feedback as required
  • check the bi-weekly pointing report generated every other monday


calendar reminders throughout the week:

  • Monday:
    • send HObS meeting reminder (currently meetings are Tue at 10am)
  • Tuesday:
    • send email requesting agenda items for operations staff meeting
  • Wednesday:
    • send agenda for operations staff meeting
  • Thursday:
  • Friday
    • check if any targets in the queue have impossible exposure times, sky constraints, etc
    • check if any "HOLD" targets in the queue have impossible synoptic date constraints - if so, mark them as D and add a comment.
    • consider whether anything noteworthy changed with HET instruments/facility this week; if so, update appropriate webpages: ​LRS2 ​VIRUS ​HPF ​HRS
    • consider posting on the HET Blog: ​https://sites.utexas.edu/het/category/general/


Monthly checks

  • submit HPF RV-stability engineering targets into the -000 program in the queue. I usually submit all six of the targets even though some aren't visible in particular months.
  • if necessary, share eng/gto program status updates (no longer common in 2023)
  • update partner time charged statistics spreadsheet, monitor for and anticipate any problems with distribution
  • during the academic year (Sep-May), attend Research Scientists and Faculty meetings at UT on Zoom
  • add images from GC1/GC2/WFS1/WFS2 at OPS in the first couple days of the month to /data1/archive/users/stevenj/probe_health, and re-run plot.py to check for any broken fibers in CFBs or other probe problems
  • run Remedy on TACC on a recent HETDEX observation and upload the reconstructed image to this page: ​https://hydra.as.utexas.edu/?a=help&h=110 and highlight any bad amps


Mid-trimester checks

  • monitor completion rates of all science programs, send updates to PIs
  • identify targets in the queue which are impossible/unlikely to be observed, work with PIs to improve feasibility if possible


Quarterly checks

  • update staffing schedule for OPS/night:
    • we currently schedule in quarters: Feb/Mar/Apr; May/Jun/Jul; Aug/Sep/Oct; Nov/Dec/Jan
    • solicit time-off requests ~40 days before end of quarter, due ~30 days before end of quarter
    • make and post schedule for OPS, nights, and day OPS
    • update off-duty calendar with any staff off-duty or off-site
  • participate in HET Users Committee meetings (may require generating reports, responding to feature requests, etc)


Trimesterly tasks

  • create begin/end trimester reports
    • create report and distribute link to beginning of trimester report in first week or two of trimester
    • create report and distribute link to end of trimester report as soon as possible after trimester ends
  • solicit Eng/GTO time requests, compile and send to HET Board
    • solicit these 3 months before trimester starts; give 2 weeks deadline
    • send to Taft 2.5 months before trimester starts; ask for feedback by 1.5 months before trimester starts
  • calculate time available each trimester and inform partners of their science time
    • requires Board decisions on Eng/GTO time
    • use historical weather data plus time charged data
    • use spreadsheet like TACcalc_24-3.xls, update relevant information; balance historic weather data with historic time used from nightdata statistics and from time charged to programs statistics.
  • create 2-page proposal summary for RAs once all partners have submitted allocations
    • UT and PSU TACs will share their proposals; G and M do not have TACs/proposals
    • share this with the RAs before the start of the new trimester
  • serve on the UT HET TAC in officio or ex officio capacity
  • when the time allocations are submitted by the TAC chair, reach out to (especially UT!) PIs and offer support with anything they need during Phase II and Phase III
  • ask the TOs to test their home remote E-stop system, usually in the first month of each trimester.
  • check Nathan's statistics on LRS2 standard star centering accuracy, telescope pointing corrections, and other performance metrics generated monthly
  • schedule Zoom office hours with Greg Z for PI support (proposals, data, reductions, etc); generally plan for those in mid-January, mid-May, and mid-September, a few weeks before the UT HET proposal deadlines.
  • ask Amy W about average x,y offsets on WFS positions (she calculates the average offsets required before activating, and this can be super helpful to adjust the arm/carriage zeropoints to get them closer to the stars when we set up on targets)
  • on first day of new trimester (Dec 1, Apr 1, Aug 1) help Sergey, if needed, to change over from the old trimester to the new trimester:
    • (((include notes here)))


Biannual tasks


Annual tasks

Last modified 4 months ago Last modified on Jan 19, 2024 10:17:04 PM