wiki:HetProcedures/RA/dailychecks

Version 16 (modified by stevenj, 7 months ago) (diff)

--

Here are some things that need to be done regularly for science operations.

Currently most of these are done by the Science Operation Manager, although some of that may be flexible in the future as our operational model evolves.


Daily checks

Before the 8:15am morning meeting:

  1. Check the ​NR Reader for last night
    • read PRs, OPS/night comments, respond/reply appropriately
    • also read the night RA's morning email summary
  1. Check the ​time accounting graph for last night
    • check that we have an activity entry at 18-deg evening twilight
    • check that there is no gap between stacking and the first target
    • check that all non-science activity entries have comments (e.g., idle/engineering)
    • check that all targets are consistent with the activity at that time (sci vs eng)
    • check for any un-used time/gaps and correct errors (or email RA if necessary)
    • eliminate "queue holes" that can be rolled into setup if OH remains <= 10min
    • add and increment targets to the -000 ENG program if appropriate to charge time for operational engineering activity entries
    • check for any other weird issues or obvious errors and take appropriate action
    • consider running nrcheck.py on zeus to catch any obvious issues
  1. Check the ​RA Night Report on nossy
    • check that all 'A/B' targets were incremented
    • check that no rejected targets were incremented
    • check for any crazy overheads
    • check that all setups with OH>10min have comments
    • check for any other weird issues or obvious errors
      • missing program IDs or other metadata?
      • acceptance/rejection errors or issues?
  1. Look at the gcplot output from last night
    • in /data1/archive/gcplot/gcplot_YYYYMMDD_targets.pdf
    • verify all targets were observed within their requested IQ/skysb/skytrans limits
    • verify all track choices seem sane
  1. Update (manually) exposure times on all OCD-extended-exposures
    • effective 21-2, all OCD observations are charged at the on-sky exposure time, not the nominal/requested
    • use htopx to edit the value of exp on all accepted VIRUS targets observed with OCD that are not 1080s (DEX, DFW, G09, NEP/TESLA, GAMA, SSA, etc)
  1. Check the Daily Site Status report, the Daily Telemetry Report, the SAMS status report, etc via emails
    • look for anything crazy/weird
    • check that plot of pointing corrections looks sane

Join the 8:15am morning meeting whenever possible to contribute as needed to any PR-related discussions and field any operational questions about morning calibrations and that day's work plans.

After the "new data" emails go out at 10:30am CST / 11:30am CDT

  1. Check each of these programs' % Used at each priority ​here
    • For programs which include time shared/joined between multiple programs, log in to Hydra and access the PI-view for up-to-date totals (may need to add yourself as co-I to see it)
    • If a program has <0.2h remaining at any priority and still has targets active (i.e., <100% in the % Complete row), email the PI to let them know that allocation has been completed and they should defer their active targets.
      • Judgement can be used here w/r/t time balancing -- e.g., if Munich is lagging behind their partner share and a Munich program is poised to modestly exceed their allocation, I generally take no action.

Generally pay attention to OPS/night emails, address problems raised, answer questions, help improve operations and efficiency.


Weekly checks

  • review FCU lamp flux levels using LRS2 cals over the past week
  • evaluate latest week of ACQ stability test data
  • evaluate weekly HPF pedestal level graph
  • review per-RA priority and dTopt average statistics (currently broken)
  • approve timesheets in Workday and update timeoff spreadsheet (usually Monday nights)
    • notify any night staff members who have more than 2 previous weeks of unsubmitted time sheets
  • collect and summarize project updates in ongoing night staff project reports (maybe ~monthly?)
  • attend Tuesday software meeting and share updates/input on night staff projects or other science operations issues
  • Organize colloquium-viewing at AL lower level meeting room
    • Monday: send email reminder if there is a talk
    • Monday: confirm with Karen at AL about cookies (usually order 3-5)
    • Tuesday: come at ~3:20 to set up screen, projector, computer
  • Organize weekly operations staff meeting
    • Tuesday: request agenda items
    • Wednesday: distribute agenda
    • Thursday: run meeting from 3:30-4:30pm CDT / 3:00-4:00pm CST
    • Friday: distribute notes from meeting


Bi-weekly checks

  • attend the bi-weekly telecons: give science operations report (edit document several days before meeting), answer questions, give feedback as required
  • check the bi-weekly pointing report generated every other monday


calendar reminders throughout the week:

  • Monday:
    • send HObS meeting reminder (currently meetings are Tue at 10am)
  • Tuesday:
    • send email requesting agenda items for operations staff meeting
  • Wednesday:
    • send agenda for operations staff meeting
  • Thursday:
  • Friday
    • check if any targets in the queue have impossible exposure times, sky constraints, etc
    • check if any "HOLD" targets in the queue have impossible synoptic date constraints - if so, mark them as D and add a comment.
    • consider whether anything noteworthy changed with HET instruments/facility this week; if so, update appropriate webpages: ​LRS2 ​VIRUS ​HPF ​HRS
    • consider posting on the HET Blog: ​https://sites.utexas.edu/het/category/general/


Monthly checks

  • if necessary, share eng/gto program status updates (no longer common in 2023)
  • update partner time charged statistics spreadsheet, monitor for and anticipate any problems with distribution
  • during the school year, attend Research Scientists and Faculty meetings at UT on Zoom


Mid-trimester checks

  • monitor completion rates of all science programs, send updates to PIs
  • identify targets in the queue which are impossible/unlikely to be observed, work with PIs to improve feasibility if possible


Quarterly checks

  • update staffing schedule for OPS/night:
    • we currently schedule in quarters: Feb/Mar/Apr?; May/Jun/Jul?; Aug/Sep/Oct?; Nov/Dec/Jan?
    • solicit time-off requests ~40 days before end of quarter, due ~30 days before end of quarter
    • make and post schedule for OPS, nights, and day OPS
    • update off-duty calendar with any staff off-duty or off-site
  • participate in HET Users Committee meetings (may require generating reports, responding to feature requests, etc)


Trimesterly tasks

  • create begin/end trimester reports
  • solicit Eng/GTO time requests, compile and send to HET Board
  • calculate time available each trimester and inform partners of their science time
  • serve on the UT HET TAC in officio or ex officio capacity


Biannual tasks

  • give Science Operations report/presentation at the HET Board meetings in ~June and ~December


Annual tasks