Jim Fowler's notes to himself about procedures and processes to fix issues and problems at HET.

• PowerOutage Procedures

• VirtualHost and/or network switch outage/recovery

• TrackerProcedures? Procedures

• Lrs2 and Virus (camra) Procedures

• Guide Cameras and Single Board Computers
  • 192.168.66.140 sbc5 guider2
  • 192.168.66.141 sbc4 wfc2
  • 192.168.66.142 sbc3 acqcam
  • 192.168.66.143 sbc2 wfs1
  • 192.168.66.144 sbc1 guider1

The guide cameras are controlled by a single board computer (SBC) TS7800 manufactured by ​Technologic Systems. The operating manual for the SBC can be found online on their ​wiki. Both telnet and ssh are available for remote logins. There is a guider account on all systems with the usual guider password. The cameras are manufactured buy Finger Lakes and communicate with the SBC via USB. The process /usr/local/bin/usbserver running on each SBC provides the USB to Ethernet translation. The source code for this program can be found in /home/pso. (Note that I need to commit this code to the SVN repository.)

• To change the IP address on a Redhat 6.x system
  • change hostname in /etc/sysconfig/network to match /etc/hosts table
  • change IPADDR from 192.168.66.9 to 192.168.66.31 in /etc/sysconfig/network-scripts/ifcfg-em1
  • shutdown virtual htcs and wait a few minutes to let arp tables reset
  • restart hardware htcs
  • possibly restart dSpace

• To look up something in tcsnamed
  • syscmd -v -n 'lookup(key="something")'
  • tcsnamed database is in /opt/het/hetdex/etc/conf/tcsnamed_db.conf

• LDAP admin
  • add new user (example .ldif files can be found in het-ldap:/root)
    • ldapadd -x -W -D "cn=manager,dc=as,dc=utexas,dc=edu" -f secook.ldif
    • ldappasswd -S -W -D "cn=manager,dc=as,dc=utexas,dc=edu" -x "uid=secook,ou=People,dc=as,dc=utexas,dc=edu"
    • ldapmodify -x -W -D "cn=manager,dc=as,dc=utexas,dc=edu" -f secook_users.ldif

• Move previous month's weather data to the weather archives
  • change directories to /weather/archive/het/
  • move any previous month's files mmmYYYY.dat to the directory YYYY, where mmm is the lower case month name and YYYY is the year.
  • move any previous month's files enclosure_temps_mmmYYYY.dat to enclosure_temps/YYYY
  • the data should now be available through Darrin's weather archive
  • Note, need to automate this action someday
  • Note that the enclosure_temps_*.dat really don't contain any data, should shutdown the collection process someday.

• Move truss temperature data to weather archives
  • rdesktop trusst
  • login guider, standard password.
  • open a finder window and change to the folder to C:\Users\guider\Temperatures
  • open a second finder window change to the folder Z:\jrfwin\trusst
  • copy the relevant months from the first folder to the second folder
  • move older files in C:\Users\guider\Temperature to the relevant year folder
  • on a Redhat machine, login and change directory /data1/het/common/jrfwin/trusst
  • run the command python convert.py nnnn, where nnnn is the year of interest. This program will read the *nnnn_Temps.dat file, convert the data to a format understandable by the weather archive system, and write the reformatted data to /weather/archive/het/trusst/nnnn.
  • you can then remove the data file in /data1/het/common/jrfwin/trusst

• Restart Truss Temperatures Computer
  • rdesktop trusst
  • login as guider
  • Start a command prompt from the Start->Accessories->Command Prompt
  • run the command shutdown -r -f -t 0 (-r reboot instead of shutdown) (-f force kill all running applications) (-t 0 zero second delay)

• Enable core dumps on Redhat
  • see the instructions at ​https://www.unixmen.com/how-to-enable-core-dumps-in-rhel6/

• To clear a port on the terminal server
  • telnet hetts2
    • enable
    • tunnel <n> # Where <n> is the channel number 1-16
    • accept
    • kill connection
    • exit (do this command three times)
  • you should be able to connect to the port now.

• To reset a locked out Temperature Setpoint on Trane
  • if you are unable to set an Temperature set point
  • select the Applications Tree
  • select the Equipment item
  • select the Application tab on the Equipment Summary
  • select the Configuration tab
  • find AHU 1 Truss Temp Setpoint
  • select the Override on the right hand side
  • under Override Details click the checkbox on "Release all Manual Overrides"
  • click the Apply button

Bugzilla Notes:

• Work flow
  • user submits bug
  • responsible is assigned automatically or by JRF/Herman, state changed to assigned
  • assignee selects proper product/category for bug
  • assignee looks in to bug and makes comments (comment by assignee or status is duplicate)
  • assignee resolved bug status (status changed to resolved *)
  • Weekly cadmin or jrf can verify bugs
  • Weekly cadmin or jrf can close bugs

• Available products and assignees
  • Camra jason ramsey
  • Ccas jrf/herman
  • Dimm jrf/gdamm/herman
  • Dmi gdamm/herman
  • Doc jrf/herman
  • Dome gdamm/herman
  • Ecs gdamm/herman
  • Enclosure gdamm/herman
  • ...

Have categories General, Hardware, Software for all Products. Only General category is open for new bugs. Assignee selects proper category for the bug. Could have further subdivision for Dome-Hardware or Shutter-Hardware or Das-Software or Dome-Pmac or Shutter-Pmac under the Dome Product for example.

• To Do List
  • Close, reassign, duplicate existing open bugs so that only open bugs are real problems
  • Reassign or close bug assigned to people who are no longer here.
  • For future
    • Need to resolve if new bugs should only go to Jim-Herman or if they can be assigned automatically.
    • Need to determine Product, Categories
    • Consider PI categories
    • Need to write documentation for users
    • Need to generate weekly report somehow