Goals

• At the behest of the Board, HET will start full time science without any routine engineering periods. This requires that we be able to install trackable changes.
• install small, incremental changes without interrupting science or causing long down times.
• deploy larger changes in a controlled fashion
• be able to switch back to a working version of software if a change does not work.
  • this allows us to do short engineering runs of 1-2 hours and then revert back to working code for science
• test changes as completely as possible during the day time
• keep track of which version is running at which time
• report to operations team the changes and impact of the changes to be installed.

Development

The environment variable $(HET_SRC_ROOT) is defined as the location of the working copy of the software$(HET_DEPLOY_DIR) is the location of the deployment directory.

You can work in your own branch or trunk. For mid-sized or extensive modifications that may require intermediate commits, a branch is recommended. Once you have tested as fully as possible the branch can be deployed in order to test on the hardware during the day or on the sky at night. If your changes don't work, redeploy the previous tag (see Deployment below). Once you are confident that your modification works, then do a single merge with trunk. In this manner, only one revision needs to be merged or back out in order to add or remove your modification. When working in a branch, you should merge from trunk frequently to pick up the current working version. This will insure that changes made by others will not cause conflicts with your changes.

If you are making only small changes that need only one or two commits, you can work in trunk... but don't break the build.

Testing is important prior to releasing code on unsuspecting RA/TOs. Test as many of your modifications as possible prior to testing at night. How you do testing is up to you, but with a new deployment scheme it is easy to revert to a prior working edition if your testing reveals fatal problems.

If you need to test your modification at night, then create a branch, make your changes, commit, and then install only the package(s) with the changes using the -p flag from het_release. Allow your version to run at night but be prepared to re-install the current build (by running make install [packages] in the current build directory).

If testing during daytime only, then you can run the packages that have the changes. Replace the modified code before night operations by re-install the existing version with make install [packages] in the current build directory. In this way we can keep track of what version are running at night during science operations.

Deployment

Old Deployment method

• roughly once per month create a new branch from trunk with the name branches/deployment/science/YYYYMMDD
• build and install the branch
• merge important bug fixes into the branch as necessary

The current deployment method is done via the script het_release, for example I would use the following

het_release -U fowler -a jrf@utexas.edu -b /opt/het/hetdex/src/tcs -r het-operations@het.as.utexas.edu -i /opt/het/hetdex -u hetdex:hetdex -P trunk -p all

or if I was installing instruments or instruments_config the build directory would be /opt/het/hetdex/src/instruments

het_release -U fowler -a jrf@utexas.edu -b /opt/het/hetdex/src/instruments -r het-operations@het.as.utexas.edu -i /opt/het/hetdex -u hetdex:hetdex -P -p instruments_config

The command line flags for het_release are shown below. Note that -i is required for installation and -b is highly recommended. I use -U because my user name on hetdex-pr is different that my user name on the machines at Het. The -P flag defaults to trunk but must be used if you want to install a branch or a tag.

$ ./het_release -h
Usage: ./het_release [-b <build dir>] [-c <changelog>] [-d] [-h]
           [-i <install dir>] [-l <log file>] [-p <packages>]
           [-P <repo path>] [-r <recipients>] [-R <repo>]
           [-t <tag>] [-u <user>[:<group>]] [-U <repo user>]

Tags the HET <source repository path> at <tag>,
then checks the tag out, builds and optionally
installs the HET software <packages>.

NOTE: This script likely doesn't support file paths with
      special characters and/or spaces.

Options are:

  -a <email author> Sets the author for the email notification
                    for the build.
  -b <build dir>    Specifies the path where the <tag> will be
                    checked out and built. Defaults to /tmp.
  -c <changelog>    Copies the <changelog> to the given file.
                    The temporary file used for the changelog
                    is removed without copying by default.
  -d                Enable debug output. To disable system
                    and svn commands, give '-dd'.
  -h                Print this message and exit.
  -i <install dir>  Specifies the directory where the build will
                    be installed.  If omitted or given an empty
                    string, the installation is skipped. Defaults
                    to "".
  -l <log path>     Writes the script console log and build log
                    under <log path>/. If empty, the log is
                    written to the current working directory. 
                    Defaults to /tmp.
  -p <packages>     List of systems to build. Packages supported
                    are determined by top level directories and
                    makefile targets. Available group targets are:
                    all, telescope, instruments, all_config,
                    telescope_config, or instruments_config.
                    Defaults to all.
  -P <repo path>    Path of the installation source in the
                    repository. Defaults to trunk.
  -r <addresses>    Specify a comma separated list of email
                    <addresses> that will be sent a copy of
                    the build and changelog.
  -R <repo>         Remote location of the repository. Defaults
                    to svn+ssh://hetdex-pr.as.utexas.edu/repos/het.
  -t <tag>          Specify the repository path for the tag.
                    This is the relative path from the root given
                    by <repo>. Defaults to
                    tags/deployment/<hostname>_<ISO8601_time>.
  -u <user>:<group> Sets ownership of the installation
                    to <user>:<group>.
                    Defaults to leaving original installation
                    permissions set.
  -U <user>         Username for all svn transactions,
                    defaults to the current user.

• you may deploy/install trunk or any previous tag or branch
• the het_release script does the following steps,
  • start-up administrative work, e.g. setup logs, initialize variables
  • record all the variable settings in the output file
  • tag the current deployment as tags/deployment/host_name_YYYYMMDDThhmmss
  • check out a clean copy of the tag into /tmp or the directory specified by the -b command line flag
  • build (Note that standard makefile.local files are pre-built for htcs, vdas, lrs2, etc.
  • install the executable programs and scripts in the deployment directory, given by the -i command line flag
  • record results in the output file
  • email a report based on the output log the the email address specified by the -r command line flag
• the servers will need to be restarted manually
• you can generate a list of changes between a working copy and the trunk with
  • cd /opt/het/hetdex/src/tcs/build
  • generate-changelog svn+ssh://fowler@hetdex-pr.as.utexas.edu/repos/het/trunk .

Requirements and Caveats

• The build directory must be writable by the user running the het_release script.
• The installation directory must be writable by the user running the het_release script or by the user designated by the -u flag on the command line.
• Currently the Makefile enforces the installation user to be hetdex. This may change in the future.
• The het_release script is normally run on the host 'htcs'. This may also change if we do separate installation of instruments and telescope packages.
• Installing telescope or instruments does not install the configuration files.
• The target make install_configs is defined in makefile.commonTargets. If invoked in $(HET_SRC_ROOT), it will install all the files in $(HET_SRC_ROOT) into $(HET_DEPLOY_DIR)/etc. You do not want to do this.
• $(HET_SRC_ROOT) must be defined if you invoke make directly. $(HET_DEPLOY_DIR) will default to /opt/het/hetdex unless explicitly defined in the environment.

Examples

Some reasons for doing different types of installation and examples of how to do them

• You are working in trunk in the pas system on a bug fix. You have thoroughly tested your change and are ready to install it. You should use the command het_release ... -P trunk -p telescope
• You are working in trunk in the camra system on a bug fix. You have thoroughly tested your change and are ready to install it. You should use the command het_release ... -P trunk -p instruments
• You are working in trunk on a bug fix in het_lib. You have thoroughly tested your change and are ready to install it. You should use the command het_release ... -P trunk -p all
• You are working in trunk in the pas system on change to the configuration files. You have thoroughly tested your change and are ready to install it. You should use the command het_release ... -P trunk -p telescope_config
• You are working in trunk and adding a new spectrograph configuration file. You have thoroughly tested your change and are ready to install it. You should use the command het_release ... -P trunk -p instruments_config
• You have made changes to the pas configuration as well as adding a new spectrograph configuration. You should use the command het_release ... -P trunk -p all_config

• You have made extensive changes to the executables in a new branch, you have tested it, and are ready to install your changes. You should use the command het_release ... -P branches/my_neat_branch -p all
  • if you need to revert for some reason use the command het_release ... -P tags/deployment/<last_good_tag> -p all

Decisions Required

• decide whether or not to clean out the directory before installing new files. This will require that no other files are in the directory.
• decide whether to split the build/installation of the camra code and the scripts from main tcs servers
  • if so, create deployment directories for each, e.g. /opt/het/tcs, /opt/het/camra with ./bin, ./etc, ./lib as required
  • decide if we build camra on vdas/lrs separate from a build on htcs, build/install script can handle this requirement
• should third-party libraries be common across hosts, e.g. ZMQ, Qt, gsl
• does this methodology support a clean migration path to later versions or the libraries as well as to Redhat 7?
• write and test het_release script (mostly complete)

Previous notes from Jason (which this procedure is partially based on)

• Any deployment, science or otherwise, will be done from a fresh checkout of the repository.
• The version of repos/het installed will be tagged, repos/het/tags/deployment/YYYYmmddTHHMMs
• The target installation directories for repos/het should signify the source, this allows those directories to be refreshed in bulk rather than piecewise. Utilities that live outside of repos/het will not reside in the repos/het installation path. The install process will delete the prior installation, so that we don't run scripts from one version against tcs at another. (ok, we can just issue a warning during the install if we find something unexpected)
• An email will be sent to the group noting the change from one version to another, the details of the related repository pointers, the changelog, and any errors/warnings emitted by the build.
• How to segregate camra configs from this process?
• The script will be named "release", for now
• Usage is: release /path/to/repo, i.e. release ​svn+ssh://hetdex-pr/repos/het/branches/my_fun_test_branch
• Deployments will be stored in a common path
• The relevant email will contain the source locations of the before and after checkouts
• Deployment emails from instrument subsystems will be easily disambiguated from emails related to the general tcs
• Support scripting the install across multiple architectures with a single command?
• Checkouts related to non-before-or-after installs will be automatically deleted, assuming no local modifications. Otherwise a note about the prior-prior will be included in the relevant email, noting why it couldn't be removed.
• Only two copies per architecture (at most three, see above) will be on disk at HET, aside from developer checkouts. Only these 3 revisions will be referenced, at any point in time, in tcs communiques.
• Everything will go through a repository, whatever the technology.
• The process of rebuilding will include restarting the relevant services.
• Remove all files before installing new ones? Will have to move non-tcs executables and libraries to another location
• split camra, configs, tcs, hetlib?
• better issues and change tracking
• build/deploy for multiple architectures, RHEL 6/7, Het/107", use autoconf if we need to build on different architectures?