Update docs to include cached outputs

Author: xylar

docs/developers_guide/api.rst

@@ -90,6 +90,17 @@ run
    run_step
 
 
+cache
+~~~~~
+
+.. currentmodule:: compass.cache
+
+.. autosummary::
+   :toctree: generated/
+
+   update_cache
+
+
 Base Classes
 ^^^^^^^^^^^^
 

docs/developers_guide/command_line.rst

@@ -276,3 +276,57 @@ Would both accomplish the same thing in this example -- skipping the
     over the config option.
 
 See :ref:`dev_run` for more about the underlying framework.
+
+.. _dev_compass_cache:
+
+compass cache
+-------------
+
+``compass`` supports caching outputs from any step in a special database
+called ``compass_cache`` (see :ref:`dev_step_input_download`). Files in this
+database have a directory structure similar to the work directory (but without
+the MPAS core subdirectory, which is redundant). The files include a date stamp
+so that new revisions can be added without removing older ones (supported by
+older compass versions).  See :ref:`dev_step_cached_output` for more details.
+
+A new command, ``compass cache`` has been added to aid in updating the file
+``cached_files.json`` within an MPAS core.  This command is only available on
+Anvil and Chrysalis, since developers can only copy files from a compass work
+directory onto the LCRC server from these two machines.  Developers run
+``compass cache`` from the base work directory, giving the relative paths of
+the step whose outputs should be cached:
+
+.. code-block:: bash
+
+    compass cache -i ocean/global_ocean/QU240/mesh/mesh \
+        ocean/global_ocean/QU240/PHC/init/initial_state
+
+This will:
+
+1. copy the output files from the steps directories into the appropriate
+   ``compass_cache`` location on the LCRC server and
+
+2. add these files to a local ``ocean_cached_files.json`` that can then be
+   copied to ``compass/ocean`` as part of a PR to add a cached version of a
+   step.
+
+The resulting ``ocean_cached_files.json`` will look something like:
+
+.. code-block:: json
+
+    {
+        "ocean/global_ocean/QU240/mesh/mesh/culled_mesh.nc": "global_ocean/QU240/mesh/mesh/culled_mesh.210803.nc",
+        "ocean/global_ocean/QU240/mesh/mesh/culled_graph.info": "global_ocean/QU240/mesh/mesh/culled_graph.210803.info",
+        "ocean/global_ocean/QU240/mesh/mesh/critical_passages_mask_final.nc": "global_ocean/QU240/mesh/mesh/critical_passages_mask_final.210803.nc",
+        "ocean/global_ocean/QU240/PHC/init/initial_state/initial_state.nc": "global_ocean/QU240/PHC/init/initial_state/initial_state.210803.nc",
+        "ocean/global_ocean/QU240/PHC/init/initial_state/init_mode_forcing_data.nc": "global_ocean/QU240/PHC/init/initial_state/init_mode_forcing_data.210803.nc"
+    }
+
+An optional flag ``--date_string`` lets the developer set the date string to
+a date they choose.  The default is today's date.
+
+The flag ``--dry_run`` can be used to sanity check the resulting ``json`` file
+and the list of files printed to stdout without actually copying the files to
+the LCRC server.
+
+See :ref:`dev_cache` for more about the underlying framework.

docs/developers_guide/framework.rst

@@ -99,6 +99,19 @@ from individual steps are stored in log files ``<step>.log`` in the test case's
 work directory.  The results of validation (if any) are displayed in the final
 stage of running the test case.
 
+.. _dev_cache:
+
+cache module
+~~~~~~~~~~~~
+
+The :py:func:`compass.cache.update_cache()` function is used by
+``compass cache`` to copy step outputs to the ``compass_cache`` database on
+the LCRC server and to update ``<mpas_core>_cached_files.json`` files that
+contain a mapping between these cached files and the original outputs.  This
+functionality enables running steps with :ref:`dev_step_cached_output`, which
+can be used to skip time-consuming initialization steps for faster development
+and debugging.
+
 .. _dev_config:
 
 Config files

docs/developers_guide/organization.rst

@@ -815,13 +815,6 @@ As was the case for test cases, the base class :py:class:`compass.Step` has a
 large number of attributes that are useful at different stages (init, setup and
 run) of the step.
 
-    logger : logging.Logger
-        A logger for output from the step
-
-    log_filename : str
-        At run time, the name of a log file where output/errors from the step
-        are being logged, or ``None`` if output is to stdout/stderr
-
 Some attributes are available after calling the base class' constructor
 ``super().__init__()``.  These include:
 
@@ -857,6 +850,10 @@ Some attributes are available after calling the base class' constructor
 ``self.threads``
     the number of threads the step will use
 
+``self.cached``
+    Whether to get all of the outputs for the step from the database of
+    cached outputs for the MPAS core that this step belongs to
+
 Another set of attributes is not useful until ``setup()`` is called by the
 ``compass`` framework:
 
@@ -897,6 +894,10 @@ framework:
     methods and functions that use the logger to write their output to the log
     file.
 
+``self.log_filename``
+    The name of a log file where output/errors from the step are being logged,
+    or ``None`` if output is to stdout/stderr
+
 The inputs and outputs should not be altered but they may be used to get file
 names to read or write.
 
@@ -1544,6 +1545,93 @@ in the test case's :ref:`dev_test_case_init`.
 The relative path in ``filename`` is with respect to the step's work directory,
 and is converted to an absolute path internally before the step is run.
 
+.. _dev_step_cached_output:
+
+Cached output files
+~~~~~~~~~~~~~~~~~~~
+
+Many ``compass`` test cases and steps are expensive enough that it can become
+time consuming to run full workflows to produce meshes and initial conditions
+in order to test simulations.  Therefore, ``compass`` provides a mechanism for
+caching the outputs of each step in a database so that they can be downloaded
+and symlinked rather than being computed each time.
+
+Cached output files are be stored in the ``compass_cache`` database within each
+MPAS core's space on that LCRC server (see :ref:`dev_step_input_download`).
+If the "cached" version of a step is selected, as we will describe below, each
+of the test case's outputs will have a corresponding "input" file added with
+the ``target`` being a cache file on the LCRC server and the ``filename`` being
+the output file.  ``compass`` uses the ``cached_files.json`` database to know
+which cache files correspond to which step outputs.
+
+A developer can indicate that ``compass`` test suite includes steps with cached
+outputs in two ways.  First, if all steps in a test case should have cached
+output, the following notation should be used:
+
+.. code-block:: none
+
+    ocean/global_ocean/QU240/mesh
+        cached
+    ocean/global_ocean/QU240/PHC/init
+        cached
+
+That is, the word ``cached`` should appear after the test case on its own line.
+The indentation is for visual clarity and is not required.
+
+
+Second, ff only some steps in a test case should have cached output, they need
+to be listed explicitly, as follows:
+
+.. code-block:: none
+
+    ocean/global_ocean/QUwISC240/mesh
+        cached: mesh
+    ocean/global_ocean/QUwISC240/PHC/init
+        cached: initial_state ssh_adjustment
+
+The line can be indented for visual clarity, but must begin with ``cached:``,
+followed by a list of steps separated by a single space.
+
+Similarly, a user setting up test cases has two mechanisms for specifying which
+test cases and steps should have cached outputs.  If all steps in a test case
+should have cached outputs, the suffix ``c`` can be added to the test number:
+
+.. code-block:: none
+
+    compass setup -n 90c 91c 92 ...
+
+In this example, test cases 90 and 91 (``mesh`` and ``init`` test cases from
+the ``SOwISC12to60`` global ocean mesh, in this case) are set up with cached
+outputs in all steps and 92 (``performance_test``) is not.  This approach is
+efficient but does not provide any control of which steps use cached outputs
+and which do not.
+
+A much more verbose approach is required if some steps use cached outputs and
+others do not within a given test case.  Each test case must be set up on its
+own with the ``-t`` and ``--cached`` flags as follows:
+
+
+.. code-block:: none
+
+    compass setup -t ocean/global_ocean/QU240/mesh --cached mesh ...
+    compass setup -t ocean/global_ocean/QU240/PHC/init --cached initial_state ...
+    ...
+
+Cache files should be generated by first running the test case as normal, then
+running the :ref:`dev_compass_cache` command-line tool at the base of the work
+directory, providing the names of the steps whose outputs should be added to
+the cache.  The resulting ``<mpas_core>_cached_files.json`` should be copied
+to ``compass/<mpas_core>/cached_files.json`` in a ``compass`` branch.
+
+Calls to ``compass cache`` must be made on Chrysalis or Anvil.  If outputs were
+produced on another machine, they must be transferred to one of these two
+machines before calling ``compass cache``.  File can be added manually to the
+LCRC server and the ``cached_files.json`` databases but this is not
+recommended.
+
+More details on cached outputs are available in the design document
+:ref:`design_doc_cached_outputs`.
+
 .. _dev_step_namelists_and_streams:
 
 Adding namelist and streams files