Draft: Create experiment, workflow, and tasks documentation from configuration, as well as metadata

[kinow]

In GitLab by @kinow on Jan 19, 2024, 12:16

Closes #939

Creating a merge request to demonstrate the idea of the automated documentation of experiments. This is not super new, as you can achieve almost the same with Cylc and ecFlow. In Cylc you have DOC and TITLE too. In ecFlow they use MANUAL.

I am extending the scope of the issue (created 11 months ago) to add something that was discussed more recently, about the documentation for models, metadata, ...

The idea is documented in the docgen/__init__.py docstrings. Pasting it here:

Goes over experiment configuration, gathering each job's
``TITLE`` and ``DOC`` to create the documentation for the
jobs.

Goes over the ``METADATA`` configuration entry, looking
for sections with ``key=value`` pairs, or links to
configuration keys, e.g.:

.. code-block:: yaml

  METADATA:
    MODEL:
      # This is a static value that will produce name = IFS
      - name: name
      # This is a dynamic value that will produce resolution = abc...
      - resolution: %RUN.IFS.RESOLUTION%
      # This resolves the key and value automatically and will produce RUN.IFS.INIPATH=/path/path/...
      - %RUN.IFS.INIPATH%
      # This is the extended format
      - name: source
        value: https://git@...
        documentation: |
          The source code is hosted at the private repository...
    DATA:
      - PROVENANCE: ...


Each property of the ``METADATA`` YAML object is rendered as
a separate section. These sections, in YAML, are arrays that
contain metadata references in each line. Extra documentation
about each key can be entered using the extended format.

The gist is that the maintainers of the experiment would be able to select configuration values from the workflow that must be exported as workflow metadata (in the METADATA YAML object/entry above).

The new command in Autosubmit will take care to go over the experiment configuration, resolve variables like %EXPID%, %RUN.IFS.RESOLUTION.ETC%, and produce the metadata list, with the current values used in the experiment.

I think this is key here, to have the documentation generated with the values used in the workflow -- i.e. the workflow run configuration is the single source of truth, to avoid having a PDF or Wiki page manually written that might be out of sync with what's actually happening in the workflow.

[kinow]

In GitLab by @kinow on Jan 19, 2024, 12:19

The current code is a draft and only produces:

$ autosubmit docgen a001
Autosubmit is running with 4.0.99

Checking configuration files...
Expdef config file is correct
Platforms sections: OK
Jobs sections OK
Autosubmit general sections OK
Configuration files OK

# Workflow Jobs

These are the jobs in the workflow “a001
”.
## INI

**Initialize the experiment**
This task runs locally, the expid is %EXPID%

## SIM

## ASIM

## POST

# Metadata

This is valid Markdown, that could be used to generate a web page, PDF, docx, etc.. The contents of the ini job configuration:

  ini:
      FILE: 1.sh
      RUNNING: once
      TITLE: Initialize the experiment
      doc: This task runs locally, the expid is %EXPID%

Pasting the content below so GitLab can render the markdown (expanding the %EXPID% is easy but I will add that later):

Workflow Jobs

These are the jobs in the workflow “a001
”.

INI

Initialize the experiment
This task runs locally, the expid is %EXPID%

SIM

ASIM

POST

Metadata

[kinow]

In GitLab by @kinow on Jan 19, 2024, 12:20

/cc @mcastril , @GinkaVanThielen , @ainagaya , @Lerriola , @franra9 , @sebastianbeyer , @fdoblasreyes wrote a quick merge request just now to demonstrate the idea of automated documentation generation ☝️

[kinow]

In GitLab by @kinow on Jan 19, 2024, 12:20

Commented on autosubmit/docgen/init.py line 1

Note to self, update to 2024

[kinow]

In GitLab by @kinow on Jan 19, 2024, 12:22

And this doesn't need to replace any other documentation planned for models; this can be used to complement any documentation/wiki/deliverable/etc., but automating the parts that contain variables from the workflow.

[kinow]

In GitLab by @kinow on Jan 19, 2024, 12:36

mentioned in issue digital-twins/de_340/project_management#579

[kinow]

In GitLab by @mcastril on Jan 19, 2024, 19:09

This is wonderful Bruno. Can you clarify one doubt?

Goes over the ``METADATA`` configuration entry, looking
for sections with ``key=value`` pairs, or links to
configuration keys, e.g.:

Wouldn't it be key: value?

[kinow]

In GitLab by @ainagaya on Jan 22, 2024, 09:55

That's great @kinow !!

So, if I understood correctly, what should be done in the workflow is to create a configuration file with the METADATA entries and fill them with placeholders? Sounds good to me.

[kinow]

In GitLab by @kinow on Jan 22, 2024, 10:01

Yup! That's the initial idea. But we can work on it and decide what would be the best way for that. I think with a few more hours (2...3) I can get it to work and produce the final markdown.

With a working prototype we can test in the climatedt workflow or another example workflow, and then discuss what needs changing.

[kinow]

In GitLab by @kinow on Jan 22, 2024, 10:33

Wouldn't it be key: value?

Ah, yes, so that it's valid YAML, good catch. Will fix that in the docs later. Thanks @mcastril !

[kinow]

In GitLab by @ainagaya on Jan 24, 2024, 13:07

mentioned in issue digital-twins/de_340/workflow#174

[dbeltrankyl]

Status update?

[kinow]

Haven't had time to update it yet. Once Postgres is done I will have a look at this one 👍