Merge pull request #118 from EcoExtreML/update_docs

Update docs and fix some issues

Author: SarahAlidoost

.github/workflows/markdown-link-check.yml

@@ -12,9 +12,9 @@ jobs:
 
   markdown-link-check:
     name: Check markdown links
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04 # there is an issue with ubuntu-latest and linkspector
     steps:
-    - uses: actions/checkout@v3
-    - uses: gaurav-nelson/github-action-markdown-link-check@v1
+    - uses: actions/checkout@v4
+    - uses: umbrelladocs/action-linkspector@v1
       with:
-        config-file: '.mlc-config.json'
+        config_file: '.linkspector.yml'

.github/workflows/publish-container.yml

@@ -0,0 +1,59 @@
+#
+name: Create and publish the PyStemmusScope container image 📦
+
+on:
+  release:
+    types: [published]
+  workflow_dispatch:
+
+# Defines two custom environment variables for the workflow. These are used for the Container registry domain, and a name for the Docker image that this workflow builds.
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository }}
+
+# There is a single job in this workflow. It's configured to run on the latest available version of Ubuntu.
+jobs:
+  build-and-push-image:
+    runs-on: ubuntu-latest
+    # Sets the permissions granted to the `GITHUB_TOKEN` for the actions in this job.
+    permissions:
+      contents: read
+      packages: write
+      attestations: write
+      id-token: write
+      #
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+      # Uses the `docker/login-action` action to log in to the Container registry registry using the account and password that will publish the packages. Once published, the packages are scoped to the account defined here.
+      - name: Log in to the Container registry
+        uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      # This step uses [docker/metadata-action](https://github.com/docker/metadata-action#about) to extract tags and labels that will be applied to the specified image. The `id` "meta" allows the output of this step to be referenced in a subsequent step. The `images` value provides the base name for the tags and labels.
+      - name: Extract metadata (tags, labels) for Docker
+        id: meta
+        uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+      # This step uses the `docker/build-push-action` action to build the image, based on your repository's `Dockerfile`. If the build succeeds, it pushes the image to GitHub Packages.
+      # It uses the `context` parameter to define the build's context as the set of files located in the specified path. For more information, see "[Usage](https://github.com/docker/build-push-action#usage)" in the README of the `docker/build-push-action` repository.
+      # It uses the `tags` and `labels` parameters to tag and label the image with the output from the "meta" step.
+      - name: Build and push Docker image
+        id: push
+        uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4
+        with:
+          context: .
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+
+      # This step generates an artifact attestation for the image, which is an unforgeable statement about where and how it was built. It increases supply chain security for people who consume the image. For more information, see "[AUTOTITLE](/actions/security-guides/using-artifact-attestations-to-establish-provenance-for-builds)."
+      - name: Generate artifact attestation
+        uses: actions/attest-build-provenance@v1
+        with:
+          subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME}}
+          subject-digest: ${{ steps.push.outputs.digest }}
+          push-to-registry: true

.github/workflows/python-publish.yml

@@ -11,6 +11,7 @@ name: Upload Python Package
 on:
   release:
     types: [published]
+  workflow_dispatch:
 
 permissions:
   contents: read

.linkspector.yml

@@ -0,0 +1,14 @@
+dirs:
+  - ./docs
+excludedDirs:
+  - ./docs/notebooks
+  - ./docs/overrides
+ignorePatterns:
+  - pattern: "^http://localhost"
+  - pattern: "^https://doi.org/<replace-with-created-DOI>"
+  - pattern: "^https://github.com/.*/settings/secrets/actions$"
+  - pattern: "^https://github.com/organizations/.*/repositories/new"
+  - pattern: "^https://test.pypi.org"
+  - pattern: "^https://bestpractices.coreinfrastructure.org/projects/<replace-with-created-project-identifier>"
+  - pattern: "^https://readthedocs.org/dashboard/import.*"
+useGitIgnore: true

.mlc-config.json

@@ -0,0 +1,5 @@
+"""Documentation about StemmusScopeBmi."""
+from .implementation import StemmusScopeBmi
+
+
+__all__ = ["StemmusScopeBmi"]

PyStemmusScope/bmi/__init__.py

@@ -104,6 +104,7 @@ def __init__(self, cfg_file: str) -> None:
         env = {
             "LD_LIBRARY_PATH": lib_path,
             "MATLAB_LOG_DIR": str(config["OutputPath"]),
+            "MCR_CACHE_ROOT": str(os.getenv("MCR_CACHE_ROOT")),
         }
 
         self.process = subprocess.Popen(

PyStemmusScope/bmi/local_process.py

@@ -18,18 +18,11 @@
 [![cffconvert](https://github.com/EcoExtreML/stemmus_scope_processing/actions/workflows/cffconvert.yml/badge.svg)](https://github.com/EcoExtreML/stemmus_scope_processing/actions/workflows/cffconvert.yml)
 [![markdown-link-check](https://github.com/EcoExtreML/stemmus_scope_processing/actions/workflows/markdown-link-check.yml/badge.svg)](https://github.com/EcoExtreML/stemmus_scope_processing/actions/workflows/markdown-link-check.yml) -->
 
-This repository includes the python package `PyStemmusScope` for running the
-STEMMUS-SCOPE model.
+This repository includes the python package `PyStemmusScope` for preparing data
+nd running the STEMMUS-SCOPE model.
 
 <!-- markdown-link-check-disable-next-line -->
 The model source code, executable file and utility files are available in the [STEMMUS_SCOPE repository](https://github.com/EcoExtreML/STEMMUS_SCOPE).
 
-The input datasets are available on Snellius and CRIB. First, make sure you have
-right access to the repository and data. Then, see the notebook
-[run_model_on_different_infra.ipynb](./docs/notebooks/run_model_on_different_infra.ipynb)
-which provides different ways to run the model depending on your system and
-[run_model_with_different_dataset.ipynb](./docs/notebooks/run_model_with_different_dataset.ipynb)
-on how to use different datasets e.g. site or global data.
-
 More information on the setup and installation, including for own machine, is available
 in the [documentation](https://pystemmusscope.readthedocs.io/).

README.md

@@ -1,16 +1,81 @@
 # Contributing Guide
 
-If you want to contribute to `PyStemmusScope`, you will need to set up your development
-environment the right way. The instructions below will guide you though the steps
+We welcome any kind of contributions to our software, from simple
+comment or question to a full fledged [pull
+request](https://help.github.com/articles/about-pull-requests/). Please
+read and follow our [Code of Conduct](./CODE_OF_CONDUCT.md).
+
+A contribution can be one of the following cases:
+
+1. you have a question;
+2. you think you may have found a bug (including unexpected behavior);
+3. you want to make some kind of change to the code base (e.g. to fix a
+    bug, to add a new feature, to update documentation).
+4. you want to make a release
+
+The sections below outline the steps in each case.
+
+## You have a question
+
+1. use the search functionality
+    [here](https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/issues) to see if
+    someone already filed the same issue;
+2. if your issue search did not yield any relevant results, make a new issue;
+3. apply the \"Question\" label; apply other labels when relevant.
+
+## You think you may have found a bug
+
+1. use the search functionality
+    [here](https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/issues) to see
+    if someone already filed the same issue;
+2. if your issue search did not yield any relevant results, make a new issue,
+    making sure to provide enough information to the rest of the community to
+    understand the cause and context of the problem. Depending on the issue, you
+    may want to include: - the [SHA
+    hashcode](https://help.github.com/articles/autolinked-references-and-urls/#commit-shas)
+    of the commit that is causing your problem; - some identifying information
+    (name and version number) for dependencies you\'re using; - information
+    about the operating system;
+3. apply relevant labels to the newly created issue.
+
+## You want to make some kind of change to the code base
+
+1. (**important**) announce your plan to the rest of the community
+    *before you start working*. This announcement should be in the form
+    of a (new) issue;
+2. (**important**) wait until some kind of consensus is reached about
+    your idea being a good idea;
+3. if needed, fork the repository to your own Github profile and create your own
+    feature branch off of the latest main commit. While working on your feature
+    branch, make sure to stay up to date with the main branch by pulling in
+    changes, possibly from the \'upstream\' repository (follow the instructions
+    [here](https://help.github.com/articles/configuring-a-remote-for-a-fork/)
+    and [here](https://help.github.com/articles/syncing-a-fork/));
+4. If you are using [Visual Studio Code](https://code.visualstudio.com), some
+   extensions will be recommended and you are offered to run inside a
+   [DevContainer](https://containers.dev) in which the dependencies are already
+   installed;
+
+In case you feel like you\'ve made a valuable contribution, but you
+don\'t know how to write or run tests for it, or how to generate the
+documentation: don\'t let this discourage you from making the pull
+request; we can help you! Just go ahead and submit the pull request, but
+keep in mind that you might be asked to append additional commits to
+your pull request.
+
+If you want to contribute to `PyStemmusScope`, we recommend installing
+the package in editable mode. The instructions below will guide you though the steps
 required.
 
-## Configure the python package for development and testing
+### Dependencies
 
-To contribute to the development of the python package, we recommend installing
-the package in development mode.
+Check the package `dependencies` and `optional dependencies` in the
+`pyproject.toml` file in the root directory of the repository. The package
+dependecies are those packages that are required to build the package as a
+software. The optional dependencies are those packages that are required to run
+the tests, build the documentation, and format the code.
 
-
-### Installation
+### Installation in editable mode
 
 First, clone this repository:
 
@@ -36,23 +101,39 @@ python3 -m pytest
 
 ### Building the documentation
 
+To install the documentation dependencies (On Windows, use `python` instead of `python3`):
+
+```sh
+cd STEMMUS_SCOPE_Processing
+python3 -m install -e .[docs]
+```
+
 To edit the documentation locally, do:
 
 ```sh
 mkdocs serve
 ```
 
 Then open the local hyperlink displayed in the terminal, e.g.:
+
 ```
 INFO     -  [13:23:44] Serving on http://127.0.0.1:8000/
 ```
 
 ### Run formatting tools
 
-You can use `prospector` to get information about errors, potential problems and convention violations. To run:
+Formatting configs are listed in the `pyproject.toml` file. You can use `ruff`
+to get information about errors, potential problems and convention violations.
+To run:
 
 ```sh
-prospector
+ruff check .
+```
+
+It is possible to fix some of the errors automatically. To do so, run:
+
+```sh
+ruff check --fix .
 ```
 
 To format the import statements, you can use `isort` as:
@@ -61,7 +142,69 @@ To format the import statements, you can use `isort` as:
 isort
 ```
 
-## Development of STEMMUS_SCOPE model
+### BMI Developer instructions
+
+The Python BMI implemented in this package communicates with the Matlab code
+through STDIN/STDOUT, or via a socket to the Docker container.
+Over this interface, three commands can be sent to Matlab:
+
+1. `initialize "path_to_cfg_file.txt"`
+2. `update`
+3. `finalize`
+
+After the initialize and update steps, the Matlab process writes the state of
+any BMI exposed variables to an hdf5-file in the directory of `OutputPath` as
+defined in the configuration file.
+
+The Python BMI interfaces with this file to allow the variables to be read and set.
+
+#### Adding/changing exposed variables
+
+Step one of changing the exposed variables is to change the Matlab code and
+generating a new MCR executable (and possibly Docker image). The exposed
+variables are defined in
+[`STEMMUS_SCOPE/src/STEMMUS_SCOPE_exe.m`](https://github.com/EcoExtreML/STEMMUS_SCOPE/blob/main/src/STEMMUS_SCOPE_exe.m).
+Under the `bmiVarNames` variable. Make sure that you add the model variable
+here, as well as any info on the variable's grid.
+
+The available variable names (`MODEL_INPUT_VARNAMES`, `MODEL_OUTPUT_VARNAMES`),
+their units (`VARNAME_UNITS`), datatypes (`VARNAME_DTYPE`) and grids
+(`VARNAME_GRID`) are defined in constants at the top of the file
+`PyStemmusScope/bmi/implementation.py`. These have to be updated to reflect the
+changes in the state file.
+
+Lastly you have to update the `get_variable` and `set_variable` functions in
+`PyStemmusScope/bmi/implementation.py`. Here you define how the python code can
+access them. While writing the code you can inspect the state using
+`model.state`, which allows you to view the full contents of the HDF5 file for
+easier debugging.
+
+After implementing the BMI changes, a new [STEMMUS_SCOPE Docker
+image](https://github.com/EcoExtreML/STEMMUS_SCOPE/pkgs/container/stemmus_scope)
+should be released that is compatible with the new BMI implementation. A new
+release usually includes a new tag. Then, you need to update the
+`compatible_tags` variable of the class `StemmusScopeDocker` in
+`PyStemmusScope/bmi/docker_process.py`.
+
+## Making a release
+
+This section describes how to make a release in 3 parts:
+
+1. preparation
+1. making a release on GitHub
+
+### (1/2) Preparation
+
+1. Update the <CHANGELOG.md> (don't forget to update links at bottom of page)
+2. Verify that the information in `CITATION.cff` is correct, and that `.zenodo.json` contains equivalent data
+3. Make sure the version has been updated.
+4. Run the unit tests with `pytest -v`
+
+### (2/2) GitHub
 
-<!-- markdown-link-check-disable-next-line -->
-To contribute to the STEMMUS_SCOPE model, you need access to the model source code that is stored in the repository [STEMMUS_SCOPE](https://github.com/EcoExtreML/STEMMUS_SCOPE).
+Don't forget to also make a [release on
+GitHub](https://github.com/EcoExtreML/stemmus_scope_processing/releases/new).
+This will trigger the github action `python-publish.yml` that publishes the
+package on PyPI. If your repository uses the GitHub-Zenodo integration this will
+also trigger Zenodo into making a snapshot of your repository and sticking a DOI
+on it.