Merge pull request #45 from EcoExtreML/add_octave

Add octave and matlab to run module

Author: SarahAlidoost

CONTRIBUTING.md

@@ -4,7 +4,8 @@ This repository includes the python package `PyStemmusScope` for running the STE
 
 ## Configure the python package for development and testing
 
-To contribute to development of the python package, we recommend installing the package in development mode.
+To contribute to the development of the python package, we recommend installing
+the package in development mode.
 
 
 ### Installation
@@ -15,26 +16,20 @@ First, clone this repository:
 git clone https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing.git
 ```
 
-Then install the package:
+Then install the package (On Windows, use `python` instead of `python3`):
 
 ```sh
 cd STEMMUS_SCOPE_Processing
-pip install -e .
-```
-
-or
-
-```sh
-python setup.py develop
+python3 -m install -e .[dev]
 ```
 
 ### Run tests
 
 The testing framework used here is [PyTest](https://pytest.org). You can run
-tests as:
+tests as (On Windows, use `python` instead of `python3`):
 
 ```sh
-pytest
+python3 -m pytest
 ```
 
 ### Build documentation
@@ -65,80 +60,4 @@ isort
 ## Development of STEMMUS_SCOPE model
 
 <!-- markdown-link-check-disable-next-line -->
-To contribute to the STEMMUS_SCOPE model, you need access to the model source code that is stored in the repository [STEMMUS_SCOPE](https://github.com/EcoExtreML/STEMMUS_SCOPE). You also need a MATLAB license.
-
-### Development on Snellius using MATLAB
-
-[Snellius](https://servicedesk.surfsara.nl/wiki/display/WIKI/Snellius) is the
-Dutch National supercomputer hosted at SURF. MATLAB `2021a` is installed on
-Snellius, see the script
-[`run_jupyter_lab_snellius_dev.sh`](https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/blob/main/run_jupyter_lab_snellius_dev.sh)
-on how to load the module.
-
-The script `run_jupyter_lab_snellius_dev.sh` activates the conda environment `pystemmusscope` and creates a jupyter lab server on Snellius for running the notebook
-interactively. Make sure that you create the `pystemmusscope` conda environment before submitting the the bash script. See **Create pystemmusscope environment** below.
-
-<details>
-  <summary>Create pystemmusscope environment</summary>
-
-Run the commands below in a terminal:
-
-```sh
-# Download and install Mamba on linux
-wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-pypy3-Linux-x86_64.sh
-bash Mambaforge-pypy3-Linux-x86_64.sh -b -p ~/mamba
-
-# Update base environment
-. ~/mamba/bin/activate
-mamba update --name base mamba
-
-# Download environment file
-wget https://raw.githubusercontent.com/EcoExtreML/STEMMUS_SCOPE_Processing/main/environment.yml
-
-# Create a conda environment called 'pystemmusscope' with all required dependencies
-mamba env create -f environment.yml
-
-# The environment can be activated with
-. ~/mamba/bin/activate pystemmusscope
-
-```
-</details>
-
-
-### Development on CRIB using MATLAB
-
-[CRIB](https://crib.utwente.nl/) is the ITC Geospatial Computing Platform.
-
-MATLAB `2021a` is installed on CRIB, and supports Python `3.8`, see [Versions of Python Compatible with MATLAB Products](https://www.mathworks.com/content/dam/mathworks/mathworks-dot-com/support/sysreq/files/python-compatibility.pdf).
-
-1. Log in CRIB with your username and password and select a proper compute unit.
-2. Install `PyStemmusScope` package. This step needs to be done once.
-  <details>
-    <summary>Install pystemmusscope with python 3.8</summary>
-
-  Run the commands below in a terminal:
-
-  ```sh
-  # Download and install Mamba on linux
-  wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-pypy3-Linux-x86_64.sh
-  bash Mambaforge-pypy3-Linux-x86_64.sh -b -p ~/mamba
-
-  # Update base environment
-  . ~/mamba/bin/activate
-  mamba update --name base mamba
-
-  # Download environment file
-  wget https://raw.githubusercontent.com/EcoExtreML/STEMMUS_SCOPE_Processing/main/environment_3.8.yml
-
-  # Create a conda environment called 'pystemmusscope' with all required dependencies
-  mamba env create -f environment_3.8.yml
-  ```
-  </details>
-
-3. click on the `Remote Desktop` in the
-Launcher. Click on the `Applications`. You will find the 'MATLAB' software under
-the `Research`.
-4. After clicking on 'MATLAB', it asks for your account information that is
-connected to a MATLAB license.
-5. Open the file `STEMMUS_SCOPE_run.m` and set the path of `config_file` to `../config_file_crib.txt` and change `WorkDir` and other configurations in `model.setup()`.
-6. Then, run the main script `STEMMUS_SCOPE_run.m`.
+To contribute to the STEMMUS_SCOPE model, you need access to the model source code that is stored in the repository [STEMMUS_SCOPE](https://github.com/EcoExtreML/STEMMUS_SCOPE).

PyStemmusScope/stemmus_scope.py

@@ -2,7 +2,9 @@
 
 import logging
 import os
+import shlex
 import subprocess
+from pathlib import Path
 from typing import Dict
 from . import config_io
 from . import forcing_io
@@ -12,32 +14,99 @@
 
 logger = logging.getLogger(__name__)
 
+def _is_model_src_exe(model_src_path: Path):
+    """Check if input exists. Returns True if input is a file and False if it is
+        a directory.
+
+    Args:
+        model_src_path(Path): path to Stemmus_Scope executable file or to a
+        directory containing model source codes.
+    """
+    if model_src_path.is_file():
+        msg = ("The model executable file can be used on a Unix system "
+            "where MCR is installed, see the "
+            "`documentaion<https://pystemmusscope.readthedocs.io/>`_.")
+        logger.info("%s", msg)
+        return True
+    if model_src_path.is_dir():
+        return False
+    msg = (
+        "Provide a valid path to an executable file or "
+        "to a directory containing model source codes, "
+        "see the `documentaion<https://pystemmusscope.readthedocs.io/>`_.")
+    raise ValueError(msg)
+
+
+def _check_interpreter(interpreter: str):
+    if interpreter not in {"Octave" , "Matlab"}:
+        msg = (
+            "Set `interpreter` as Octave or Matlab to run the model using source codes."
+            "Otherwise set `model_src_path` to the model executable file, "
+            "see the `documentaion<https://pystemmusscope.readthedocs.io/>`_.")
+        raise ValueError(msg)
+
+
+def _run_sub_process(args: list, cwd):
+    # pylint: disable=consider-using-with
+    result = subprocess.Popen(
+        args, cwd=cwd,
+        stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+        shell=True,
+    )
+    exit_code = result.wait()
+    stdout, stderr = result.communicate()
+    #TODO handle stderr properly
+    # when using octave, exit_code might be 139
+    # see issue STEMMUS_SCOPE_Processing/issues/46
+    if exit_code not in [0, 139]:
+        raise subprocess.CalledProcessError(
+            returncode=exit_code, cmd=args, stderr=stderr, output=stdout
+        )
+    if exit_code == 139:
+        logger.warning(stderr)
+
+    # TODO return log info line by line!
+    logger.info(stdout)
+    return stdout.decode('utf-8')
+
 
 class StemmusScope():
     """PyStemmusScope wrapper around Stemmus_Scope model.
     see https://gmd.copernicus.org/articles/14/1379/2021/
 
-    It sets the model with a configuration file and executable file.
-    It also prepares forcing and soil data for model run.
+   Configures the model and prepares forcing and soil data for the model run.
 
     Args:
         config_file(str): path to Stemmus_Scope configuration file. An example
         config_file can be found in tests/test_data in `STEMMUS_SCOPE_Processing
         repository <https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing>`_
-        exe_file(str): path to Stemmus_Scope executable file.
+        model_src_path(str): path to Stemmus_Scope executable file or to a
+        directory containing model source codes.
+        interpreter(str, optional): use `Matlab` or `Octave`. Only required if
+        `model_src_path` is a path to model source codes.
 
     Example:
         See notebooks/run_model_in_notebook.ipynb in `STEMMUS_SCOPE_Processing
         repository <https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing>`_
     """
 
-    def __init__(self, config_file: str, exe_file: str):
+    def __init__(self, config_file: str, model_src_path: str, interpreter: str = None):
         # make sure paths are abolute and path objects
         config_file = utils.to_absolute_path(config_file)
-        self.exe_file = utils.to_absolute_path(exe_file)
+        model_src_path = utils.to_absolute_path(model_src_path)
+
+        # check the path to model source
+        self.exe_file = None
+        if _is_model_src_exe(model_src_path):
+            self.exe_file = model_src_path
+        else:
+            _check_interpreter(interpreter)
+
+        self.model_src = model_src_path
+        self.interpreter = interpreter
 
         # read config template
-        self._configs = config_io.read_config(config_file)
+        self._config = config_io.read_config(config_file)
 
     def setup(
         self,
@@ -61,30 +130,27 @@ def setup(
         """
         # update config template if needed
         if WorkDir:
-            self._configs["WorkDir"] = WorkDir
+            self._config["WorkDir"] = WorkDir
 
         if ForcingFileName:
-            self._configs["ForcingFileName"] = ForcingFileName
+            self._config["ForcingFileName"] = ForcingFileName
 
         if NumberOfTimeSteps:
-            self._configs["NumberOfTimeSteps"] = NumberOfTimeSteps
+            self._config["NumberOfTimeSteps"] = NumberOfTimeSteps
 
         # create customized config file and input/output directories for model run
         _, _, self.cfg_file = config_io.create_io_dir(
-            self._configs["ForcingFileName"], self._configs
+            self._config["ForcingFileName"], self._config
             )
 
         # read the run config file
-        self._configs = config_io.read_config(self.cfg_file)
+        self._config = config_io.read_config(self.cfg_file)
 
         # prepare forcing data
-        forcing_io.prepare_forcing(self._configs)
+        forcing_io.prepare_forcing(self._config)
 
         # prepare soil data
-        soil_io.prepare_soil_data(self._configs)
-
-        # set matlab log dir
-        os.environ['MATLAB_LOG_DIR'] = str(self._configs["InputPath"])
+        soil_io.prepare_soil_data(self._config)
 
         return str(self.cfg_file)
 
@@ -94,23 +160,40 @@ def run(self) -> str:
         Args:
 
         Returns:
-            Tuple with stdout and stderr
+            string, the model log
         """
-
-        # run the model
-        args = [f"{self.exe_file} {self.cfg_file}"]
-        result = subprocess.run(
-            args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, check=True,
-        )
-        stdout = result.stdout
-
-        # TODO return log info line by line!
-        logger.info("%s", stdout)
-
-        return stdout
+        if self.exe_file:
+            # run using MCR
+            args = [f"{self.exe_file} {self.cfg_file}"]
+            # set matlab log dir
+            os.environ['MATLAB_LOG_DIR'] = str(self._config["InputPath"])
+            result = _run_sub_process(args, None)
+        if self.interpreter=="Matlab":
+            # set Matlab arguments
+            path_to_config = f"'{self.cfg_file}'"
+            eval_code= f'STEMMUS_SCOPE_exe({path_to_config});exit;'
+            args = ["matlab", "-r", eval_code, "-nodisplay", "-nosplash", "-nodesktop"]
+            # seperate args dont work on linux!
+            if utils.os_name() !="nt":
+                args = shlex.join(args)
+            result = _run_sub_process(args, self.model_src)
+        if self.interpreter=="Octave":
+            # set Octave arguments
+            # use subprocess instead of oct2py,
+            # see issue STEMMUS_SCOPE_Processing/issues/46
+            path_to_config = f"'{self.cfg_file}'"
+            # fix for windows
+            path_to_config = path_to_config.replace("\\", "/")
+            eval_code = f'STEMMUS_SCOPE_exe({path_to_config});exit;'
+            args = ["octave", "--eval", eval_code, "--no-gui", "--silent"]
+            # seperate args dont work on linux!
+            if utils.os_name() !="nt":
+                args = shlex.join(args)
+            result = _run_sub_process(args, self.model_src)
+        return result
 
 
     @property
     def config(self) -> Dict:
         """Return the configurations for this model."""
-        return self._configs
+        return self._config