Skip to content

Commit 38a05d3

Browse files
SarahAlidoostSarahAlidoost
authored
Merge pull request #32 from EcoExtreML/remove_matlab_script
Remove matlab scripts and revise documentations
2 parents 85f6e66 + aef68e7 commit 38a05d3

8 files changed

Lines changed: 179 additions & 202 deletions

‎CONTRIBUTING.md‎

Lines changed: 108 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -1,86 +1,144 @@
11
# Contributing Guide
22

3-
This repositary includes python modules for running the STEMMUS-SCOPE model in a
4-
notebook.
3+
This repository includes the python package `PyStemmusScope` for running the STEMMUS-SCOPE model.
54

6-
The workflow is executed using python and MATLAB on a Unix-like system.
7-
The python packages are listed in the
8-
[`environment.yml`](https://github.com/EcoExtreML/processing/blob/main/environment.yml)
9-
file. Follow the instructions below to create conda environment and install
10-
MATLAB Runtime.
5+
## Configure the python package for development and testing
6+
7+
To contribute to development of the python package, we recommend installing the package in development mode.
8+
9+
10+
### Installation
11+
12+
First, clone this repository:
13+
14+
```sh
15+
git clone https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing.git
16+
```
17+
18+
Then install the package:
19+
20+
```sh
21+
cd STEMMUS_SCOPE_Processing
22+
pip install -e .
23+
```
24+
25+
or
26+
27+
```sh
28+
python setup.py develop
29+
```
30+
31+
### Run tests
32+
33+
The testing framework used here is [PyTest](https://pytest.org). You can run
34+
tests as:
35+
36+
```sh
37+
pytest
38+
```
39+
40+
### Build documentation
41+
42+
To build the documentation locally:
43+
44+
```sh
45+
cd docs/
46+
make html
47+
```
48+
49+
Then open `_build/html/index.html` in a web broser to preview the documentation.
50+
51+
### Run formatting tools
52+
53+
You can use `prospector` to get information about errors, potential problems and convention violations. To run:
54+
55+
```sh
56+
prospector
57+
```
58+
59+
To format the import statements, you can use `isort` as:
60+
61+
```sh
62+
isort
63+
```
64+
65+
## Development of STEMMUS_SCOPE model
66+
67+
<!-- markdown-link-check-disable-next-line -->
68+
To contribute to the STEMMUS_SCOPE model, you need access to the model source code that is stored in the repository [STEMMUS_SCOPE](https://github.com/EcoExtreML/STEMMUS_SCOPE). You also need a MATLAB license.
69+
70+
### Development on Snellius using MATLAB
71+
72+
[Snellius](https://servicedesk.surfsara.nl/wiki/display/WIKI/Snellius) is the
73+
Dutch National supercomputer hosted at SURF. MATLAB `2021a` is installed on
74+
Snellius, see the script
75+
[`run_jupyter_lab_snellius_dev.sh`](https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/blob/main/run_jupyter_lab_snellius_dev.sh)
76+
on how to load the module.
77+
78+
The script `run_jupyter_lab_snellius_dev.sh` activates the conda environment `pystemmusscope` and creates a jupyter lab server on Snellius for running the notebook
79+
interactively. Make sure that you create the `pystemmusscope` conda environment before submitting the the bash script. See **Create pystemmusscope environment** below.
1180

1281
<details>
13-
<summary>Create conda environment </summary>
82+
<summary>Create pystemmusscope environment</summary>
1483

1584
Run the commands below in a terminal:
1685

1786
```sh
18-
# Download and install Conda
87+
# Download and install Mamba on linux
1988
wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-pypy3-Linux-x86_64.sh
20-
bash Mambaforge-pypy3-Linux-x86_64.sh
21-
-b -p ~/mamba
89+
bash Mambaforge-pypy3-Linux-x86_64.sh -b -p ~/mamba
2290

2391
# Update base environment
2492
. ~/mamba/bin/activate
2593
mamba update --name base mamba
2694

27-
# Clone this repository
28-
git clone https://github.com/EcoExtreML/processing.git
95+
# Download environment file
96+
wget https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/blob/main/environment.yml
2997

30-
# Create a conda environment called 'stemmus' with all required dependencies
31-
cd processing
32-
mamba env create
98+
# Create a conda environment called 'pystemmusscope' with all required dependencies
99+
mamba env create -f environment.yml
33100

34101
# The environment can be activated with
35-
. ~/mamba/bin/activate stemmus
102+
. ~/mamba/bin/activate pystemmusscope
36103

37104
```
38105
</details>
39106

40-
<details>
41-
<summary>Use MATLAB </summary>
42107

43-
To run the STEMMUS_SCOPE, you need MATLAB version `>=2019`.
108+
### Development on CRIB using MATLAB
44109

45-
**On Snellius:**
110+
[CRIB](https://crib.utwente.nl/) is the ITC Geospatial Computing Platform.
46111

47-
[Snellius](https://servicedesk.surfsara.nl/wiki/display/WIKI/Snellius) is the
48-
Dutch National supercomputer hosted at SURF. MATLAB Runtime is installed on
49-
Snellius, see the script
50-
[`run_jupyter_lab_snellius_dev.sh`](https://github.com/EcoExtreML/processing/blob/main/run_jupyter_lab_snellius_dev.sh)
51-
on how to load the module.
52-
</details>
112+
MATLAB `2021a` is installed on CRIB, and supports Python `3.8`, see [Versions of Python Compatible with MATLAB Products](https://www.mathworks.com/content/dam/mathworks/mathworks-dot-com/support/sysreq/files/python-compatibility.pdf).
53113

54-
## Run jupyter notebook
114+
1. Log in CRIB with your username and password and select a proper compute unit.
115+
2. Install `PyStemmusScope` package. This step needs to be done once.
116+
<details>
117+
<summary>Install pystemmusscope with python 3.8</summary>
55118

56-
**On Snellius:**
119+
Run the commands below in a terminal:
57120

58-
Use the script
59-
[`run_jupyter_lab_snellius_dev.sh`](https://github.com/EcoExtreML/processing/blob/main/run_jupyter_lab_snellius_dev.sh)
60-
to create a jupyter lab server on Snellius for running the notebook
61-
interactively.
121+
```sh
122+
# Download and install Mamba on linux
123+
wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-pypy3-Linux-x86_64.sh
124+
bash Mambaforge-pypy3-Linux-x86_64.sh -b -p ~/mamba
62125

63-
**On CRIB:**
126+
# Update base environment
127+
. ~/mamba/bin/activate
128+
mamba update --name base mamba
64129

65-
[CRIB](https://crib.utwente.nl/) is the ITC Geospatial Computing Platform.
130+
# Download environment file
131+
wget https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/blob/main/environment_3.8.yml
132+
133+
# Create a conda environment called 'pystemmusscope' with all required dependencies
134+
mamba env create -f environment_3.8.yml
135+
```
136+
</details>
66137

67-
1. Log in CRIB with your username and password and select a proper compute unit.
68-
2. Check `config_file_crib.txt` and change the paths if needed, specifically
69-
"InputPath" and "OutputPath".
70138
3. click on the `Remote Desktop` in the
71139
Launcher. Click on the `Applications`. You will find the 'MATLAB' software under
72140
the `Research`.
73141
4. After clicking on 'MATLAB', it asks for your account information that is
74142
connected to a MATLAB license.
75-
5. Open the file `run_model_in_matlab_dev.m` and set the paths inside the script.
76-
6. Then, run the main script `run_model_in_matlab_dev.m`.
77-
78-
## Recipe of model execution
79-
80-
The execution of the model includes following steps:
81-
82-
- Update/set config files
83-
- Create input directories, prepare input files
84-
- Run the model
85-
- Create output directories, prepare output files
86-
- Create exe file
143+
5. Open the file `STEMMUS_SCOPE_run.m` and set the path of `config_file` to `../config_file_crib.txt` and change `WorkDir` and other configurations in `model.setup()`.
144+
6. Then, run the main script `STEMMUS_SCOPE_run.m`.

‎README.md‎

Lines changed: 41 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -17,38 +17,37 @@
1717
[![cffconvert](https://github.com/EcoExtreML/stemmus_scope_processing/actions/workflows/cffconvert.yml/badge.svg)](https://github.com/EcoExtreML/stemmus_scope_processing/actions/workflows/cffconvert.yml)
1818
[![markdown-link-check](https://github.com/EcoExtreML/stemmus_scope_processing/actions/workflows/markdown-link-check.yml/badge.svg)](https://github.com/EcoExtreML/stemmus_scope_processing/actions/workflows/markdown-link-check.yml) -->
1919

20-
This repositary includes python modules for running the STEMMUS-SCOPE model in a jupyter
21-
notebook.
20+
This repository includes the python package `PyStemmusScope` for running the STEMMUS-SCOPE model.
21+
<!-- markdown-link-check-disable-next-line -->
22+
The model source code is in MATLAB and available in the [STEMMUS_SCOPE repository](https://github.com/EcoExtreML/STEMMUS_SCOPE). See the relevant
23+
instructions for `Users` or `Developers` on how to run the model.
2224

23-
The workflow is executed using python and MATLAB Runtime on a Unix-like system.
24-
The python packages are listed in the
25-
[`environment.yml`](https://github.com/EcoExtreML/processing/blob/main/environment.yml)
26-
file. Follow the instructions below to create conda environment and install
27-
MATLAB Runtime.
25+
## Users
26+
27+
As a user, you don't need to have a MATLAB license to run the STEMMUS-SCOPE model. The workflow is executed using python and MATLAB Runtime on a Unix-like system.
28+
29+
### Installations
30+
31+
Follow the instructions below to install `PyStemmusScope`, `jupyterlab` and MATLAB Runtime.
2832

2933
<details>
30-
<summary>Create conda environment </summary>
34+
<summary>Install PyStemmusScope</summary>
3135

3236
Run the commands below in a terminal:
3337

3438
```sh
35-
# Download and install Conda
36-
wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-pypy3-Linux-x86_64.sh
37-
bash Mambaforge-pypy3-Linux-x86_64.sh -b -p ~/mamba
38-
39-
# Update base environment
40-
. ~/mamba/bin/activate
41-
mamba update --name base mamba
39+
# will be replaced by `pip install pystemmusscope`
40+
pip install git+https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing.git@main
41+
```
4242

43-
# Clone this repository
44-
git clone https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing.git
43+
</details>
4544

46-
# Create a conda environment called 'stemmus' with all required dependencies
47-
cd STEMMUS_SCOPE_Processing
48-
mamba env create
45+
<details>
46+
<summary>Install jupyterlab</summary>
47+
Jupyterlab is needed to run notebooks. Run the commands below in a terminal:
4948

50-
# The environment can be activated with
51-
. ~/mamba/bin/activate stemmus
49+
```sh
50+
pip install jupyterlab
5251

5352
```
5453
</details>
@@ -65,73 +64,50 @@ In a terminal:
6564
wget https://ssd.mathworks.com/supportfiles/downloads/R2021a/Release/6/deployment_files/installer/complete/glnxa64/MATLAB_Runtime_R2021a_Update_6_glnxa64.zip
6665

6766
# Unzip the file
68-
unzip MATLAB_Runtime_R2021a_Update_6_glnxa64.zip
67+
unzip MATLAB_Runtime_R2021a_Update_6_glnxa64.zip -d MATLAB_Runtime
6968

7069
# Install it
71-
cd MATLAB_Runtime_R2021a_Update_6_glnxa64
70+
cd MATLAB_Runtime
7271
sudo -H ./install -mode silent -agreeToLicense yes
7372
```
7473

75-
For more information on how to download and install it, see the links below:
76-
- [download](https://nl.mathworks.com/products/compiler/matlab-runtime.html)
77-
- [intallation](https://nl.mathworks.com/help/compiler/install-the-matlab-runtime.html)
78-
79-
**On Snellius:**
74+
For more information on how to download and install MATLAB Runtime, see the links below:
75+
- [download](https://nl.mathworks.com/products/compiler/matlab-runtime.html)
76+
- [installation](https://nl.mathworks.com/help/compiler/install-the-matlab-runtime.html)
8077

81-
[Snellius](https://servicedesk.surfsara.nl/wiki/display/WIKI/Snellius) is the
82-
Dutch National supercomputer hosted at SURF. MATLAB Runtime is installed on
83-
Snellius, see the script
84-
[`run_jupyter_lab_snellius.sh`](https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/blob/main/run_jupyter_lab_snellius.sh)
85-
on how to load the module.
8678
</details>
8779

88-
## Run jupyter notebook
89-
90-
Open a terminal and run:
80+
Open a terminal, make sure the environment is activated. Then, run `jupyter lab`:
9181

9282
```sh
9383
jupyter lab
9484
```
9585

96-
JupyterLab will open automatically in your browser.
86+
JupyterLab will open automatically in your browser. Now, you can run the notebook [run_model_in_notebook.ipynb](https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/blob/main/notebooks/run_model_in_notebook.ipynb).
9787

98-
**On Snellius:**
88+
### On Snellius
9989

100-
Use the script
90+
[Snellius](https://servicedesk.surfsara.nl/wiki/display/WIKI/Snellius) is the
91+
Dutch National supercomputer hosted at SURF. MATLAB Runtime is installed on
92+
Snellius, see the script
93+
[`run_jupyter_lab_snellius.sh`](https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/blob/main/run_jupyter_lab_snellius.sh)
94+
on how to load the module. Also, use the script
10195
[`run_jupyter_lab_snellius.sh`](https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/blob/main/run_jupyter_lab_snellius.sh)
10296
to create a jupyter lab server on Snellius for running the notebook
97+
[run_model_in_notebook.ipynb](https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/blob/main/notebooks/run_model_in_notebook.ipynb)
10398
interactively.
10499

105-
**On CRIB:**
100+
### On CRIB
106101

107102
[CRIB](https://crib.utwente.nl/) is the ITC Geospatial Computing Platform.
108-
Currently, running the notebook on CRIB is not supported because MATLAB Runtime
109-
can not be installed there.
110-
111-
## Recipe of model execution
112-
113-
The execution of the model includes following steps:
114-
115-
- Update/set config files
116-
- Create input directories, prepare input files
117-
- Run the model
118-
- Create output directories, prepare output files
119-
120-
## Configure the package for development and testing
121-
The testing framework used here is [PyTest](https://pytest.org). Before running the test, the package need to be installed and configured as via the command:
122-
123-
```py
124-
pip install -e .
125-
```
126-
or
127-
```py
128-
python setup.py develop
129-
```
103+
Currently, running the model exceutable file and therefore the notebook
104+
[run_model_in_notebook.ipynb](https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/blob/main/notebooks/run_model_in_notebook.ipynb)
105+
on CRIB is not supported because MATLAB Runtime can not be installed there.
130106

131-
## Contributing
107+
## Developers
132108

133109
If you want to contribute to the development of PyStemmusScope,
134-
have a look at the [contribution guidelines](CONTRIBUTING.md).
110+
have a look at the [contribution guidelines](https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing/blob/main/CONTRIBUTING.md).
135111

136112
## Credits
137113

‎environment.yml‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
---
2-
name: stemmus
2+
name: pystemmusscope
33
channels:
44
- conda-forge
55
dependencies:
6-
- python=3.9
7-
- netcdf4>=1.5.8,<2
8-
- pandas
9-
- jupyterlab
6+
- pip
7+
- python>=3.8
8+
- jupyterlab # needed to run notebooks
9+
- pip:
10+
- git+https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing.git@main # will be replaced by `pip install pystemmusscope`

‎environment_3.8.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
name: pystemmusscope_3.8
3+
channels:
4+
- conda-forge
5+
dependencies:
6+
- pip
7+
- python=3.8 # supported by matlab R2021a, see issue #31
8+
- jupyterlab # needed to run notebooks
9+
- pip:
10+
- git+https://github.com/EcoExtreML/STEMMUS_SCOPE_Processing.git@main # will be replaced by `pip install pystemmusscope`

0 commit comments

Comments
 (0)