Merge branch 'master' into docs/colab-tpu-runtime

Author: jonbrenas

.github/actions/setup-python/action.yaml

@@ -19,4 +19,4 @@ runs:
       shell: bash
       run: |
         poetry env use ${{ inputs.python-version }}
-        poetry install --extras dev
+        poetry install --with dev,test,docs

.gitignore

@@ -1,8 +1,10 @@
 .idea
 .vscode
 __pycache__
+.mypy_cache
 *.pyc
 dist
+.venv/
 .coverage
 coverage.xml
 .ipynb_checkpoints/

CONTRIBUTING.md

@@ -12,9 +12,10 @@ This package provides Python tools for accessing and analyzing genomic data from
 
 You'll need:
 
-- Python 3.10.x (CI-tested version)
-- [Poetry](https://python-poetry.org/) for dependency management
-- [Git](https://git-scm.com/) for version control
+- [pipx](https://pipx.pypa.io/) for installing Python tools
+- [git](https://git-scm.com/) for version control
+
+Both of these can be installed using your distribution's package manager or [Homebrew](https://brew.sh/) on Mac.
 
 ### Initial setup
 
@@ -33,18 +34,31 @@ You'll need:
    git remote add upstream https://github.com/malariagen/malariagen-data-python.git
    ```
 
-3. **Install Poetry** (if not already installed)
+3. **Install Poetry**
 
    ```bash
    pipx install poetry
    ```
 
-4. **Install the project and its dependencies**
+4. **Install Python 3.12**
+
+   Python 3.12 is tested in the CI-system and is the recommended version to use.
+
+   ```bash
+   poetry python install 3.12
+   ```
+
+5. **Install the project and its dependencies**
 
    ```bash
-   poetry install
+   poetry env use 3.12
+   poetry install --with dev,test,docs
    ```
 
+   This installs the runtime dependencies along with the `dev`, `test`, and `docs`
+   [dependency groups](https://python-poetry.org/docs/managing-dependencies/#dependency-groups).
+   If you only need to run tests, `poetry install --with test` is sufficient.
+
    **Recommended**: Use `poetry run` to run commands inside the virtual environment:
 
    ```bash
@@ -71,7 +85,7 @@ You'll need:
    python script.py
    ```
 
-5. **Install pre-commit hooks**
+6. **Install pre-commit hooks**
 
    ```bash
    pipx install pre-commit
@@ -107,16 +121,40 @@ You'll need:
 
 4. **Run tests locally**
 
-   Fast unit tests (no external data access):
+   Fast unit tests using simulated data (no external data access):
 
    ```bash
-   poetry run pytest -v tests/anoph
+   poetry run pytest -v tests --ignore tests/integration
    ```
 
-   All unit tests (requires setting up credentials for legacy tests):
+   To run integration tests which read data from GCS, you'll need to [request access to MalariaGEN data on GCS](https://malariagen.github.io/vector-data/vobs/vobs-data-access.html).
+
+   Once access has been granted, [install the Google Cloud CLI](https://cloud.google.com/sdk/docs/install). E.g., if on Linux:
 
    ```bash
-   poetry run pytest -v tests --ignore tests/integration
+   ./install_gcloud.sh
+   ```
+
+   You'll then need to obtain application-default credentials, e.g.:
+
+   ```bash
+   ./google-cloud-sdk/bin/gcloud auth application-default login
+   ```
+
+   Once this is done, you can run integration tests:
+
+   ```bash
+   poetry run pytest -v tests/integration
+   ```
+
+   Tests will run slowly the first time, as data required for testing will be read from GCS. Subsequent runs will be faster as data will be cached locally in the "gcs_cache" folder.
+
+6. **Run typechecking**
+
+   Run static typechecking with mypy:
+
+   ```bash
+   poetry run mypy malariagen_data tests --ignore-missing-imports
    ```
 
 5. **Check code quality**
@@ -150,7 +188,7 @@ ruff format .
 - **Fast tests**: Unit tests should use simulated data when possible (see `tests/anoph/`)
 - **Integration tests**: Tests requiring GCS data access are slower and run separately
 
-Run type checking with:
+Run dynamic type checking with:
 
 ```bash
 poetry run pytest -v tests --typeguard-packages=malariagen_data,malariagen_data.anoph

LINUX_SETUP.md

@@ -1,65 +1,138 @@
 # Developer setup (Linux)
 
-To get setup for development, see [this video if you prefer VS Code](https://youtu.be/zddl3n1DCFM), or [this older video if you prefer PyCharm](https://youtu.be/QniQi-Hoo9A), and the instructions below.
+## 1. Install Git
+
+Choose the command for your Linux distribution:
+
+**Ubuntu, Debian, and Mint:**
+
+```bash
+sudo apt update
+sudo apt install -y git
+```
+
+**Fedora:**
 
-## 1. Fork and clone this repo
 ```bash
-git clone git@github.com:[username]/malariagen-data-python.git
+sudo dnf install -y git
+```
+
+**Arch Linux:**
+
+```bash
+sudo pacman -S sudo
+sudo pacman -S git
+sudo pacman -S openssh
+```
+
+If your Arch install does not have `sudo` configured yet, run the commands above as `root`, then configure `sudo` for your user.
+
+## 2. Fork and clone this repo
+
+After forking the repository on GitHub, clone your fork.
+
+Use SSH if your SSH keys are set up:
+
+```bash
+git clone git@github.com:[YOUR_GITHUB_USERNAME]/malariagen-data-python.git
 cd malariagen-data-python
 ```
 
-## 2. Install Python
+Use HTTPS if you prefer, or if you do not have SSH keys configured (common on WSL):
+
+```bash
+git clone https://github.com/[YOUR_GITHUB_USERNAME]/malariagen-data-python.git
+cd malariagen-data-python
+```
+
+## 3. Install pipx
+
+Choose the command for your Linux distribution:
+
+**Ubuntu, Debian, and Mint:**
+
+```bash
+sudo apt update
+sudo apt install -y pipx
+pipx ensurepath
+```
+
+**Fedora:**
+
+```bash
+sudo dnf install -y pipx
+pipx ensurepath
+```
+
+**Arch Linux:**
+
 ```bash
-sudo add-apt-repository ppa:deadsnakes/ppa
-sudo apt install python3.10 python3.10-venv
+sudo pacman -S python-pipx
+pipx ensurepath
 ```
 
-## 3. Install pipx and poetry
+Close and reopen your terminal to apply PATH changes.
+If you prefer to reload the shell in-place, run:
+
+```bash
+exec bash
+```
+
+## 4. Install Poetry and Python 3.12
+
+The package requires `>=3.10,<3.13`. We use Poetry's built-in installer to handle the Python version universally across all distributions.
+
 ```bash
-python3.10 -m pip install --user pipx
-python3.10 -m pipx ensurepath
 pipx install poetry
+poetry python install 3.12
 ```
 
-## 4. Create and activate development environment
+## 5. Create development environment
+
 ```bash
-poetry install
-poetry shell
+poetry env use 3.12
+poetry install --extras dev
 ```
 
-## 5. Install pre-commit hooks
+## 6. Install pre-commit hooks
+
 ```bash
 pipx install pre-commit
 pre-commit install
 ```
 
 Run pre-commit checks manually:
+
 ```bash
 pre-commit run --all-files
 ```
 
-## 6. Run tests
+## 7. Run tests
 
 Run fast unit tests using simulated data:
+
 ```bash
 poetry run pytest -v tests/anoph
 ```
 
-## 7. Google Cloud authentication (for legacy tests)
+## 8. Google Cloud authentication (for legacy tests)
 
 To run legacy tests which read data from GCS, you'll need to [request access to MalariaGEN data on GCS](https://malariagen.github.io/vector-data/vobs/vobs-data-access.html).
 
 Once access has been granted, [install the Google Cloud CLI](https://cloud.google.com/sdk/docs/install):
+
 ```bash
 ./install_gcloud.sh
 ```
 
 Then obtain application-default credentials:
+
 ```bash
 ./google-cloud-sdk/bin/gcloud auth application-default login
 ```
 
 Once authenticated, run legacy tests:
+
 ```bash
 poetry run pytest --ignore=tests/anoph -v tests
 ```

README.md

@@ -49,6 +49,8 @@ To get setup for development, see [this video if you prefer VS Code](https://you
 For detailed setup instructions, see:
 - [Linux setup guide](LINUX_SETUP.md)
 - [macOS setup guide](MACOS_SETUP.md)
+- [Windows setup guide](WINDOWS_SETUP.md)
+- [Google Colab (TPU) setup guide](docs/source/colab_tpu_runtime.rst)
 Detailed instructions can be found in the [Contributors guide](https://github.com/malariagen/malariagen-data-python/blob/master/CONTRIBUTING.md).
 
 ## AI use policy and guidelines

WINDOWS_SETUP.md

@@ -0,0 +1,90 @@
+# Windows Setup Guide
+
+To get setup for development on Windows, see
+[this video if you prefer VS Code](https://youtu.be/zddl3n1DCFM),
+or [this older video if you prefer PyCharm](https://youtu.be/QniQi-Hoo9A),
+and the instructions below.
+
+## 1. Fork and clone this repo
+```bash
+git clone https://github.com/[username]/malariagen-data-python.git
+cd malariagen-data-python
+```
+
+## 2. Install Python
+
+Download and install Python 3.10 from the official website:
+https://www.python.org/downloads/windows/
+
+During installation, check the box that says Add Python to PATH
+before clicking Install.
+
+Verify the installation worked:
+```bash
+python --version
+```
+
+## 3. Install pipx and poetry
+```bash
+python -m pip install --user pipx
+python -m pipx ensurepath
+pipx install poetry
+```
+
+After running ensurepath, close and reopen PowerShell before continuing.
+
+## 4. Create and activate development environment
+```bash
+poetry install
+poetry shell
+```
+
+## 5. Install pre-commit hooks
+```bash
+pipx install pre-commit
+pre-commit install
+```
+
+## 6. Add upstream remote and get latest code
+```bash
+git remote add upstream https://github.com/malariagen/malariagen-data-python
+git pull upstream master
+```
+
+Note: On Windows the default branch is called master, not main.
+
+## 7. Verify everything works
+```bash
+python -c "import malariagen_data; print('Setup successful!')"
+```
+
+## Common Issues on Windows
+
+**poetry not found after install**
+
+Close and reopen PowerShell, then try again.
+
+**git not recognized**
+
+Install Git from https://git-scm.com/download/win
+and restart PowerShell.
+
+**python not recognized**
+
+Reinstall Python and make sure to check
+Add Python to PATH during installation.
+
+**fatal: not a git repository**
+
+Make sure you are inside the malariagen-data-python
+folder before running any git commands.
+```bash
+cd malariagen-data-python
+```
+
+**error: pathspec main did not match**
+
+On Windows use master instead of main.
+```bash
+git checkout master
+```