Merge branch 'master' into GH907-n-jack-validation

Author: adilraza99

.github/workflows/integration_tests.yml

@@ -3,9 +3,6 @@ on:
   push:
     branches:
       - master
-  pull_request:
-    branches:
-      - master
 jobs:
   integration_tests:
     strategy:

.github/workflows/latest_docs.yml

@@ -30,7 +30,7 @@ jobs:
         run: poetry run sphinx-build -b html docs/source docs/build/html
 
       - name: Deploy HTML to GitHub Pages 🚀
-        uses: peaceiris/actions-gh-pages@v3.9.3
+        uses: peaceiris/actions-gh-pages@v4
         with:
           publish_branch: gh-pages
           github_token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/notebooks.yml

@@ -3,9 +3,6 @@ on:
   push:
     branches:
       - master
-  pull_request:
-    branches:
-      - master
 jobs:
   notebooks:
     strategy:

AI-POLICY.md

@@ -0,0 +1,27 @@
+# AI use policy and guidelines
+
+The goal of the MalariaGEN data API is to make access, use, and interpretation of the genomic data collected by our partners as easy and intuitive as possible. Maintainers have limited time and attention to focus on reviews, which means that each review request has to be for code that you can be proud of.
+
+Any tool that can help produce better code and understand better the existing codebase, including AI tools, can be used. The only key questions are: “Is this an improvement?” and “Why is the code better now?”.
+
+NEVER submit an AI-generated PR if you are not able to understand and explain the changes and why they matter. Maintainers WILL close PRs without reviewing them if they feel like they are a waste of time.
+
+## Using AI as a coding assistant
+
+1.	Understanding and familiarising yourself with the codebase is key. No matter how good the AI code assistant, it will return useless code if you do not provide a smart and accurate enough prompt.
+2.	Always check that your changes make sense. LLMs are terrible at saying no to a prompt and will lie and make false claims if they can’t do otherwise. It is particularly true if they lack key information.
+3.	Each commit should be its own piece of coherent change. LLMs like to do everything at once but digestible change is easier to understand and process.
+4.	Commenting your code is important, but LLMs really like to listen to themselves talking and will be very verbose. A small comment explaining why you made a choice is better than a paragraph explaining how a loop iterates through a list.
+
+## Using AI for communication
+
+As noted above, maintainers have a limited amount of time to spend on malariaGEN data API maintenance and do not want to waste it going through long, sloppy PR descriptions of simple issue. We strongly prefer clear and concise communication, even if it means we have to ask questions when more details are needed.
+
+You are responsible for your own PRs and comments. Even if you use an LLM to write a PR description or comment, you are expected to read through everything and make sure that it accurately and concisely reflects your opinions, ideas and contributions. If reading your own PRs and comments is too much work for you, it is going to be the same for everyone else.
+Here are some concrete guidelines for using AI as part of your communication toolbox.
+
+1.	In general, the question that needs answering is why not what. Maintainers can see the files and lines of codes that were modified, what they will want to know is the reasoning behind the choices. Sadly, LLMs are not great at explaining their reasoning so you probably will have to chip in.
+2.	In the same way, if you are responding to a comment or a review, you will need to justify your choice and explain how you made the decision.
+3.	Make sure that the description of your work is accurate. Errors can happen but it is fairly obvious when an LLM claims more than it delivers.
+4.	We are aware that English is not everyone’s first language. The grammar of your communications isn’t as important as the quality of your contribution. Feel free to use AI to improve your writing style but make sure that you still understand the message, that its content is conserved and that it doesn’t turn into an epic poem.
+5.	Maintainers are more interested in your ideas and thoughts than in the standard answer provided by an LLM. We work with genomic data, and contributors are not expected to be experts in computer science, software engineering, genomics, entomology, …  You are allowed not to know or not to be sure and it is miles better to say so than it is to regurgitate an answer that you do not understand.

CONTRIBUTING.md

@@ -0,0 +1,229 @@
+# Contributing to malariagen-data-python
+
+Thanks for your interest in contributing to this project! This guide will help you get started.
+
+## About the project
+
+This package provides Python tools for accessing and analyzing genomic data from [MalariaGEN](https://www.malariagen.net/), a global research network studying the genomic epidemiology of malaria and its vectors. It provides access to data on _Anopheles_ mosquito species and _Plasmodium_ malaria parasites, with functionality for variant analysis, haplotype clustering, population genetics, and visualization.
+
+## Setting up your development environment
+
+### Prerequisites
+
+You'll need:
+
+- Python 3.10.x (CI-tested version)
+- [Poetry](https://python-poetry.org/) for dependency management
+- [Git](https://git-scm.com/) for version control
+
+### Initial setup
+
+1. **Fork and clone the repository**
+
+   Fork the repository on GitHub, then clone your fork:
+
+   ```bash
+   git clone git@github.com:[your-username]/malariagen-data-python.git
+   cd malariagen-data-python
+   ```
+
+2. **Add the upstream remote**
+
+   ```bash
+   git remote add upstream https://github.com/malariagen/malariagen-data-python.git
+   ```
+
+3. **Install Poetry** (if not already installed)
+
+   ```bash
+   pipx install poetry
+   ```
+
+4. **Install the project and its dependencies**
+
+   ```bash
+   poetry install
+   ```
+
+   **Recommended**: Use `poetry run` to run commands inside the virtual environment:
+
+   ```bash
+   poetry run pytest
+   poetry run python script.py
+   ```
+
+   **Optional**: If you prefer an interactive shell session, install the shell plugin first:
+
+   ```bash
+   poetry self add poetry-plugin-shell
+   ```
+
+   Then activate the environment with:
+
+   ```bash
+   poetry shell
+   ```
+
+   After activation, commands run directly inside the virtual environment:
+
+   ```bash
+   pytest
+   python script.py
+   ```
+
+5. **Install pre-commit hooks**
+
+   ```bash
+   pipx install pre-commit
+   pre-commit install
+   ```
+
+   Pre-commit hooks will automatically run `ruff` (linter and formatter) on your changes before each commit.
+
+## Development workflow
+
+### Creating a new feature or fix
+
+1. **Sync with upstream**
+
+   ```bash
+   git checkout master
+   git pull upstream master
+   ```
+
+2. **Create a feature branch**
+
+   If an issue does not already exist for your change, [create one](https://github.com/malariagen/malariagen-data-python/issues/new) first. Then create a branch using the convention `GH{issue number}-{short description}`:
+
+   ```bash
+   git checkout -b GH123-fix-broken-filter
+   # or
+   git checkout -b GH456-add-new-analysis
+   ```
+
+3. **Make your changes**
+
+   Write your code, add tests, update documentation as needed.
+
+4. **Run tests locally**
+
+   Fast unit tests (no external data access):
+
+   ```bash
+   poetry run pytest -v tests/anoph
+   ```
+
+   All unit tests (requires setting up credentials for legacy tests):
+
+   ```bash
+   poetry run pytest -v tests --ignore tests/integration
+   ```
+
+5. **Check code quality**
+
+   The pre-commit hooks will run automatically, but you can also run them manually:
+
+   ```bash
+   pre-commit run --all-files
+   ```
+
+### Code style
+
+We use `ruff` for both linting and formatting. The configuration is in `pyproject.toml`. Key points:
+
+- Line length: 88 characters (black default)
+- Follow PEP 8 conventions
+- Use type hints where appropriate
+- Write clear docstrings (we use numpydoc format)
+
+The pre-commit hooks will handle most formatting automatically. If you want to run ruff manually:
+
+```bash
+ruff check .
+ruff format .
+```
+
+### Testing
+
+- **Write tests for new functionality**: Add unit tests in the `tests/` directory
+- **Test coverage**: Aim to maintain or improve test coverage
+- **Fast tests**: Unit tests should use simulated data when possible (see `tests/anoph/`)
+- **Integration tests**: Tests requiring GCS data access are slower and run separately
+
+Run type checking with:
+
+```bash
+poetry run pytest -v tests --typeguard-packages=malariagen_data,malariagen_data.anoph
+```
+
+### Documentation
+
+- Update docstrings if you modify public APIs
+- Documentation is built using Sphinx with the pydata theme
+- API docs are auto-generated from docstrings
+- Follow the [numpydoc](https://numpydoc.readthedocs.io/) style guide
+
+## Submitting your contribution
+
+### Before opening a pull request
+
+- [ ] Tests pass locally
+- [ ] Pre-commit hooks pass (or run `pre-commit run --all-files`)
+- [ ] Code is well-documented
+- [ ] Commit messages are clear and descriptive
+
+### Opening a pull request
+
+1. **Push your branch**
+
+   ```bash
+   git push origin your-branch-name
+   ```
+
+2. **Create the pull request**
+   - Go to the [repository on GitHub](https://github.com/malariagen/malariagen-data-python)
+   - Click "Pull requests" → "New pull request"
+   - Select your fork and branch
+   - Write a clear title and description
+
+3. **Pull request description should include:**
+   - What problem does this solve?
+   - How does it solve it?
+   - Any relevant issue numbers (e.g., "Fixes #123")
+   - Testing done
+   - Any breaking changes or migration notes
+
+### Review process
+
+- PRs require approval from a project maintainer
+- CI tests must pass (pytest on Python 3.10 with NumPy 1.26.4)
+- Address review feedback by pushing new commits to your branch
+- Once approved, a maintainer will merge your PR
+
+## Communication
+
+- **Issues**: Use [GitHub Issues](https://github.com/malariagen/malariagen-data-python/issues) for bug reports and feature requests
+- **Discussions**: For questions and general discussion, use [GitHub Discussions](https://github.com/malariagen/malariagen-data-python/discussions)
+- **Pull requests**: Use PR comments for code review discussions
+- **Email**: For data access questions, contact [support@malariagen.net](mailto:support@malariagen.net)
+
+## Finding something to work on
+
+- Look for issues labeled [`good first issue`](https://github.com/malariagen/malariagen-data-python/labels/good%20first%20issue)
+- Check for issues labeled [`help wanted`](https://github.com/malariagen/malariagen-data-python/labels/help%20wanted)
+- Improve documentation or add examples
+- Increase test coverage
+
+## Questions?
+
+If you're unsure about anything, feel free to:
+
+- Open an issue to ask
+- Start a discussion on GitHub Discussions
+- Ask in your pull request
+
+We appreciate your contributions and will do our best to help you succeed!
+
+## License
+
+By contributing to this project, you agree that your contributions will be licensed under the [MIT License](LICENSE).

README.md

@@ -46,88 +46,11 @@ for release notes.
 
 To get setup for development, see [this video if you prefer VS Code](https://youtu.be/zddl3n1DCFM), or [this older video if you prefer PyCharm](https://youtu.be/QniQi-Hoo9A), and the instructions below.
 
-Fork and clone this repo:
+Detailed instructions can be found in the [Contributors guide](https://github.com/malariagen/malariagen-data-python/blob/master/CONTRIBUTING.md).
 
-```bash
-git clone git@github.com:[username]/malariagen-data-python.git
-```
-
-Install Python, e.g.:
-
-```bash
-sudo add-apt-repository ppa:deadsnakes/ppa
-sudo apt install python3.10 python3.10-venv
-```
-
-Install pipx, e.g.:
-
-```bash
-python3.10 -m pip install --user pipx
-python3.10 -m pipx ensurepath
-```
-
-Install [poetry](https://python-poetry.org/docs/#installation), e.g.:
-
-```bash
-pipx install poetry
-```
-
-Create development environment:
-
-```bash
-cd malariagen-data-python
-poetry use 3.10
-poetry install
-```
-
-Activate development environment:
-
-```bash
-poetry shell
-```
-
-Install pre-commit and pre-commit hooks:
-
-```bash
-pipx install pre-commit
-pre-commit install
-```
-
-Run pre-commit checks (isort, black, blackdoc, flake8, ...) manually:
-
-```bash
-pre-commit run --all-files
-```
-
-Run fast unit tests using simulated data:
-
-```bash
-poetry run pytest -v tests/anoph
-```
-
-To run legacy tests which read data from GCS, you'll need to [request access to MalariaGEN data on GCS](https://malariagen.github.io/vector-data/vobs/vobs-data-access.html).
-
-Once access has been granted, [install the Google Cloud CLI](https://cloud.google.com/sdk/docs/install). E.g., if on Linux:
-
-```bash
-./install_gcloud.sh
-```
-
-You'll then need to obtain application-default credentials, e.g.:
-
-```bash
-./google-cloud-sdk/bin/gcloud auth application-default login
-```
-
-Once this is done, you can run legacy tests:
-
-```bash
-poetry run pytest --ignore=tests/anoph -v tests
-```
+## AI use policy and guidelines
 
-Tests will run slowly the first time, as data required for testing
-will be read from GCS. Subsequent runs will be faster as data will be
-cached locally in the "gcs_cache" folder.
+See [AI use policy and guidelines](https://github.com/malariagen/malariagen-data-python/blob/master/AI-POLICY.md) for more details.
 
 ## Release process
 
@@ -142,7 +65,7 @@ modifying the `docs/source/_static/switcher.json` file accordingly.
 
 If you use the `malariagen_data` package in a publication
 or include any of its functions or code in other materials (_e.g._ training resources),
-please cite: [doi.org/10.5281/zenodo.11173411](doi.org/10.5281/zenodo.11173411)
+please cite: [doi.org/10.5281/zenodo.11173411](https://doi.org/10.5281/zenodo.11173411)
 
 Some functions may require additional citations to acknowledge specific contributions. These are indicated in the description for each relevant function.
 

docs/source/Ag3.rst

@@ -15,7 +15,7 @@ All the functions below can then be accessed as methods on the ``ag3`` object. E
     df_samples = ag3.sample_metadata()
 
 For more information about the data and terms of use, please see the
-`MalariaGEN Anopheles gambiae genomic surveillance project <https://www.malariagen.net/anopheles-gambiae-genomic-surveillance-project>`_
+`MalariaGEN Anopheles gambiae genomic surveillance project <https://www.malariagen.net/project/anopheles-gambiae-genomic-surveillance-project>`_
 home page.
 
 .. currentmodule:: malariagen_data.ag3.Ag3