Merge branch 'master' into GH856-AI-policy-document

Author: jonbrenas

CONTRIBUTING.md

@@ -0,0 +1,238 @@
+# Contributing to malariagen-data-python
+
+Thanks for your interest in contributing to this project! This guide will help you get started.
+
+## About the project
+
+This package provides Python tools for accessing and analyzing genomic data from [MalariaGEN](https://www.malariagen.net/), a global research network studying the genomic epidemiology of malaria and its vectors. It provides access to data on _Anopheles_ mosquito species and _Plasmodium_ malaria parasites, with functionality for variant analysis, haplotype clustering, population genetics, and visualization.
+
+## Setting up your development environment
+
+### Prerequisites
+
+You'll need:
+
+- Python 3.10.x (CI-tested version)
+- [Poetry](https://python-poetry.org/) for dependency management
+- [Git](https://git-scm.com/) for version control
+
+### Initial setup
+
+1. **Fork and clone the repository**
+
+   Fork the repository on GitHub, then clone your fork:
+
+   ```bash
+   git clone git@github.com:[your-username]/malariagen-data-python.git
+   cd malariagen-data-python
+   ```
+
+2. **Add the upstream remote**
+
+   ```bash
+   git remote add upstream https://github.com/malariagen/malariagen-data-python.git
+   ```
+
+3. **Install Poetry** (if not already installed)
+
+   ```bash
+   pipx install poetry
+   ```
+
+4. **Install the project and its dependencies**
+
+   ```bash
+   poetry install
+   ```
+
+   **Recommended**: Use `poetry run` to run commands inside the virtual environment:
+
+   ```bash
+   poetry run pytest
+   poetry run python script.py
+   ```
+
+   **Optional**: If you prefer an interactive shell session, install the shell plugin first:
+
+   ```bash
+   poetry self add poetry-plugin-shell
+   ```
+
+   Then activate the environment with:
+
+   ```bash
+   poetry shell
+   ```
+
+   After activation, commands run directly inside the virtual environment:
+
+   ```bash
+   pytest
+   python script.py
+   ```
+
+5. **Install pre-commit hooks**
+
+   ```bash
+   pipx install pre-commit
+   pre-commit install
+   ```
+
+   Pre-commit hooks will automatically run `ruff` (linter and formatter) on your changes before each commit.
+
+## Development workflow
+
+### Creating a new feature or fix
+
+1. **Sync with upstream**
+
+   ```bash
+   git checkout master
+   git pull upstream master
+   ```
+
+2. **Create a feature branch**
+
+   If an issue does not already exist for your change, [create one](https://github.com/malariagen/malariagen-data-python/issues/new) first. Then create a branch using the convention `GH{issue number}-{short description}`:
+
+   ```bash
+   git checkout -b GH123-fix-broken-filter
+   # or
+   git checkout -b GH456-add-new-analysis
+   ```
+
+3. **Make your changes**
+
+   Write your code, add tests, update documentation as needed.
+
+4. **Run tests locally**
+
+   Fast unit tests (no external data access):
+
+   ```bash
+   poetry run pytest -v tests/anoph
+   ```
+
+   All unit tests (requires setting up credentials for legacy tests):
+
+   ```bash
+   poetry run pytest -v tests --ignore tests/integration
+   ```
+
+5. **Check code quality**
+
+   The pre-commit hooks will run automatically, but you can also run them manually:
+
+   ```bash
+   pre-commit run --all-files
+   ```
+
+### Code style
+
+We use `ruff` for both linting and formatting. The configuration is in `pyproject.toml`. Key points:
+
+- Line length: 88 characters (black default)
+- Follow PEP 8 conventions
+- Use type hints where appropriate
+- Write clear docstrings (we use numpydoc format)
+
+The pre-commit hooks will handle most formatting automatically. If you want to run ruff manually:
+
+```bash
+ruff check .
+ruff format .
+```
+
+### Testing
+
+- **Write tests for new functionality**: Add unit tests in the `tests/` directory
+- **Test coverage**: Aim to maintain or improve test coverage
+- **Fast tests**: Unit tests should use simulated data when possible (see `tests/anoph/`)
+- **Integration tests**: Tests requiring GCS data access are slower and run separately
+
+Run type checking with:
+
+```bash
+poetry run pytest -v tests --typeguard-packages=malariagen_data,malariagen_data.anoph
+```
+
+### Documentation
+
+- Update docstrings if you modify public APIs
+- Documentation is built using Sphinx with the pydata theme
+- API docs are auto-generated from docstrings
+- Follow the [numpydoc](https://numpydoc.readthedocs.io/) style guide
+
+## Submitting your contribution
+
+### Before opening a pull request
+
+- [ ] Tests pass locally
+- [ ] Pre-commit hooks pass (or run `pre-commit run --all-files`)
+- [ ] Code is well-documented
+- [ ] Commit messages are clear and descriptive
+
+### Opening a pull request
+
+1. **Push your branch**
+
+   ```bash
+   git push origin your-branch-name
+   ```
+
+2. **Create the pull request**
+   - Go to the [repository on GitHub](https://github.com/malariagen/malariagen-data-python)
+   - Click "Pull requests" → "New pull request"
+   - Select your fork and branch
+   - Write a clear title and description
+
+3. **Pull request description should include:**
+   - What problem does this solve?
+   - How does it solve it?
+   - Any relevant issue numbers (e.g., "Fixes #123")
+   - Testing done
+   - Any breaking changes or migration notes
+
+### Review process
+
+- PRs require approval from a project maintainer
+- CI tests must pass (pytest on Python 3.10 with NumPy 1.26.4)
+- Address review feedback by pushing new commits to your branch
+- Once approved, a maintainer will merge your PR
+
+## AI-assisted contributions
+
+We welcome contributions that involve AI tools (like GitHub Copilot, ChatGPT, or similar). If you use AI assistance:
+
+- Review and understand any AI-generated code before submitting
+- Ensure the code follows project conventions and passes all tests
+- You remain responsible for the quality and correctness of the contribution
+- Disclosure of AI usage is optional. Regardless of tools used, contributors remain responsible for the quality and correctness of their submissions.
+
+## Communication
+
+- **Issues**: Use [GitHub Issues](https://github.com/malariagen/malariagen-data-python/issues) for bug reports and feature requests
+- **Discussions**: For questions and general discussion, use [GitHub Discussions](https://github.com/malariagen/malariagen-data-python/discussions)
+- **Pull requests**: Use PR comments for code review discussions
+- **Email**: For data access questions, contact [support@malariagen.net](mailto:support@malariagen.net)
+
+## Finding something to work on
+
+- Look for issues labeled [`good first issue`](https://github.com/malariagen/malariagen-data-python/labels/good%20first%20issue)
+- Check for issues labeled [`help wanted`](https://github.com/malariagen/malariagen-data-python/labels/help%20wanted)
+- Improve documentation or add examples
+- Increase test coverage
+
+## Questions?
+
+If you're unsure about anything, feel free to:
+
+- Open an issue to ask
+- Start a discussion on GitHub Discussions
+- Ask in your pull request
+
+We appreciate your contributions and will do our best to help you succeed!
+
+## License
+
+By contributing to this project, you agree that your contributions will be licensed under the [MIT License](LICENSE).

README.md

@@ -65,7 +65,7 @@ modifying the `docs/source/_static/switcher.json` file accordingly.
 
 If you use the `malariagen_data` package in a publication
 or include any of its functions or code in other materials (_e.g._ training resources),
-please cite: [doi.org/10.5281/zenodo.11173411](doi.org/10.5281/zenodo.11173411)
+please cite: [doi.org/10.5281/zenodo.11173411](https://doi.org/10.5281/zenodo.11173411)
 
 Some functions may require additional citations to acknowledge specific contributions. These are indicated in the description for each relevant function.
 

malariagen_data/anoph/cnv_data.py

@@ -886,11 +886,13 @@ def plot_cnv_hmm_heatmap_track(
         width: gplt_params.width = gplt_params.width_default,
         row_height: gplt_params.row_height = 7,
         height: Optional[gplt_params.height] = None,
-        palette: Optional[gplt_params.colors] = cnv_params.colorscale_default,
+        palette: Optional[gplt_params.colors] = None,
         show: gplt_params.show = True,
         output_backend: gplt_params.output_backend = gplt_params.output_backend_default,
     ) -> gplt_params.optional_figure:
         debug = self._log.debug
+        if palette is None:
+            palette = cnv_params.colorscale_default
 
         import bokeh.models as bkmod
         import bokeh.plotting as bkplt
@@ -1028,13 +1030,15 @@ def plot_cnv_hmm_heatmap(
         width: gplt_params.width = gplt_params.width_default,
         row_height: gplt_params.row_height = 7,
         track_height: Optional[gplt_params.track_height] = None,
-        palette: Optional[gplt_params.colors] = cnv_params.colorscale_default,
+        palette: Optional[gplt_params.colors] = None,
         genes_height: gplt_params.genes_height = gplt_params.genes_height_default,
         show: gplt_params.show = True,
         gene_labels: Optional[gplt_params.gene_labels] = None,
         gene_labelset: Optional[gplt_params.gene_labelset] = None,
     ) -> gplt_params.optional_figure:
         debug = self._log.debug
+        if palette is None:
+            palette = cnv_params.colorscale_default
 
         import bokeh.layouts as bklay
         import bokeh.plotting as bkplt

malariagen_data/anoph/dipclust.py

@@ -622,7 +622,7 @@ def plot_diplotype_clustering_advanced(
         snp_filter_min_maf: float = 0.05,
         snp_query: Optional[base_params.snp_query] = AA_CHANGE_QUERY,
         cnv_region: Optional[base_params.regions] = None,
-        cnv_colorscale: plotly_params.color_continuous_scale = cnv_params.colorscale_default,
+        cnv_colorscale: plotly_params.color_continuous_scale = None,
         cnv_max_coverage_variance: cnv_params.max_coverage_variance = 0.2,
         site_mask: Optional[base_params.site_mask] = None,
         sample_sets: Optional[base_params.sample_sets] = None,
@@ -657,6 +657,8 @@ def plot_diplotype_clustering_advanced(
         chunks: base_params.chunks = base_params.native_chunks,
         inline_array: base_params.inline_array = base_params.inline_array_default,
     ):
+        if cnv_colorscale is None:
+            cnv_colorscale = cnv_params.colorscale_default
         if cohort_size and snp_transcript:
             cohort_size = None
             print(

malariagen_data/anoph/genome_features.py

@@ -552,5 +552,5 @@ def _transcript_to_parent_name(self, transcript):
             return self._gene_name_overrides[parent_id]
         except KeyError:
             rec_parent = df_genome_features.loc[parent_id]
-            # Try to access "Name" attribute, fall back to "ID" if not present.
-            return rec_parent.get("Name", parent_id)
+            # Try to access gene name attribute, fall back to "ID" if not present.
+            return rec_parent.get(self._gff_gene_name_attribute, parent_id)

malariagen_data/veff.py

@@ -84,7 +84,7 @@ def get_ref_allele_coords(self, chrom, pos, ref):
         ref_seq = self.get_ref_seq(chrom, ref_start, ref_stop).lower()
         assert ref_seq == ref.lower(), (
             "reference allele does not match reference sequence, "
-            "expected %r, found %r" % (ref_seq, ref.lower())
+            f"expected {ref_seq!r}, found {ref.lower()!r}"
         )
 
         return ref_start, ref_stop
@@ -262,11 +262,11 @@ def _get_within_cds_effect(ann, base_effect, cds, cdss):
     base_effect = base_effect._replace(
         ref_codon=ref_codon,
         alt_codon=alt_codon,
-        codon_change="%s/%s" % (ref_codon, alt_codon),
+        codon_change=f"{ref_codon}/{alt_codon}",
         aa_pos=aa_pos,
         ref_aa=ref_aa,
         alt_aa=alt_aa,
-        aa_change="%s%s%s" % (ref_aa, aa_pos, alt_aa),
+        aa_change=f"{ref_aa}{aa_pos}{alt_aa}",
     )
 
     if len(ref) == 1 and len(alt) == 1:

tests/anoph/test_genome_features.py

@@ -169,7 +169,9 @@ def test_plot_genes_with_gene_labels(fixture, api: AnophelesGenomeFeaturesData):
     # For each contig in the fixture...
     for contig in fixture.contigs:
         # Get the genes for this contig.
-        genes_df = api.genome_features(region=contig).query("type == 'gene'")
+        genes_df = api.genome_features(region=contig).query(
+            f"type == '{api._gff_gene_type}'"
+        )
 
         # If there are no genes, we cannot label them.
         if not genes_df.empty:
@@ -181,7 +183,10 @@ def test_plot_genes_with_gene_labels(fixture, api: AnophelesGenomeFeaturesData):
 
             # Put the random gene "ID" and its "Name" in a dictionary.
             random_gene_labels = dict(
-                zip(random_sample_genes_df["ID"], random_sample_genes_df["Name"])
+                zip(
+                    random_sample_genes_df["ID"],
+                    random_sample_genes_df[api._gff_gene_name_attribute],
+                )
             )
 
             # Check that we get a Bokeh figure from plot_genes() with these gene_labels.