Skip to content

fix: replace hardcoded GCS cache keys with monthly-rotating dynamic keys #1297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kunal-10-cloud wants to merge 3 commits into
base: master
Choose a base branch
from
Open

fix: replace hardcoded GCS cache keys with monthly-rotating dynamic keys #1297

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1302998
fix: replace hardcoded GCS cache keys with monthly-rotating dynamic k…
kunal-10-cloud Apr 14, 2026
8be6e73
Merge branch 'master' into fix/issue-1296-dynamic-gcs-cache-key
kunal-10-cloud Apr 14, 2026
50e260a
Merge branch 'master' into fix/issue-1296-dynamic-gcs-cache-key
kunal-10-cloud Apr 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .github/workflows/integration_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,17 @@ jobs:
- name: 'Check gcloud CLI'
run: 'gcloud info'

- name: Generate cache key
id: cache-key
run: echo "key=gcs_cache_integration_tests_$(date +%Y%m)" >> "$GITHUB_OUTPUT"

Comment on lines +42 to +45
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The step id cache-key contains a hyphen, which cannot be referenced via dot-notation in expressions. ${{ steps.cache-key.outputs.key }} will be parsed incorrectly and the workflow may fail to evaluate the expression. Rename the step id to something like cache_key (and update references) or use bracket notation (steps['cache-key'].outputs.key).

Copilot uses AI. Check for mistakes.
- name: Restore GCS cache
uses: actions/cache/restore@v3
with:
path: gcs_cache
key: gcs_cache_integration_tests_20240922
key: ${{ steps.cache-key.outputs.key }}
restore-keys: |
gcs_cache_integration_tests_
Comment on lines +51 to +52
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

restore-keys will cause the first run after this change to restore the old September 2024 cache (it matches the gcs_cache_integration_tests_ prefix). Because gcs_cache is used by fsspec simplecache (which doesn’t revalidate cached files by default), this can result in re-saving the same stale data under the new monthly key, so the intended cache refresh may never actually happen. Consider either removing restore-keys (force a cold start once per month) or scoping the fallback to the current year/month (e.g., generate a gcs_cache_integration_tests_YYYY prefix output) so you don’t pull in pre-rotation caches.

Suggested change
restore-keys: |
gcs_cache_integration_tests_

Copilot uses AI. Check for mistakes.

- name: Run integration tests
run: poetry run pytest --durations=20 -v tests/integration
Expand All @@ -53,4 +59,4 @@ jobs:
if: always()
with:
path: gcs_cache
key: gcs_cache_integration_tests_20240922
key: ${{ steps.cache-key.outputs.key }}
10 changes: 8 additions & 2 deletions .github/workflows/notebooks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,17 @@ jobs:
- name: 'Check gcloud CLI'
run: 'gcloud info'

- name: Generate cache key
id: cache-key
run: echo "key=gcs_cache_notebooks_$(date +%Y%m)" >> "$GITHUB_OUTPUT"

Comment on lines +38 to +41
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The step id cache-key contains a hyphen, which cannot be referenced via dot-notation in expressions. ${{ steps.cache-key.outputs.key }} will be parsed incorrectly and the workflow may fail to evaluate the expression. Rename the step id to something like cache_key (and update references) or use bracket notation (steps['cache-key'].outputs.key).

Copilot uses AI. Check for mistakes.
- name: Restore GCS cache
uses: actions/cache/restore@v3
with:
path: gcs_cache
key: gcs_cache_notebooks_20240922
key: ${{ steps.cache-key.outputs.key }}
restore-keys: |
gcs_cache_notebooks_
Comment on lines +47 to +48
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

restore-keys will cause the first run after this change to restore the old September 2024 cache (it matches the gcs_cache_notebooks_ prefix). Because gcs_cache is used by fsspec simplecache (which doesn’t revalidate cached files by default), this can result in re-saving the same stale data under the new monthly key, so the intended cache refresh may never actually happen. Consider either removing restore-keys (force a cold start once per month) or scoping the fallback to the current year/month (e.g., generate a gcs_cache_notebooks_YYYY prefix output) so you don’t pull in pre-rotation caches.

Suggested change
restore-keys: |
gcs_cache_notebooks_

Copilot uses AI. Check for mistakes.

- name: Run notebooks
run: poetry run jupyter nbconvert --execute notebooks/*.ipynb --inplace
Expand All @@ -49,4 +55,4 @@ jobs:
if: always()
with:
path: gcs_cache
key: gcs_cache_notebooks_20240922
key: ${{ steps.cache-key.outputs.key }}
Loading