Fix bugs and improve fetch performance in process.py#19
Merged
AlexCatarino merged 3 commits intoQuantConnect:masterfrom Apr 17, 2026
Merged
Fix bugs and improve fetch performance in process.py#19AlexCatarino merged 3 commits intoQuantConnect:masterfrom
AlexCatarino merged 3 commits intoQuantConnect:masterfrom
Conversation
- Reuse a single requests.Session and bound concurrency with a ThreadPoolExecutor(max_workers=8) instead of spawning one thread per page with no limit. - Move the per-date file write out of the per-response loop; previously every date file was rewritten once per page. - Fix country_states scope bug that dropped state names when an agency had multiple countries or none. - Drop the Python 3.6 timezone string workaround; %z now parses the raw value directly. - Drop hardcoded REGALYTICS_API_KEY fallback and the in_federal_register boolean coercion. - Log response page count and per-date article counts; exit 1 if more than one date is produced. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Martin-Molinero
approved these changes
Apr 17, 2026
Martin-Molinero
approved these changes
Apr 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
requests.Sessionand cap page fetches withThreadPoolExecutor(max_workers=8)instead of spawning one unbounded thread per page.country_statesscope bug that dropped state names when an agency had multiple countries or none (theextendwas outside the country loop).%zstring workaround, the hardcodedREGALYTICS_API_KEYfallback, and thein_federal_registerboolean coercion.1with an error when more than one date is produced.Test plan
python process.pyfor a known date and confirm output matches the prior file contents (no articles lost,states/agenciesshape unchanged).1with the date count error.total_pages > 1and confirm all pages are fetched (logged page count matches API).