Implement continuous scrolling with smart gap-based timeout

Browse Source

Major refactoring to achieve 100% review collection:

CONTINUOUS SCROLLING:
- Background thread scrolls NON-STOP at 5ms intervals (no gaps!)
- Main thread checks every 2s while scrolling continues
- Stops immediately when all reviews collected
- Solves the core problem: gaps between bursts caused Google to stop loading

SMART TIMEOUT:
- Gap-based: 3x average gap between review loads
- Initial timeout: 3x time since first load (or 15s default)
- Adaptive: evolves from conservative early timeout to smart gap-based
- Detailed logging shows timeout calculations

RESULTS:
- 100% completion (271/271) vs previous 91% (247/271)
- 3.5x faster (~17s vs 60s)
- Clean thread management with proper shutdown

REMOVED:
- All burst scrolling code (~100 lines)
- Scroll stuck detection (no longer needed)
- Dynamic sleep logic (replaced with continuous scrolling)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

...

This commit is contained in:

Alejandro Gutiérrez

2026-01-19 01:39:47 +00:00

parent 4ad5c96a36

commit 8b925ba965

1 changed files with 490 additions and 453 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

943

modules/scraper.py

File diff suppressed because it is too large Load Diff