Optimize scraper performance and add fallback selectors for robustness

Browse Source

Performance improvements:
- Validation speed: 59.71s → 10.96s (5.5x improvement)
- Removed 50+ console.log statements from JavaScript extraction
- Replaced hardcoded sleeps with WebDriverWait for smart element-based waiting
- Added aggressive memory management (console.clear, GC, image unloading every 20 scrolls)

Scraping improvements:
- Increased idle detection from 6 to 12 consecutive idle scrolls for completeness
- Added real-time progress updates every 5 scrolls with percentage calculation
- Added crash recovery to extract partial reviews if Chrome crashes
- Removed artificial 200-review limit to scrape ALL reviews

Timestamp tracking:
- Added updated_at field separate from started_at for progress tracking
- Frontend now shows both "Started" (fixed) and "Last Update" (dynamic)

Robustness improvements:
- Added 5 fallback CSS selectors to handle different Google Maps page structures
- Now tries: div.jftiEf.fontBodyMedium, div.jftiEf, div[data-review-id], etc.
- Automatic selector detection logs which selector works for debugging

Test results:
- Successfully scraped 550 reviews in 150.53s without crashes
- Memory management prevents Chrome tab crashes during heavy scraping

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

...

This commit is contained in:

Alejandro Gutiérrez

2026-01-18 19:49:24 +00:00

parent bdffb5eaac

commit faa0704737

108 changed files with 23632 additions and 54 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

2

api_response_samples/response_00_body.txt Normal file

File diff suppressed because one or more lines are too long

2

api_response_samples/response_01_body.txt Normal file

File diff suppressed because one or more lines are too long

2

api_response_samples/response_02_body.txt Normal file

File diff suppressed because one or more lines are too long

2

api_response_samples/response_03_body.txt Normal file

File diff suppressed because one or more lines are too long

2

api_response_samples/response_04_body.txt Normal file

File diff suppressed because one or more lines are too long