Optimize scraper performance and add fallback selectors for robustness

Performance improvements:
- Validation speed: 59.71s → 10.96s (5.5x improvement)
- Removed 50+ console.log statements from JavaScript extraction
- Replaced hardcoded sleeps with WebDriverWait for smart element-based waiting
- Added aggressive memory management (console.clear, GC, image unloading every 20 scrolls)

Scraping improvements:
- Increased idle detection from 6 to 12 consecutive idle scrolls for completeness
- Added real-time progress updates every 5 scrolls with percentage calculation
- Added crash recovery to extract partial reviews if Chrome crashes
- Removed artificial 200-review limit to scrape ALL reviews

Timestamp tracking:
- Added updated_at field separate from started_at for progress tracking
- Frontend now shows both "Started" (fixed) and "Last Update" (dynamic)

Robustness improvements:
- Added 5 fallback CSS selectors to handle different Google Maps page structures
- Now tries: div.jftiEf.fontBodyMedium, div.jftiEf, div[data-review-id], etc.
- Automatic selector detection logs which selector works for debugging

Test results:
- Successfully scraped 550 reviews in 150.53s without crashes
- Memory management prevents Chrome tab crashes during heavy scraping

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Alejandro Gutiérrez
2026-01-18 19:49:24 +00:00
parent bdffb5eaac
commit faa0704737
108 changed files with 23632 additions and 54 deletions

26
.env.example Normal file
View File

@@ -0,0 +1,26 @@
# Production Environment Variables
# Copy this to .env and configure for your environment
# Database
DB_PASSWORD=scraper123
DATABASE_URL=postgresql://scraper:scraper123@localhost:5432/scraper
# API Configuration
API_BASE_URL=http://localhost:8000
PORT=8000
# Job Concurrency (limits simultaneous Chrome instances)
# Recommendation: 5 jobs per 8GB RAM (each Chrome = ~500MB)
# 8GB server: MAX_CONCURRENT_JOBS=5
# 16GB server: MAX_CONCURRENT_JOBS=10
# 32GB server: MAX_CONCURRENT_JOBS=20
MAX_CONCURRENT_JOBS=5
# Canary Test Configuration
CANARY_TEST_URL=https://www.google.com/maps/place/Soho+Factory/@54.6738155,25.2595844,17z/
# Alerting (Optional)
SLACK_WEBHOOK_URL=
ALERT_EMAIL=
# For production deployment, use stronger passwords and HTTPS URLs