faa0704737
Performance improvements: - Validation speed: 59.71s → 10.96s (5.5x improvement) - Removed 50+ console.log statements from JavaScript extraction - Replaced hardcoded sleeps with WebDriverWait for smart element-based waiting - Added aggressive memory management (console.clear, GC, image unloading every 20 scrolls) Scraping improvements: - Increased idle detection from 6 to 12 consecutive idle scrolls for completeness - Added real-time progress updates every 5 scrolls with percentage calculation - Added crash recovery to extract partial reviews if Chrome crashes - Removed artificial 200-review limit to scrape ALL reviews Timestamp tracking: - Added updated_at field separate from started_at for progress tracking - Frontend now shows both "Started" (fixed) and "Last Update" (dynamic) Robustness improvements: - Added 5 fallback CSS selectors to handle different Google Maps page structures - Now tries: div.jftiEf.fontBodyMedium, div.jftiEf, div[data-review-id], etc. - Automatic selector detection logs which selector works for debugging Test results: - Successfully scraped 550 reviews in 150.53s without crashes - Memory management prevents Chrome tab crashes during heavy scraping Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
27 lines
754 B
Plaintext
27 lines
754 B
Plaintext
# Production Environment Variables
|
|
# Copy this to .env and configure for your environment
|
|
|
|
# Database
|
|
DB_PASSWORD=scraper123
|
|
DATABASE_URL=postgresql://scraper:scraper123@localhost:5432/scraper
|
|
|
|
# API Configuration
|
|
API_BASE_URL=http://localhost:8000
|
|
PORT=8000
|
|
|
|
# Job Concurrency (limits simultaneous Chrome instances)
|
|
# Recommendation: 5 jobs per 8GB RAM (each Chrome = ~500MB)
|
|
# 8GB server: MAX_CONCURRENT_JOBS=5
|
|
# 16GB server: MAX_CONCURRENT_JOBS=10
|
|
# 32GB server: MAX_CONCURRENT_JOBS=20
|
|
MAX_CONCURRENT_JOBS=5
|
|
|
|
# Canary Test Configuration
|
|
CANARY_TEST_URL=https://www.google.com/maps/place/Soho+Factory/@54.6738155,25.2595844,17z/
|
|
|
|
# Alerting (Optional)
|
|
SLACK_WEBHOOK_URL=
|
|
ALERT_EMAIL=
|
|
|
|
# For production deployment, use stronger passwords and HTTPS URLs
|