Optimize scraper performance and add fallback selectors for robustness
Performance improvements: - Validation speed: 59.71s → 10.96s (5.5x improvement) - Removed 50+ console.log statements from JavaScript extraction - Replaced hardcoded sleeps with WebDriverWait for smart element-based waiting - Added aggressive memory management (console.clear, GC, image unloading every 20 scrolls) Scraping improvements: - Increased idle detection from 6 to 12 consecutive idle scrolls for completeness - Added real-time progress updates every 5 scrolls with percentage calculation - Added crash recovery to extract partial reviews if Chrome crashes - Removed artificial 200-review limit to scrape ALL reviews Timestamp tracking: - Added updated_at field separate from started_at for progress tracking - Frontend now shows both "Started" (fixed) and "Last Update" (dynamic) Robustness improvements: - Added 5 fallback CSS selectors to handle different Google Maps page structures - Now tries: div.jftiEf.fontBodyMedium, div.jftiEf, div[data-review-id], etc. - Automatic selector detection logs which selector works for debugging Test results: - Successfully scraped 550 reviews in 150.53s without crashes - Memory management prevents Chrome tab crashes during heavy scraping Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
23
requirements-production.txt
Normal file
23
requirements-production.txt
Normal file
@@ -0,0 +1,23 @@
|
||||
# Production Requirements for Google Reviews Scraper API
|
||||
# Phase 1: PostgreSQL + Webhooks + Health Checks
|
||||
|
||||
# Core Framework
|
||||
fastapi==0.109.0
|
||||
uvicorn[standard]==0.27.0
|
||||
pydantic==2.5.3
|
||||
|
||||
# Database
|
||||
asyncpg==0.29.0 # PostgreSQL async driver
|
||||
|
||||
# HTTP Client (for webhooks)
|
||||
httpx==0.26.0
|
||||
|
||||
# Scraping
|
||||
seleniumbase==4.24.0
|
||||
pyyaml==6.0.1
|
||||
|
||||
# Logging & Monitoring
|
||||
python-multipart==0.0.6
|
||||
|
||||
# CORS
|
||||
starlette==0.35.1
|
||||
Reference in New Issue
Block a user