3 Commits

Author	SHA1	Message	Date
Alejandro Gutiérrez	8ccf72a489	Remove old scraper files - consolidate to scraper_clean ... Production (api_server_production.py) only uses: - modules/scraper_clean.py - main scraping logic - modules/fast_scraper.py - validation helpers - modules/database.py, webhooks.py, health_checks.py, chrome_pool.py Deleted 33 unused Python files including: - Old API server (api_server.py) - 14 start*.py experimental scrapers - 7 *_scraper.py variants - Old modules: scraper.py, api_interceptor.py, job_manager.py, cli.py - Various debug/test/utility scripts Saves ~11,000 lines of unmaintained code. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2026-01-23 17:25:00 +00:00
Alejandro Gutiérrez	8b36850838	Switch Docker production API to use scraper_clean ... - Import fast_scrape_reviews from scraper_clean instead of fast_scraper - Keeps helper functions (check_reviews_available, get_business_card_info) from fast_scraper - Production now uses clean scraper with hard refresh recovery Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2026-01-22 14:19:40 +00:00
Alejandro Gutiérrez	faa0704737	Optimize scraper performance and add fallback selectors for robustness ... Performance improvements: - Validation speed: 59.71s → 10.96s (5.5x improvement) - Removed 50+ console.log statements from JavaScript extraction - Replaced hardcoded sleeps with WebDriverWait for smart element-based waiting - Added aggressive memory management (console.clear, GC, image unloading every 20 scrolls) Scraping improvements: - Increased idle detection from 6 to 12 consecutive idle scrolls for completeness - Added real-time progress updates every 5 scrolls with percentage calculation - Added crash recovery to extract partial reviews if Chrome crashes - Removed artificial 200-review limit to scrape ALL reviews Timestamp tracking: - Added updated_at field separate from started_at for progress tracking - Frontend now shows both "Started" (fixed) and "Last Update" (dynamic) Robustness improvements: - Added 5 fallback CSS selectors to handle different Google Maps page structures - Now tries: div.jftiEf.fontBodyMedium, div.jftiEf, div[data-review-id], etc. - Automatic selector detection logs which selector works for debugging Test results: - Successfully scraped 550 reviews in 150.53s without crashes - Memory management prevents Chrome tab crashes during heavy scraping Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>	2026-01-18 19:49:24 +00:00