Optimize scraper performance and add fallback selectors for robustness

Performance improvements:
- Validation speed: 59.71s → 10.96s (5.5x improvement)
- Removed 50+ console.log statements from JavaScript extraction
- Replaced hardcoded sleeps with WebDriverWait for smart element-based waiting
- Added aggressive memory management (console.clear, GC, image unloading every 20 scrolls)

Scraping improvements:
- Increased idle detection from 6 to 12 consecutive idle scrolls for completeness
- Added real-time progress updates every 5 scrolls with percentage calculation
- Added crash recovery to extract partial reviews if Chrome crashes
- Removed artificial 200-review limit to scrape ALL reviews

Timestamp tracking:
- Added updated_at field separate from started_at for progress tracking
- Frontend now shows both "Started" (fixed) and "Last Update" (dynamic)

Robustness improvements:
- Added 5 fallback CSS selectors to handle different Google Maps page structures
- Now tries: div.jftiEf.fontBodyMedium, div.jftiEf, div[data-review-id], etc.
- Automatic selector detection logs which selector works for debugging

Test results:
- Successfully scraped 550 reviews in 150.53s without crashes
- Memory management prevents Chrome tab crashes during heavy scraping

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Alejandro Gutiérrez
2026-01-18 19:49:24 +00:00
parent bdffb5eaac
commit faa0704737
108 changed files with 23632 additions and 54 deletions

47
test_debug_extraction.py Normal file
View File

@@ -0,0 +1,47 @@
#!/usr/bin/env python3
"""
Test script to check what debug data we can extract from Google Maps
"""
import json
from modules.fast_scraper import fast_scrape_reviews
url = "https://www.google.com/maps/place/Soho+Club/data=!4m7!3m6!1s0x46dd947294b213bf:0x864c7a232527adb4!8m2!3d54.67869!4d25.2667181!16s%2Fg%2F1thhj5ml!19sChIJvxOylHKU3UYRtK0nJSN6TIY?authuser=0&hl=es&rclk=1"
print("Starting scrape...")
result = fast_scrape_reviews(url, headless=True)
reviews = result.get('reviews', [])
print(f"\nExtracted {len(reviews)} reviews")
if reviews:
print("\n" + "="*80)
print("FIRST REVIEW:")
print("="*80)
first_review = reviews[0]
# Print all keys
print(f"Keys: {list(first_review.keys())}")
print()
# Print full first review
print(json.dumps(first_review, indent=2, default=str))
if '_google_state_debug' in first_review:
print("\n" + "="*80)
print("GOOGLE STATE DEBUG:")
print("="*80)
print(json.dumps(first_review['_google_state_debug'], indent=2))
if 'debug_date_info' in first_review and first_review['debug_date_info']:
print("\n" + "="*80)
print("DATE DEBUG INFO:")
print("="*80)
print(json.dumps(first_review['debug_date_info'], indent=2, default=str))
# Save all to file
with open('/tmp/google_maps_debug_dump.json', 'w') as f:
json.dump(reviews[:5], f, indent=2, default=str) # Save first 5 reviews
print(f"\nFirst 5 reviews saved to: /tmp/google_maps_debug_dump.json")
else:
print("No reviews extracted!")
print(f"Result: {result}")