Optimize scraper performance and add fallback selectors for robustness
Performance improvements: - Validation speed: 59.71s → 10.96s (5.5x improvement) - Removed 50+ console.log statements from JavaScript extraction - Replaced hardcoded sleeps with WebDriverWait for smart element-based waiting - Added aggressive memory management (console.clear, GC, image unloading every 20 scrolls) Scraping improvements: - Increased idle detection from 6 to 12 consecutive idle scrolls for completeness - Added real-time progress updates every 5 scrolls with percentage calculation - Added crash recovery to extract partial reviews if Chrome crashes - Removed artificial 200-review limit to scrape ALL reviews Timestamp tracking: - Added updated_at field separate from started_at for progress tracking - Frontend now shows both "Started" (fixed) and "Last Update" (dynamic) Robustness improvements: - Added 5 fallback CSS selectors to handle different Google Maps page structures - Now tries: div.jftiEf.fontBodyMedium, div.jftiEf, div[data-review-id], etc. - Automatic selector detection logs which selector works for debugging Test results: - Successfully scraped 550 reviews in 150.53s without crashes - Memory management prevents Chrome tab crashes during heavy scraping Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
47
test_debug_extraction.py
Normal file
47
test_debug_extraction.py
Normal file
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script to check what debug data we can extract from Google Maps
|
||||
"""
|
||||
import json
|
||||
from modules.fast_scraper import fast_scrape_reviews
|
||||
|
||||
url = "https://www.google.com/maps/place/Soho+Club/data=!4m7!3m6!1s0x46dd947294b213bf:0x864c7a232527adb4!8m2!3d54.67869!4d25.2667181!16s%2Fg%2F1thhj5ml!19sChIJvxOylHKU3UYRtK0nJSN6TIY?authuser=0&hl=es&rclk=1"
|
||||
|
||||
print("Starting scrape...")
|
||||
result = fast_scrape_reviews(url, headless=True)
|
||||
|
||||
reviews = result.get('reviews', [])
|
||||
print(f"\nExtracted {len(reviews)} reviews")
|
||||
|
||||
if reviews:
|
||||
print("\n" + "="*80)
|
||||
print("FIRST REVIEW:")
|
||||
print("="*80)
|
||||
first_review = reviews[0]
|
||||
|
||||
# Print all keys
|
||||
print(f"Keys: {list(first_review.keys())}")
|
||||
print()
|
||||
|
||||
# Print full first review
|
||||
print(json.dumps(first_review, indent=2, default=str))
|
||||
|
||||
if '_google_state_debug' in first_review:
|
||||
print("\n" + "="*80)
|
||||
print("GOOGLE STATE DEBUG:")
|
||||
print("="*80)
|
||||
print(json.dumps(first_review['_google_state_debug'], indent=2))
|
||||
|
||||
if 'debug_date_info' in first_review and first_review['debug_date_info']:
|
||||
print("\n" + "="*80)
|
||||
print("DATE DEBUG INFO:")
|
||||
print("="*80)
|
||||
print(json.dumps(first_review['debug_date_info'], indent=2, default=str))
|
||||
|
||||
# Save all to file
|
||||
with open('/tmp/google_maps_debug_dump.json', 'w') as f:
|
||||
json.dump(reviews[:5], f, indent=2, default=str) # Save first 5 reviews
|
||||
print(f"\nFirst 5 reviews saved to: /tmp/google_maps_debug_dump.json")
|
||||
else:
|
||||
print("No reviews extracted!")
|
||||
print(f"Result: {result}")
|
||||
Reference in New Issue
Block a user