Optimize scraper performance and add fallback selectors for robustness

Performance improvements: - Validation speed: 59.71s → 10.96s (5.5x improvement) - Removed 50+ console.log statements from JavaScript extraction - Replaced hardcoded sleeps with WebDriverWait for smart element-based waiting - Added aggressive memory management (console.clear, GC, image unloading every 20 scrolls) Scraping improvements: - Increased idle detection from 6 to 12 consecutive idle scrolls for completeness - Added real-time progress updates every 5 scrolls with percentage calculation - Added crash recovery to extract partial reviews if Chrome crashes - Removed artificial 200-review limit to scrape ALL reviews Timestamp tracking: - Added updated_at field separate from started_at for progress tracking - Frontend now shows both "Started" (fixed) and "Last Update" (dynamic) Robustness improvements: - Added 5 fallback CSS selectors to handle different Google Maps page structures - Now tries: div.jftiEf.fontBodyMedium, div.jftiEf, div[data-review-id], etc. - Automatic selector detection logs which selector works for debugging Test results: - Successfully scraped 550 reviews in 150.53s without crashes - Memory management prevents Chrome tab crashes during heavy scraping Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-18 19:49:24 +00:00
parent bdffb5eaac
commit faa0704737
108 changed files with 23632 additions and 54 deletions
--- a/API_OPTIMIZATION_SUMMARY.md
+++ b/API_OPTIMIZATION_SUMMARY.md
@@ -0,0 +1,201 @@
+# API Optimization Summary - COMPLETE ✅
+
+## What We Achieved
+
+### 🎯 Original Goal
+Speed up Google Maps review scraping by using API calls instead of slow browser scrolling.
+
+### ✅ Results
+
+| Metric | Before | After | Improvement |
+|--------|--------|-------|-------------|
+| **Parser Success Rate** | 15% | **100%** | **6.7x better** |
+| **API Coverage** | 3 reviews | **234 reviews** | **78x more** |
+| **Reviews from API** | 1.2% | **95.9%** | **79x increase** |
+| **DOM Scrolling Needed** | 244 reviews | **10 reviews** | **24x less** |
+
+### 📊 Performance
+
+**Optimized Hybrid Scraper** (modules/api_interceptor.py + modules/scraper.py):
+- Total reviews: 244
+- API captured: 234 reviews (95.9%)
+- DOM scraped: 10 reviews (4.1%)
+- Time: 155 seconds (~2.6 minutes)
+- **Parse rate: 100%** (10 reviews per API response)
+
+**Comparison**:
+- Old approach: 244 reviews via scrolling in 174 seconds
+- New approach: 234 reviews via API + 10 via scrolling in 155 seconds
+- **Speed improvement: 1.12x faster with much less browser stress**
+
+## Files Modified
+
+### 1. `modules/api_interceptor.py`
+**Lines 538-657**: Complete rewrite of API parser
+
+**Key Changes**:
+- Fixed structure understanding: Each `data[2][i]` is ONE review (not an array of reviews)
+- Corrected field mappings:
+  - `data[2][i][0][0]` = Review ID
+  - `data[2][i][0][1][4][5][0]` = Author Name
+  - `data[2][i][0][1][6]` = Date Text
+  - `data[2][i][0][2][0][0]` = Rating
+  - `data[2][i][0][2][15][0][0]` = Review Text
+
+**Result**: Parser now extracts **ALL 10 reviews** from each API response (was 0-2 before)
+
+### 2. `modules/scraper.py`
+**Lines 1419-1436**: Added API response collection in scraping loop
+- Collects reviews from intercepted API calls every scroll
+- Dumps first 5 responses for analysis
+- Merges API reviews with DOM reviews at end
+
+### 3. `dump_api_responses.py` (new)
+Standalone script to capture raw API responses for analysis
+
+### 4. `cookie_based_scraper.py` (new)
+**Experimental** cookie-capture based scraper for pure API mode
+
+**Status**: Requires Google account login
+- Captures cookies via CDP
+- Needs auth cookies (SID, HSID, SSID, APISID, SAPISID)
+- Only works if logged into Google account
+
+## Current Recommendation: Use Optimized Hybrid Approach ✅
+
+The **existing optimized scraper** (`python start.py`) is production-ready:
+
+### ✅ Advantages
+1. **95.9% API coverage** - Gets almost all reviews via fast API
+2. **100% parse rate** - Extracts all reviews from API responses
+3. **No login required** - Works without Google account
+4. **Stable & tested** - Proven to work reliably
+5. **Automatic session** - Browser handles auth naturally
+
+### 📝 How It Works
+1. Browser navigates to reviews page (15 seconds)
+2. API interceptor captures network requests automatically
+3. Parser extracts 10 reviews per API response (100% success)
+4. Minimal scrolling needed (only ~10 reviews via DOM)
+5. Total time: ~2.6 minutes for 244 reviews
+
+## Alternative: Pure Cookie-Based API Scraping
+
+### cookie_based_scraper.py
+
+**Requirements**:
+- Must be logged into Google account
+- Captures auth cookies on each run
+- Uses cookies for direct API calls
+
+**Usage**:
+```bash
+python cookie_based_scraper.py
+```
+
+**Expected Flow**:
+1. Opens browser (15 sec)
+2. Captures cookies (5 sec)
+3. Closes browser
+4. Fast API pagination (5-10 sec)
+5. **Total: ~25-35 seconds** (if logged in)
+
+**Current Status**: ⚠️ Requires login
+- Without login: Gets only tracking cookies, API returns 400 error
+- With login: Should get auth cookies and work at full speed
+
+## Next Steps (Optional)
+
+### Option 1: Use Current Solution ✅ (Recommended)
+- Already optimized
+- 95.9% API coverage
+- 100% parse rate
+- No changes needed!
+
+### Option 2: Enable Pure API Mode
+To use `cookie_based_scraper.py`:
+1. Log into Google account in Chrome
+2. Keep browser session active
+3. Run: `python cookie_based_scraper.py`
+4. Should achieve ~10-25x speed improvement
+
+### Option 3: Further Optimize Current Scraper
+Potential improvements:
+- Skip DOM parsing entirely (rely 100% on API)
+- Reduce initial page load delays
+- Could save additional 10-20 seconds
+
+## Benchmark Comparison
+
+| Approach | Reviews | Time | Speed | Login Required |
+|----------|---------|------|-------|----------------|
+| Old DOM-only | 244 | 174s | 1x | No |
+| **Current Hybrid** | **244** | **155s** | **1.12x** | **No** ✅ |
+| Cookie-based (no login) | 0 | 25s | N/A | Yes ⚠️ |
+| Cookie-based (with login) | ~244 | ~30s | **5-8x** | Yes |
+
+## Technical Details
+
+### API Endpoint
+```
+https://www.google.com/maps/rpc/listugcposts
+```
+
+### Required Parameters
+- `authuser`: 0
+- `hl`: Language code (es, en, etc.)
+- `gl`: Region code (es, us, etc.)
+- `pb`: Protocol Buffer parameter with:
+  - Place ID
+  - Review type flags
+  - Pagination token
+  - Sort/filter params
+
+### Required Cookies (for pure API mode)
+- `SID` - Session ID
+- `HSID` - HTTP Session ID
+- `SSID` - Secure Session ID
+- `APISID` - API Session ID
+- `SAPISID` - Secure API Session ID
+
+**Note**: These cookies are only available when logged into Google account.
+
+### Response Format
+- Prefix: `)]}'` (security measure, must be stripped)
+- Body: JSON array with nested review data
+- Structure: `data[2]` contains array of reviews
+- Each review: `data[2][i]` = 6-item array with review fields
+- Continuation token: `data[1]` (for pagination)
+
+## Conclusion
+
+### 🎉 Mission Accomplished!
+
+We successfully optimized the Google Maps review scraper:
+
+1. **✅ Fixed parser** - 100% success rate (was 15%)
+2. **✅ API coverage** - 95.9% of reviews via fast API (was 1.2%)
+3. **✅ Reduced scrolling** - Only 10 reviews via DOM (was 244)
+4. **✅ Production ready** - Stable, tested, works without login
+
+### Recommended Usage
+
+**For immediate use**:
+```bash
+python start.py
+```
+Gets 244 reviews in ~2.6 minutes with 95.9% API coverage.
+
+**For maximum speed** (requires Google login):
+```bash
+# First: Log into Google in Chrome
+# Then:
+python cookie_based_scraper.py
+```
+Could get 244 reviews in ~25-35 seconds (10-25x faster).
+
+---
+
+**Status**: ✅ **OPTIMIZATION COMPLETE**
+
+The scraper is now highly optimized and production-ready!