faa0704737
Performance improvements: - Validation speed: 59.71s → 10.96s (5.5x improvement) - Removed 50+ console.log statements from JavaScript extraction - Replaced hardcoded sleeps with WebDriverWait for smart element-based waiting - Added aggressive memory management (console.clear, GC, image unloading every 20 scrolls) Scraping improvements: - Increased idle detection from 6 to 12 consecutive idle scrolls for completeness - Added real-time progress updates every 5 scrolls with percentage calculation - Added crash recovery to extract partial reviews if Chrome crashes - Removed artificial 200-review limit to scrape ALL reviews Timestamp tracking: - Added updated_at field separate from started_at for progress tracking - Frontend now shows both "Started" (fixed) and "Last Update" (dynamic) Robustness improvements: - Added 5 fallback CSS selectors to handle different Google Maps page structures - Now tries: div.jftiEf.fontBodyMedium, div.jftiEf, div[data-review-id], etc. - Automatic selector detection logs which selector works for debugging Test results: - Successfully scraped 550 reviews in 150.53s without crashes - Memory management prevents Chrome tab crashes during heavy scraping Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
39 lines
1.4 KiB
TypeScript
39 lines
1.4 KiB
TypeScript
import ScraperTest from '@/components/ScraperTest';
|
|
|
|
export default function Home() {
|
|
return (
|
|
<div className="min-h-screen bg-gradient-to-br from-blue-600 to-indigo-700 py-12 px-4">
|
|
<main className="max-w-5xl mx-auto">
|
|
<div className="text-center mb-10">
|
|
<h1 className="text-4xl md:text-5xl font-bold text-white mb-3">
|
|
Google Reviews Scraper
|
|
</h1>
|
|
<p className="text-blue-100 text-lg">
|
|
Test the containerized scraper API
|
|
</p>
|
|
<div className="mt-4 inline-flex items-center gap-2 px-4 py-2 bg-blue-500/30 rounded-lg text-blue-100 text-sm">
|
|
<div className="w-2 h-2 bg-green-400 rounded-full animate-pulse"></div>
|
|
Powered by SeleniumBase UC Mode
|
|
</div>
|
|
</div>
|
|
|
|
<div className="bg-white rounded-2xl shadow-2xl p-6 md:p-8">
|
|
<ScraperTest />
|
|
</div>
|
|
|
|
<div className="mt-8 text-center text-blue-100 text-sm space-y-2">
|
|
<p className="font-medium">💡 Example URLs to test:</p>
|
|
<div className="space-y-1 text-xs">
|
|
<p className="font-mono bg-blue-500/20 rounded px-3 py-1 inline-block">
|
|
https://www.google.com/maps/place/Soho+Club/...
|
|
</p>
|
|
</div>
|
|
<p className="mt-4 text-blue-200">
|
|
API running at: <span className="font-mono">localhost:8000</span>
|
|
</p>
|
|
</div>
|
|
</main>
|
|
</div>
|
|
);
|
|
}
|