Files
whyrating-engine-legacy/web/app/api/jobs/[jobId]/route.ts
Alejandro Gutiérrez faa0704737 Optimize scraper performance and add fallback selectors for robustness
Performance improvements:
- Validation speed: 59.71s → 10.96s (5.5x improvement)
- Removed 50+ console.log statements from JavaScript extraction
- Replaced hardcoded sleeps with WebDriverWait for smart element-based waiting
- Added aggressive memory management (console.clear, GC, image unloading every 20 scrolls)

Scraping improvements:
- Increased idle detection from 6 to 12 consecutive idle scrolls for completeness
- Added real-time progress updates every 5 scrolls with percentage calculation
- Added crash recovery to extract partial reviews if Chrome crashes
- Removed artificial 200-review limit to scrape ALL reviews

Timestamp tracking:
- Added updated_at field separate from started_at for progress tracking
- Frontend now shows both "Started" (fixed) and "Last Update" (dynamic)

Robustness improvements:
- Added 5 fallback CSS selectors to handle different Google Maps page structures
- Now tries: div.jftiEf.fontBodyMedium, div.jftiEf, div[data-review-id], etc.
- Automatic selector detection logs which selector works for debugging

Test results:
- Successfully scraped 550 reviews in 150.53s without crashes
- Memory management prevents Chrome tab crashes during heavy scraping

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-18 19:49:24 +00:00

31 lines
777 B
TypeScript

import { NextRequest, NextResponse } from 'next/server';
const API_BASE_URL = process.env.NEXT_PUBLIC_API_URL || 'http://localhost:8000';
export async function GET(
request: NextRequest,
{ params }: { params: Promise<{ jobId: string }> }
) {
try {
const { jobId } = await params;
const response = await fetch(`${API_BASE_URL}/jobs/${jobId}`);
const data = await response.json();
if (!response.ok) {
return NextResponse.json(
{ error: data.detail || 'Job not found' },
{ status: response.status }
);
}
return NextResponse.json(data);
} catch (error) {
console.error('Job status API error:', error);
return NextResponse.json(
{ error: 'Failed to get job status' },
{ status: 500 }
);
}
}