- Add run pipeline page with job selection UI
- Add execution detail page with stage metrics visualization
- Add stage_metrics and total_duration_ms to pipeline.executions table
- Create Next.js API proxy routes for all pipeline endpoints
- Fix trailing slash issues in pipeline-api.ts URLs
- Add Docker volume mounts for pipeline packages
- Add REVIEWIQ_DATABASE_URL and LLM API keys to docker-compose
- Fix JSONB field parsing in execution detail endpoint
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add version selector dropdown in scrape confirmation modal
- Default to v1.1.0 (Multi-Sort) which bypasses ~1000 review limit
- Pass scraper_version through API proxy to backend
- Update /new page fallback to show v1.1.0 as available
- Show version description explaining multi-sort benefits
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Task #18: Complete integration of all JobDevTools components
- Updated job detail page (/jobs/[id]) with full JobDevTools UI
- Connected SSE stream for real-time structured logs + metrics
- Added crash-report and retry API routes for Next.js
- Added format conversion for old/new log formats
- Added DevTools links to JobsView modal and actions column
- Wired up CrashReport retry with auto-fix parameters
- Integrated SessionPanel for fingerprint display
- Integrated MetricsDashboard for real-time charts
Job DevTools implementation complete: 18/18 tasks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Performance improvements:
- Validation speed: 59.71s → 10.96s (5.5x improvement)
- Removed 50+ console.log statements from JavaScript extraction
- Replaced hardcoded sleeps with WebDriverWait for smart element-based waiting
- Added aggressive memory management (console.clear, GC, image unloading every 20 scrolls)
Scraping improvements:
- Increased idle detection from 6 to 12 consecutive idle scrolls for completeness
- Added real-time progress updates every 5 scrolls with percentage calculation
- Added crash recovery to extract partial reviews if Chrome crashes
- Removed artificial 200-review limit to scrape ALL reviews
Timestamp tracking:
- Added updated_at field separate from started_at for progress tracking
- Frontend now shows both "Started" (fixed) and "Last Update" (dynamic)
Robustness improvements:
- Added 5 fallback CSS selectors to handle different Google Maps page structures
- Now tries: div.jftiEf.fontBodyMedium, div.jftiEf, div[data-review-id], etc.
- Automatic selector detection logs which selector works for debugging
Test results:
- Successfully scraped 550 reviews in 150.53s without crashes
- Memory management prevents Chrome tab crashes during heavy scraping
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
