Transform synthesis stage from consultant memo to productized report:
- New 6-section structure: Executive Summary, Risk Scorecard,
Critical Issues, Strengths, Action Matrix, 90-Day Tracking
- Add chart data aggregation (7 charts: gauge, pies, trends)
- Evidence-grounded LLM prompt requiring quote citations
- Hyper-specific solution generation with WHAT/WHO/WHEN/WHY
- Taxonomy-guided solutions contextualized to business type
- Staff name extraction from quotes for leverage actions
- Success metrics tied to actual complaint keywords
Report sections now include:
- Health score (1-100) with revenue-at-risk estimates
- Risk indicators with color-coded scores and trends
- Critical issues with evidence, root cause, and specific solutions
- Strengths with named staff and specific marketing actions
- Action matrix with effort/impact quadrants and deadlines
- 90-day KPIs with 30/60/90 day targets
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Changed outer button to div with role="button" to avoid HTML validation
error of nested buttons (translate button inside complaint card).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add Stage5Synthesizer class that generates AI narratives and action plans
- Add generate() method to LLMClient for synthesis generation
- Integrate Stage 5 into pipeline runner after route stage
- Add synthesis JSONB column to pipeline.executions table
- Update reviewiq_analytics API to return synthesis data
- Synthesis includes: executive narrative, sentiment/category/timeline insights,
action plan, marketing angles, and priority recommendations
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Frontend:
- Add Synthesis type with action plan, insights, annotations
- ExecutiveSummary: Accept synthesis prop for AI narrative
- SentimentPie: Accept insight prop for contextual explanation
- IntensityHeatmap: Accept insight + highlightDomain props
- TimelineChart: Accept insight + annotations props
- All components gracefully degrade when synthesis is null
Backend:
- Add Stage 4: Synthesize for generating AI narratives
- Gathers context from classified spans
- Generates executive narrative, section insights, action plan
- Produces timeline annotations and marketing angles
- Stores synthesis in pipeline.executions table
Components show AI insights with purple gradient styling when available,
fall back to existing behavior when synthesis is not yet generated.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add 2x2 matrix visualization (Quick Wins, Critical, Strategic, Nice to Have)
- Position items based on frequency/effort coordinates with dots and leader lines
- Add L-shaped axes with arrows showing Frequency (X) and Effort (Y) directions
- Include unified coordinate grid overlay across all quadrants
- Add clickable subcode labels with hover effects
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create ExecutionsView component with TanStack Table
- Add status filter buttons with count badges
- Add action buttons: Analytics, Metrics, Debug
- Add debug modal with AI copy-paste button for failed executions
- Generate detailed debug report with stage metrics and error context
- Update executions page to use new component
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add run pipeline page with job selection UI
- Add execution detail page with stage metrics visualization
- Add stage_metrics and total_duration_ms to pipeline.executions table
- Create Next.js API proxy routes for all pipeline endpoints
- Fix trailing slash issues in pipeline-api.ts URLs
- Add Docker volume mounts for pipeline packages
- Add REVIEWIQ_DATABASE_URL and LLM API keys to docker-compose
- Fix JSONB field parsing in execution detail endpoint
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add version selector dropdown in scrape confirmation modal
- Default to v1.1.0 (Multi-Sort) which bypasses ~1000 review limit
- Pass scraper_version through API proxy to backend
- Update /new page fallback to show v1.1.0 as available
- Show version description explaining multi-sort benefits
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit implements a plugin-like pipeline architecture with:
Pipeline Core Package (packages/pipeline-core/):
- BasePipeline abstract class all pipelines implement
- PipelineRegistry for database-backed discovery/management
- PipelineRunner for execution with status tracking
- DashboardConfig contracts for dynamic widget definitions
Database Migration (006_pipeline_registry.sql):
- pipeline.registry table for registered pipelines
- pipeline.executions table for execution history
- Views for execution stats and monitoring
ReviewIQ Pipeline Refactor:
- Implements BasePipeline interface
- Adds get_dashboard_config() with widget definitions
- Adds get_widget_data() methods for all dashboard widgets
- Maintains backward compatibility with Pipeline alias
Generic Pipeline API (api/routes/pipelines.py):
- GET /api/pipelines - List all registered pipelines
- GET /api/pipelines/{id} - Pipeline details
- POST /api/pipelines/{id}/execute - Execute pipeline
- GET /api/pipelines/{id}/dashboard - Dashboard config
- GET /api/pipelines/{id}/widgets/{w} - Widget data
- GET /api/pipelines/{id}/executions - Execution history
Frontend Dynamic Dashboard System:
- DynamicDashboard component renders from config
- WidgetRegistry maps types to components
- Widget components: StatCard, LineChart, BarChart,
PieChart, DataTable, Heatmap
- Pipeline API client library
Frontend Pipeline Pages:
- /pipelines - List all registered pipelines
- /pipelines/[id] - Dynamic dashboard for pipeline
- /pipelines/[id]/executions - Execution history
- Pipelines nav item in Sidebar
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Import both v1.0.0 and v1.1.0 scraper versions
- Add SCRAPER_VERSIONS registry mapping version strings to functions
- Add get_scraper_for_version() to route based on job metadata
- Default to v1.1.0 (multi-sort) for new jobs
- Frontend can select specific version via scraper_version parameter
- Validation endpoint continues using v1.0.0 for speed
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The multi-sort loop was calling get_dom_reviews() which doesn't exist.
API interception alone is sufficient for capturing reviews during
multi-sort passes, so we now use only api_reviews.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
v1.0.0 improvements:
- Add captcha detection (reCAPTCHA, unusual traffic, challenges)
- Block fonts, analytics, maps tiles for faster scrolling
- Add 95% close-enough threshold to skip unnecessary retries
- Stop immediately if captcha detected instead of retrying
v1.1.0 new features:
- Multi-sort strategy to bypass ~1000 review limit
- Cycles through newest/lowest/highest/relevant sorts
- Auto mode: enables multi-sort when total > 1000
- Diminishing returns detection (stops if <5% new per pass)
- Configurable sort order and thresholds
Also adds test_scraper_v110.py CLI tool for testing multi-sort.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ScraperV1Adapter to transform scraped reviews into pipeline format
- Handles relative timestamps (centerDate)
- Generates deterministic IDs for DOM-sourced reviews
- Filters out empty (rating-only) reviews
- Add sample barbershop reviews (79 reviews, 46 with text)
- Real data from Las Palmas barbershop
- Multi-language: Spanish, English, German, Norwegian, Italian
- Add test_pipeline_real_data.py for E2E testing with real data
- Uses mock classifier based on keywords and rating
- Full pipeline flow: raw -> enriched -> spans -> issues -> facts
Test results with real data:
- 46 reviews processed
- 6 languages detected (es: 35, en: 7, de: 1, no: 1, it: 1, ca: 1)
- 3 issues identified from negative reviews
- 29 fact records aggregated across date range 2017-2025
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Tests the full pipeline flow:
- Stage 1: Insert raw reviews, normalize text
- Stage 2: Mock LLM classification, insert spans
- Stage 3: Route negative spans to issues
- Stage 4: Aggregate facts by URT code and date
Validates all pipeline.* tables are populated correctly.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Import LogCapture from scraper module
- Remove unused StructuredLogger import
- Use correct log_capture parameter name
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Root cause: Cards were hidden but not removed from DOM, causing
memory buildup (400+ nodes) that crashed Chrome tabs.
Changes:
- Actually remove processed cards from DOM (not just hide them)
- Keep last 50 cards for scroll reference/continuity
- Remove adjacent separator elements along with cards
- Add logging when DOM cleanup removes cards
- Cards near scroll end stay visible for reference
This should prevent "tab crashed" errors during long scraping
sessions with 500+ reviews.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use updated_at (last successful data loop) instead of Date.now()
- Speed now reflects actual data retrieval rate, not declining over time
- Updated in table column, monitored job view, and stats row
- Fall back to Date.now() if updated_at is not available
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use fixed positioning with top/left 50% and translate -50%
- More reliable centering regardless of parent containers
- Add max-width for mobile responsiveness
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Keep blue background when isCheckingReviews is true
- Add cursor-wait during validation
- Move disabled styling to explicit condition check
- White spinner now visible on blue background
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Reset search fields after job is successfully launched
- Allow user to immediately start another scrape
- Save active jobs to localStorage for persistence across refresh
- Restore jobs from localStorage on page load
- Resume polling for non-terminal jobs (pending/running)
- Filter out jobs older than 24 hours
- Add remove button (X) to each job card
- Clean up localStorage when jobs are removed
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove w-full that caused alignment issues
- Use fixed width (400px) for consistent centering
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove w-full and mx-auto that caused alignment issues
- Use fixed width (280px) instead of max-w-xs
- Let flex container handle centering
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Center modal properly within map preview area
- Add 24px padding from map edges
- Make modal more compact (max-w-xs)
- Reduce text and element sizes for better fit
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Place Business Name, Location, and Validate button in same row
- Reduce padding and font sizes for compact inline layout
- Show abbreviated text on mobile (responsive)
- Use checkmark indicator for auto-detected location
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Split single search input into two fields: Business Name (required)
and Location (auto-detected from IP geolocation)
- Auto-fill location field with city/country from IP on page load
- Add click overlay on map iframe to prevent interaction
- Add warning modal when user clicks map, directing them to use search
- Update test URLs to use split format
- Make Validate button full-width for better UX
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Restore original Google Maps embed iframe approach
- URL: maps.google.com/maps?q=...&output=embed&z=15
- Add "Open in Maps" overlay button on the map
- Height 300px for better visibility
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace non-working Google Maps embed iframe with animated location preview
- Add "Open in Google Maps" button to open location in new tab
- Add scraper type selection dropdown fetching from /api/admin/scrapers
- Show selected scraper info with formatted labels (Google Reviews v1.0.0)
- Include scraper_version and scraper_variant in job submission
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Dashboard page:
- Fetch top clients from /api/dashboard/by-client
- Show loading state while fetching
- Display empty state when no client data
- Show real client_id, job count, and success rate
Scrapers page:
- Fetch versions from /api/admin/scrapers
- Wire promote/deprecate buttons to real API calls
- Wire add version form to POST /api/admin/scrapers
- Wire traffic allocation to PUT /api/admin/scrapers/{id}/traffic
- Add loading and error states
Dockerfile:
- Add COPY commands for new directories (api/, core/, scrapers/, etc.)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase 5 - Main Dashboard:
- Dashboard overview page with system health stats
- Jobs by status breakdown, success rates, top clients
- Dashboard API (/api/dashboard/overview, by-client, problems, by-version)
Phase 6 - Admin/Scraper Management:
- Scrapers management page with traffic allocation UI
- Admin API for scraper CRUD operations
- Traffic percentage updates for A/B testing
- Promote/deprecate scraper versions
Phase 7 - Authentication:
- API key authentication middleware
- SHA-256 key hashing (keys never stored in plain text)
- Scope-based authorization (jobs:read, jobs:write, admin)
- Rate limiting per API key
Also:
- Updated api_server_production.py to include new routers
- Extended core/database.py with dashboard query methods
- Added dashboard link to sidebar navigation
- Updated CONTEXT-KEEPER.md to mark all phases complete
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Quick-reference document for resuming work after context compaction.
Contains: project overview, current state, spec summary, phases,
key decisions, file locations, and resumption instructions.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Wrap handleJobsChange in useCallback to prevent infinite re-renders
caused by onJobsChange dependency changing on every render.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Task #18: Complete integration of all JobDevTools components
- Updated job detail page (/jobs/[id]) with full JobDevTools UI
- Connected SSE stream for real-time structured logs + metrics
- Added crash-report and retry API routes for Next.js
- Added format conversion for old/new log formats
- Added DevTools links to JobsView modal and actions column
- Wired up CrashReport retry with auto-fix parameters
- Integrated SessionPanel for fingerprint display
- Integrated MetricsDashboard for real-time charts
Job DevTools implementation complete: 18/18 tasks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add chk_dedup_scoped constraint enforcing tenant-scoped dedup format
- Filter location_type='owned' in populate_facts() for 'ALL' rollup
- Document competitor exclusion from 'ALL' sentinel rollups
- Add explicit comments in aggregation code for maintainability
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
