824634aa76
This commit implements a plugin-like pipeline architecture with:
Pipeline Core Package (packages/pipeline-core/):
- BasePipeline abstract class all pipelines implement
- PipelineRegistry for database-backed discovery/management
- PipelineRunner for execution with status tracking
- DashboardConfig contracts for dynamic widget definitions
Database Migration (006_pipeline_registry.sql):
- pipeline.registry table for registered pipelines
- pipeline.executions table for execution history
- Views for execution stats and monitoring
ReviewIQ Pipeline Refactor:
- Implements BasePipeline interface
- Adds get_dashboard_config() with widget definitions
- Adds get_widget_data() methods for all dashboard widgets
- Maintains backward compatibility with Pipeline alias
Generic Pipeline API (api/routes/pipelines.py):
- GET /api/pipelines - List all registered pipelines
- GET /api/pipelines/{id} - Pipeline details
- POST /api/pipelines/{id}/execute - Execute pipeline
- GET /api/pipelines/{id}/dashboard - Dashboard config
- GET /api/pipelines/{id}/widgets/{w} - Widget data
- GET /api/pipelines/{id}/executions - Execution history
Frontend Dynamic Dashboard System:
- DynamicDashboard component renders from config
- WidgetRegistry maps types to components
- Widget components: StatCard, LineChart, BarChart,
PieChart, DataTable, Heatmap
- Pipeline API client library
Frontend Pipeline Pages:
- /pipelines - List all registered pipelines
- /pipelines/[id] - Dynamic dashboard for pipeline
- /pipelines/[id]/executions - Execution history
- Pipelines nav item in Sidebar
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Google Reviews Scraper - Testing Interface
A Next.js web interface for testing the containerized Google Reviews Scraper API.
Features
- 🎯 URL Input - Paste any Google Maps business URL
- 📊 Real-time Status - Live job tracking with polling
- ⚡ Performance Metrics - Reviews count, time, speed
- 📱 Review Display - Beautiful UI for scraped reviews
- 💾 Export JSON - Download reviews as JSON
Quick Start
1. Start the Scraper API
First, make sure the containerized scraper is running:
cd ..
docker-compose -f docker-compose.production.yml up -d
The API should be running at http://localhost:8000
2. Start the Web Interface
npm install
npm run dev
Usage
-
Paste a Google Maps URL
https://www.google.com/maps/place/Business+Name/... -
Click "Scrape"
- Job is submitted to the API
- Status updates in real-time
- Reviews appear when complete
-
View Results
- See all scraped reviews
- Export as JSON
- View performance metrics
Environment Variables
Create .env.local if you need to customize:
# API URL (default: http://localhost:8000)
NEXT_PUBLIC_API_URL=http://localhost:8000
API Endpoints Used
This interface connects to:
POST /scrape- Submit scraping jobGET /jobs/{job_id}- Get job statusGET /jobs/{job_id}/reviews- Get reviews
Tech Stack
- Next.js 15 - React framework
- TypeScript - Type safety
- Tailwind CSS - Styling
- API Proxy - Next.js API routes proxy to scraper API
Development
npm run dev # Start dev server
npm run build # Build for production
npm run start # Start production server
npm run lint # Run ESLint
Notes
- The interface polls job status every 2 seconds
- Polling stops when job completes or fails
- Reviews are fetched with a limit of 1000 by default
- Export button downloads reviews as formatted JSON