Phases 5-7: Dashboard UI, Admin API, and Auth middleware

Phase 5 - Main Dashboard:
- Dashboard overview page with system health stats
- Jobs by status breakdown, success rates, top clients
- Dashboard API (/api/dashboard/overview, by-client, problems, by-version)

Phase 6 - Admin/Scraper Management:
- Scrapers management page with traffic allocation UI
- Admin API for scraper CRUD operations
- Traffic percentage updates for A/B testing
- Promote/deprecate scraper versions

Phase 7 - Authentication:
- API key authentication middleware
- SHA-256 key hashing (keys never stored in plain text)
- Scope-based authorization (jobs:read, jobs:write, admin)
- Rate limiting per API key

Also:
- Updated api_server_production.py to include new routers
- Extended core/database.py with dashboard query methods
- Added dashboard link to sidebar navigation
- Updated CONTEXT-KEEPER.md to mark all phases complete

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Alejandro Gutiérrez
2026-01-24 15:43:00 +00:00
parent 788ef84756
commit 39c80fc8be
11 changed files with 3465 additions and 16 deletions

View File

@@ -12,16 +12,23 @@
---
## Current State (as of 2025-01-24)
## Current State (as of 2026-01-24)
### Working Features
- Google Reviews scraper (`modules/scraper_clean.py`) - fully functional
- Google Reviews scraper (`scrapers/google_reviews/v1_0_0.py`) - fully functional
- Job queue with PostgreSQL storage
- Real-time SSE streaming of logs/progress
- Web UI for job management and analytics
- Chrome pool for browser management
- Crash detection and analysis
- JobDevTools observability panel
- **NEW**: Requester tracking (client_id, source, purpose)
- **NEW**: Batch job submission API
- **NEW**: Webhook/callback delivery with retries
- **NEW**: Scraper versioning with A/B traffic routing
- **NEW**: Main dashboard with system health stats
- **NEW**: Admin API for scraper management
- **NEW**: API key authentication middleware
### Repository
- **Location**: `/Users/agutierrez/Desktop/google-reviews-scraper-pro`
@@ -105,14 +112,14 @@ reviewiq/ # Will rename from google-reviews-scraper-pro
|-------|-------------|--------|
| 0 | Project restructure (move files to new locations) | ✅ COMPLETE |
| 1 | Database migrations (new fields + tables) | ✅ COMPLETE |
| 2 | Requester & batch support | Not started |
| 3 | Webhooks | Not started |
| 4 | Scraper versioning & registry | Not started |
| 5 | Main dashboard UI | Not started |
| 6 | A/B traffic management | Not started |
| 7 | Authentication (API keys) | Not started |
| 2 | Requester & batch support | ✅ COMPLETE |
| 3 | Webhooks | ✅ COMPLETE |
| 4 | Scraper versioning & registry | ✅ COMPLETE |
| 5 | Main dashboard UI | ✅ COMPLETE |
| 6 | A/B traffic management (Admin API) | ✅ COMPLETE |
| 7 | Authentication middleware | ✅ COMPLETE |
**Phase 0 must complete first.** Then phases 1-5 can parallelize.
**All phases complete!** Core platform ready for integration testing.
---
@@ -137,12 +144,20 @@ reviewiq/ # Will rename from google-reviews-scraper-pro
## Files to Know
| Current Location | Purpose |
|------------------|---------|
| `modules/scraper_clean.py` | Main Google Reviews scraper (96KB) |
| `modules/database.py` | PostgreSQL database manager |
| `api_server_production.py` | FastAPI server (will be split into api/) |
| Location | Purpose |
|----------|---------|
| `scrapers/google_reviews/v1_0_0.py` | Main Google Reviews scraper (migrated) |
| `scrapers/registry.py` | Scraper version registry with A/B routing |
| `core/database.py` | PostgreSQL database manager |
| `api_server_production.py` | FastAPI server with all routers |
| `api/routes/dashboard.py` | Dashboard API endpoints |
| `api/routes/admin.py` | Admin/scraper management API |
| `api/routes/batches.py` | Batch job submission API |
| `api/middleware/auth.py` | API key authentication middleware |
| `web/app/dashboard/page.tsx` | Main dashboard UI |
| `web/app/dashboard/scrapers/page.tsx` | Scraper management UI |
| `web/app/jobs/[id]/page.tsx` | Job detail page with DevTools |
| `migrations/versions/` | SQL migration files (001-004) |
| `.artifacts/ReviewIQ-Platform-Spec.md` | Full specification document |
---
@@ -177,4 +192,4 @@ When resuming after context compaction:
---
*Last updated: 2025-01-24*
*Last updated: 2026-01-24*