824634aa76
This commit implements a plugin-like pipeline architecture with:
Pipeline Core Package (packages/pipeline-core/):
- BasePipeline abstract class all pipelines implement
- PipelineRegistry for database-backed discovery/management
- PipelineRunner for execution with status tracking
- DashboardConfig contracts for dynamic widget definitions
Database Migration (006_pipeline_registry.sql):
- pipeline.registry table for registered pipelines
- pipeline.executions table for execution history
- Views for execution stats and monitoring
ReviewIQ Pipeline Refactor:
- Implements BasePipeline interface
- Adds get_dashboard_config() with widget definitions
- Adds get_widget_data() methods for all dashboard widgets
- Maintains backward compatibility with Pipeline alias
Generic Pipeline API (api/routes/pipelines.py):
- GET /api/pipelines - List all registered pipelines
- GET /api/pipelines/{id} - Pipeline details
- POST /api/pipelines/{id}/execute - Execute pipeline
- GET /api/pipelines/{id}/dashboard - Dashboard config
- GET /api/pipelines/{id}/widgets/{w} - Widget data
- GET /api/pipelines/{id}/executions - Execution history
Frontend Dynamic Dashboard System:
- DynamicDashboard component renders from config
- WidgetRegistry maps types to components
- Widget components: StatCard, LineChart, BarChart,
PieChart, DataTable, Heatmap
- Pipeline API client library
Frontend Pipeline Pages:
- /pipelines - List all registered pipelines
- /pipelines/[id] - Dynamic dashboard for pipeline
- /pipelines/[id]/executions - Execution history
- Pipelines nav item in Sidebar
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
120 lines
2.8 KiB
Markdown
120 lines
2.8 KiB
Markdown
# Pipeline Core
|
|
|
|
Extensible multi-pipeline framework with dynamic dashboards.
|
|
|
|
## Overview
|
|
|
|
Pipeline Core provides the base abstractions for building pipelines that can be:
|
|
- Discovered and registered dynamically
|
|
- Executed with status tracking
|
|
- Rendered with auto-generated dashboards
|
|
|
|
## Features
|
|
|
|
- **BasePipeline** - Abstract base class all pipelines implement
|
|
- **PipelineRegistry** - Database-backed pipeline discovery and management
|
|
- **PipelineRunner** - Execution with status tracking
|
|
- **Dashboard Contracts** - TypedDicts for widget configuration
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
pip install -e packages/pipeline-core
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Implementing a Pipeline
|
|
|
|
```python
|
|
from pipeline_core import BasePipeline, PipelineMetadata, DashboardConfig
|
|
|
|
class MyPipeline(BasePipeline):
|
|
@property
|
|
def metadata(self) -> PipelineMetadata:
|
|
return {
|
|
"id": "my-pipeline",
|
|
"name": "My Pipeline",
|
|
"description": "Does something useful",
|
|
"version": "1.0.0",
|
|
"stages": ["stage1", "stage2"],
|
|
"input_type": "MyInputType",
|
|
}
|
|
|
|
async def initialize(self) -> None:
|
|
# Set up connections
|
|
pass
|
|
|
|
async def close(self) -> None:
|
|
# Clean up
|
|
pass
|
|
|
|
async def process(self, input_data, stages=None):
|
|
# Run the pipeline
|
|
pass
|
|
|
|
def get_dashboard_config(self) -> DashboardConfig:
|
|
return {
|
|
"pipeline_id": "my-pipeline",
|
|
"title": "My Dashboard",
|
|
"sections": [...]
|
|
}
|
|
|
|
async def get_widget_data(self, widget_id, params):
|
|
# Return widget data
|
|
pass
|
|
```
|
|
|
|
### Registering a Pipeline
|
|
|
|
```python
|
|
from pipeline_core import PipelineRegistry
|
|
import asyncpg
|
|
|
|
pool = await asyncpg.create_pool(database_url)
|
|
registry = PipelineRegistry(pool)
|
|
|
|
await registry.register(
|
|
pipeline_id="my-pipeline",
|
|
name="My Pipeline",
|
|
description="Does something useful",
|
|
version="1.0.0",
|
|
module_path="my_package.pipeline:MyPipeline",
|
|
stages=["stage1", "stage2"],
|
|
input_type="MyInputType",
|
|
)
|
|
```
|
|
|
|
### Executing a Pipeline
|
|
|
|
```python
|
|
from pipeline_core import PipelineRunner
|
|
|
|
runner = PipelineRunner(pool, registry)
|
|
|
|
execution_id, result = await runner.execute(
|
|
pipeline_id="my-pipeline",
|
|
request={
|
|
"input_data": {"key": "value"},
|
|
"stages": ["stage1"],
|
|
}
|
|
)
|
|
```
|
|
|
|
## Dashboard Widgets
|
|
|
|
Pipelines declare dashboard widgets via `get_dashboard_config()`. Available widget types:
|
|
|
|
- `stat_card` - KPI stat card with value and trend
|
|
- `line_chart` - Time series line chart
|
|
- `bar_chart` - Bar chart (horizontal or vertical)
|
|
- `pie_chart` - Pie/donut chart
|
|
- `table` - Data table with columns
|
|
- `heatmap` - Heatmap grid visualization
|
|
- `area_chart` - Stacked area chart
|
|
- `gauge` - Gauge/meter visualization
|
|
|
|
## License
|
|
|
|
MIT
|