Files

Alejandro Gutiérrez 824634aa76 feat: Add extensible multi-pipeline integration system

This commit implements a plugin-like pipeline architecture with:

Pipeline Core Package (packages/pipeline-core/):
- BasePipeline abstract class all pipelines implement
- PipelineRegistry for database-backed discovery/management
- PipelineRunner for execution with status tracking
- DashboardConfig contracts for dynamic widget definitions

Database Migration (006_pipeline_registry.sql):
- pipeline.registry table for registered pipelines
- pipeline.executions table for execution history
- Views for execution stats and monitoring

ReviewIQ Pipeline Refactor:
- Implements BasePipeline interface
- Adds get_dashboard_config() with widget definitions
- Adds get_widget_data() methods for all dashboard widgets
- Maintains backward compatibility with Pipeline alias

Generic Pipeline API (api/routes/pipelines.py):
- GET /api/pipelines - List all registered pipelines
- GET /api/pipelines/{id} - Pipeline details
- POST /api/pipelines/{id}/execute - Execute pipeline
- GET /api/pipelines/{id}/dashboard - Dashboard config
- GET /api/pipelines/{id}/widgets/{w} - Widget data
- GET /api/pipelines/{id}/executions - Execution history

Frontend Dynamic Dashboard System:
- DynamicDashboard component renders from config
- WidgetRegistry maps types to components
- Widget components: StatCard, LineChart, BarChart,
  PieChart, DataTable, Heatmap
- Pipeline API client library

Frontend Pipeline Pages:
- /pipelines - List all registered pipelines
- /pipelines/[id] - Dynamic dashboard for pipeline
- /pipelines/[id]/executions - Execution history
- Pipelines nav item in Sidebar

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-01-24 19:05:38 +00:00

2.8 KiB

Raw Blame History

Pipeline Core

Extensible multi-pipeline framework with dynamic dashboards.

Overview

Pipeline Core provides the base abstractions for building pipelines that can be:

Discovered and registered dynamically
Executed with status tracking
Rendered with auto-generated dashboards

Features

BasePipeline - Abstract base class all pipelines implement
PipelineRegistry - Database-backed pipeline discovery and management
PipelineRunner - Execution with status tracking
Dashboard Contracts - TypedDicts for widget configuration

Installation

pip install -e packages/pipeline-core

Usage

Implementing a Pipeline

from pipeline_core import BasePipeline, PipelineMetadata, DashboardConfig

class MyPipeline(BasePipeline):
    @property
    def metadata(self) -> PipelineMetadata:
        return {
            "id": "my-pipeline",
            "name": "My Pipeline",
            "description": "Does something useful",
            "version": "1.0.0",
            "stages": ["stage1", "stage2"],
            "input_type": "MyInputType",
        }

    async def initialize(self) -> None:
        # Set up connections
        pass

    async def close(self) -> None:
        # Clean up
        pass

    async def process(self, input_data, stages=None):
        # Run the pipeline
        pass

    def get_dashboard_config(self) -> DashboardConfig:
        return {
            "pipeline_id": "my-pipeline",
            "title": "My Dashboard",
            "sections": [...]
        }

    async def get_widget_data(self, widget_id, params):
        # Return widget data
        pass

Registering a Pipeline

from pipeline_core import PipelineRegistry
import asyncpg

pool = await asyncpg.create_pool(database_url)
registry = PipelineRegistry(pool)

await registry.register(
    pipeline_id="my-pipeline",
    name="My Pipeline",
    description="Does something useful",
    version="1.0.0",
    module_path="my_package.pipeline:MyPipeline",
    stages=["stage1", "stage2"],
    input_type="MyInputType",
)

Executing a Pipeline

from pipeline_core import PipelineRunner

runner = PipelineRunner(pool, registry)

execution_id, result = await runner.execute(
    pipeline_id="my-pipeline",
    request={
        "input_data": {"key": "value"},
        "stages": ["stage1"],
    }
)

Dashboard Widgets

Pipelines declare dashboard widgets via get_dashboard_config(). Available widget types:

stat_card - KPI stat card with value and trend
line_chart - Time series line chart
bar_chart - Bar chart (horizontal or vertical)
pie_chart - Pie/donut chart
table - Data table with columns
heatmap - Heatmap grid visualization
area_chart - Stacked area chart
gauge - Gauge/meter visualization

License

MIT

2.8 KiB Raw Blame History