Pipeline Core

Extensible multi-pipeline framework with dynamic dashboards.

Overview

Pipeline Core provides the base abstractions for building pipelines that can be:

Discovered and registered dynamically
Executed with status tracking
Rendered with auto-generated dashboards

Features

BasePipeline - Abstract base class all pipelines implement
PipelineRegistry - Database-backed pipeline discovery and management
PipelineRunner - Execution with status tracking
Dashboard Contracts - TypedDicts for widget configuration

Installation

pip install -e packages/pipeline-core

Usage

Implementing a Pipeline

from pipeline_core import BasePipeline, PipelineMetadata, DashboardConfig

class MyPipeline(BasePipeline):
    @property
    def metadata(self) -> PipelineMetadata:
        return {
            "id": "my-pipeline",
            "name": "My Pipeline",
            "description": "Does something useful",
            "version": "1.0.0",
            "stages": ["stage1", "stage2"],
            "input_type": "MyInputType",
        }

    async def initialize(self) -> None:
        # Set up connections
        pass

    async def close(self) -> None:
        # Clean up
        pass

    async def process(self, input_data, stages=None):
        # Run the pipeline
        pass

    def get_dashboard_config(self) -> DashboardConfig:
        return {
            "pipeline_id": "my-pipeline",
            "title": "My Dashboard",
            "sections": [...]
        }

    async def get_widget_data(self, widget_id, params):
        # Return widget data
        pass

Registering a Pipeline

from pipeline_core import PipelineRegistry
import asyncpg

pool = await asyncpg.create_pool(database_url)
registry = PipelineRegistry(pool)

await registry.register(
    pipeline_id="my-pipeline",
    name="My Pipeline",
    description="Does something useful",
    version="1.0.0",
    module_path="my_package.pipeline:MyPipeline",
    stages=["stage1", "stage2"],
    input_type="MyInputType",
)

Executing a Pipeline

from pipeline_core import PipelineRunner

runner = PipelineRunner(pool, registry)

execution_id, result = await runner.execute(
    pipeline_id="my-pipeline",
    request={
        "input_data": {"key": "value"},
        "stages": ["stage1"],
    }
)

Dashboard Widgets

Pipelines declare dashboard widgets via get_dashboard_config(). Available widget types:

stat_card - KPI stat card with value and trend
line_chart - Time series line chart
bar_chart - Bar chart (horizontal or vertical)
pie_chart - Pie/donut chart
table - Data table with columns
heatmap - Heatmap grid visualization
area_chart - Stacked area chart
gauge - Gauge/meter visualization

License

MIT