# ReviewIQ Pipeline Improvement Suggestions

Based on validation testing and analysis of the classification pipeline.

---

## 🔴 High Priority (Quality & Cost Impact)

### 1. Multi-Aspect Detection Gap
**Problem**: LLM misses secondary codes in multi-aspect reviews.
- "not too expensive" → V4.01 missed
- "easy and fast" → J1.01 missed

**Solution**: Update classification prompt to:
```
For reviews with multiple distinct topics:
1. Extract ALL aspects, not just the dominant one
2. Assign urt_secondary codes for each additional aspect
3. Flag reviews with 3+ aspects as "complex"
```

**Impact**: ~15-20% of reviews have multiple aspects being partially captured.

---

### 2. Enable Smart Router (Cost Savings)
**Problem**: All reviews go through expensive Sonnet model.

**Solution**: Enable the implemented router:
```python
Config(
    router_enabled=True,
    router_conservative=True,
    router_cheap_model="claude-3-5-haiku-20241022",
)
```

**Impact**:
- SKIP (1.6%): $0 cost (was ~$0.05)
- CHEAP (31.4%): ~10x cheaper with Haiku
- **Estimated 25-30% cost reduction**

---

### 3. JSON Truncation Recovery
**Problem**: ~33% of batches hit JSON truncation, causing partial failures.

**Current State**: Partial recovery implemented but still loses some reviews.

**Solution**:
1. Reduce batch size when reviews are long
2. Add `max_tokens` buffer based on expected output
3. Implement streaming JSON parser for real-time recovery

```python
# Dynamic batch sizing based on review length
if avg_review_length > 200:
    batch_size = min(batch_size, 15)
if avg_review_length > 500:
    batch_size = min(batch_size, 8)
```

**Impact**: Reduce fallback processing by ~50%, saving time and cost.

---

## 🟡 Medium Priority (Reliability & Accuracy)

### 4. LLM Response Caching
**Problem**: Retries reprocess already-classified reviews.

**Solution**: Cache successful LLM responses by content hash:
```python
class ResponseCache:
    async def get(self, text_hash: str) -> dict | None:
        return await redis.get(f"llm:classify:{text_hash}")

    async def set(self, text_hash: str, response: dict, ttl: int = 86400):
        await redis.setex(f"llm:classify:{text_hash}", ttl, json.dumps(response))
```

**Impact**:
- Zero cost for re-runs on same reviews
- Faster pipeline retries
- Useful for A/B testing prompts

---

### 5. Confidence-Based Routing
**Problem**: LLM assigns codes even when uncertain.

**Solution**: Add confidence threshold in prompt:
```
If confidence < 70%:
  - Set confidence: "low"
  - Use generic code (V4.03) instead of guessing
  - Flag for human review
```

**Impact**: Reduces misclassifications, improves data quality.

---

### 6. Post-Classification Validation
**Problem**: Some classifications don't match review content.

**Solution**: Add rule-based validation layer:
```python
def validate_classification(text: str, urt_code: str) -> bool:
    # Price mentioned but not V4.xx code?
    if has_price_mention(text) and not urt_code.startswith("V4"):
        return False, "V4.01"  # Suggest correction

    # Staff mentioned but not P1.xx code?
    if has_staff_mention(text) and not urt_code.startswith("P1"):
        return False, "P1.01"

    return True, None
```

**Impact**: Catch ~5-10% of obvious misclassifications.

---

### 7. Span Coverage Validation
**Problem**: Some review text not covered by any span.

**Solution**: Track span coverage percentage:
```python
def calculate_coverage(text: str, spans: list) -> float:
    covered_chars = set()
    for span in spans:
        covered_chars.update(range(span['start'], span['end']))
    return len(covered_chars) / len(text)

# Flag if coverage < 60%
if coverage < 0.6:
    log.warning(f"Low span coverage: {coverage:.0%}")
```

**Impact**: Identify reviews where LLM skipped important content.

---

## 🟢 Lower Priority (Optimization & Monitoring)

### 8. Taxonomy Alignment Scoring
**Problem**: Hard to measure classification quality at scale.

**Solution**: Build automated taxonomy alignment checker:
```python
# Check if keywords in text match expected domain
DOMAIN_KEYWORDS = {
    "V4": ["price", "money", "worth", "cost", "expensive", "cheap"],
    "P1": ["staff", "employee", "service", "friendly", "rude"],
    "J1": ["wait", "fast", "slow", "quick", "time", "minutes"],
    "E1": ["clean", "dirty", "comfortable", "space", "room"],
}

def alignment_score(text: str, urt_code: str) -> float:
    domain = urt_code[0:2]
    keywords = DOMAIN_KEYWORDS.get(domain, [])
    matches = sum(1 for kw in keywords if kw in text.lower())
    return matches / len(keywords) if keywords else 0.5
```

**Impact**: Quality dashboard, regression detection.

---

### 9. Batch Size Auto-Tuning
**Problem**: Fixed batch size doesn't adapt to review complexity.

**Solution**: Implement adaptive batch sizing:
```python
class AdaptiveBatchSizer:
    def __init__(self):
        self.history = []  # (batch_size, success_rate, avg_tokens)

    def recommend_size(self, reviews: list) -> int:
        avg_length = sum(len(r['text']) for r in reviews) / len(reviews)

        # Learn from history
        if self.history:
            # Find optimal size for similar review lengths
            similar = [h for h in self.history if abs(h['avg_len'] - avg_length) < 50]
            if similar:
                return max(h['size'] for h in similar if h['success_rate'] > 0.95)

        # Default heuristics
        if avg_length > 300:
            return 10
        elif avg_length > 150:
            return 20
        else:
            return 30
```

---

### 10. Cost Tracking Dashboard
**Problem**: No visibility into per-job, per-stage costs.

**Solution**: Add cost tracking to pipeline output:
```python
@dataclass
class CostBreakdown:
    stage: str
    model: str
    input_tokens: int
    output_tokens: int
    cached_tokens: int
    cost_usd: float
    reviews_processed: int
    cost_per_review: float

# Store in database
CREATE TABLE pipeline.cost_tracking (
    id SERIAL PRIMARY KEY,
    execution_id UUID,
    job_id UUID,
    stage VARCHAR(50),
    model VARCHAR(100),
    input_tokens INT,
    output_tokens INT,
    cached_tokens INT,
    cost_usd DECIMAL(10, 6),
    reviews_processed INT,
    created_at TIMESTAMP DEFAULT NOW()
);
```

---

### 11. Streaming Classification
**Problem**: Large batches block until complete.

**Solution**: Implement streaming for real-time progress:
```python
async def classify_streaming(reviews: list):
    async for partial_result in llm_client.stream_batch(reviews):
        # Yield each review as it completes
        yield partial_result

        # Persist immediately
        await persist_classification(partial_result)
```

**Impact**: Better UX, faster partial results, resilience to failures.

---

### 12. A/B Testing Framework
**Problem**: Hard to compare prompt/model changes.

**Solution**: Built-in A/B testing:
```python
class ABTestConfig:
    test_name: str
    variant_a: ClassificationConfig  # Control
    variant_b: ClassificationConfig  # Treatment
    split_ratio: float = 0.1  # 10% to treatment
    metrics: list[str] = ["accuracy", "cost", "latency"]

# Run both variants on same reviews
results_a = await classify(reviews, config_a)
results_b = await classify(reviews[:int(len(reviews)*0.1)], config_b)

# Compare metrics
compare_results(results_a, results_b)
```

---

## Implementation Priority Matrix

| Improvement | Effort | Impact | Priority |
|-------------|--------|--------|----------|
| 1. Multi-Aspect Detection | Medium | High | 🔴 P1 |
| 2. Enable Smart Router | Low | High | 🔴 P1 |
| 3. JSON Truncation Fix | Medium | High | 🔴 P1 |
| 4. Response Caching | Medium | Medium | 🟡 P2 |
| 5. Confidence Routing | Medium | Medium | 🟡 P2 |
| 6. Post-Classification Validation | Low | Medium | 🟡 P2 |
| 7. Span Coverage Validation | Low | Low | 🟢 P3 |
| 8. Taxonomy Alignment | Medium | Low | 🟢 P3 |
| 9. Adaptive Batch Sizing | High | Medium | 🟢 P3 |
| 10. Cost Dashboard | Medium | Low | 🟢 P3 |
| 11. Streaming Classification | High | Medium | 🟢 P3 |
| 12. A/B Testing | High | Low | 🟢 P3 |

---

## Quick Wins (Can implement today)

1. **Enable router** - Already implemented, just needs config flag
2. **Reduce batch size** - Change `classification_batch_size=15` for long reviews
3. **Add span coverage logging** - Simple metric to track quality
4. **Post-classification keyword check** - Basic validation rules

---

## Estimated Impact Summary

| Area | Current | After Improvements |
|------|---------|-------------------|
| Cost per 1000 reviews | ~$3.40 | ~$2.40 (-30%) |
| Classification accuracy | ~85% | ~92% |
| Multi-aspect capture | ~65% | ~90% |
| Batch failure rate | ~33% | ~10% |
| Pipeline retry cost | 100% | ~20% (with caching) |