alezmad/whyrating-engine-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add complete URT v5.1 taxonomy framework (11 artifacts)

Browse Source 
Universal Review Taxonomy v5.1 implementation with:
- Track A (Training): A1 Quickstart, A2 QA Protocol, A3 Calibration Set, A4 Full Manual
- Track B (Engineering): B1 Code Registry, B2 Database Schema, B3 Owner Routing, B4 API Contract
- Track C (Analytics): C1 Issue Lifecycle, C2 KPI Mapping Guide
- Track D (Integration): D1 Dashboard Specification

Covers 7 domains, 28 categories, 138 subcodes, 16 causal codes, and 7 metadata dimensions.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Alejandro Gutiérrez
2026-01-24 10:51:41 +00:00
parent a540ab97b1
commit 3eda9bdbfa
13 changed files with 21264 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1997
urt-taxonomy/spec/URT-Specification-v5.1.md Normal file
View File

File diff suppressed because it is too large Load Diff

925
urt-taxonomy/track-a-training/A1-Annotator-Quickstart.md Normal file
View File

@@ -0,0 +1,925 @@
# A1: Annotator Quickstart Guide
## Universal Review Taxonomy (URT) v5.1
**Purpose**: Quick reference for human annotators to classify review spans
**Version**: 5.1 | **Status**: Production Ready | **Date**: 2026-01-23
---
## Table of Contents
1. [Quick Decision Tree](#1-quick-decision-tree)
2. [Domain Summary Table](#2-domain-summary-table)
3. [Category Quick Reference (28 Categories)](#3-category-quick-reference)
- [3.5 High-Frequency Subcodes](#35-high-frequency-subcodes-quick-reference)
4. [Metadata Cheat Sheet](#4-metadata-cheat-sheet)
5. [Common Mistakes to Avoid](#5-common-mistakes-to-avoid)
6. [V vs R Rule: Scam vs Rip-off](#6-v-vs-r-rule-scam-vs-rip-off)
7. [J4 vs R3 Rule: Process vs Ownership](#7-j4-vs-r3-rule-process-vs-ownership)
8. [Span Boundary Rules](#8-span-boundary-rules)
9. [Worked Examples](#9-worked-examples)
10. [USN Quick Reference](#10-usn-quick-reference)
11. [Billing/Returns/Refunds Decision Table](#11-billingreturnsrefunds-decision-table)
12. [Data Privacy/Security Decision Table](#12-data-privacysecurity-decision-table)
13. [CR Trigger Word Reference](#13-cr-trigger-word-reference)
14. [Causal Chain Quick Guide](#14-causal-chain-quick-guide-full-profile-only)
15. [Inter-Annotator Agreement Rules](#15-inter-annotator-agreement-rules)
---
## 1. Quick Decision Tree
Use this flowchart for every span. Go top-to-bottom; first "YES" wins.
```
START: Read the span
|
v
+-----------------------------------------------+
| Is it about the THING they bought/received? |
| (product, service, treatment, outcome) |
+-----------------------------------------------+
| YES --> O (Offering)
| NO |
v
+-----------------------------------------------+
| Is it about HOW PEOPLE behaved? |
| (staff attitude, skill, communication) |
+-----------------------------------------------+
| YES --> P (People)
| NO |
v
+-----------------------------------------------+
| Is it about TIME, STEPS, or FRICTION? |
| (waiting, process, ease, resolution steps) |
+-----------------------------------------------+
| YES --> J (Journey)
| NO |
v
+-----------------------------------------------+
| Is it about the SPACE or INTERFACE? |
| (physical place, app/website, ambiance) |
+-----------------------------------------------+
| YES --> E (Environment)
| NO |
v
+-----------------------------------------------+
| Is it about ABILITY TO ACCESS or INCLUSION? |
| (availability, disability, language, bias) |
+-----------------------------------------------+
| YES --> A (Access)
| NO |
v
+-----------------------------------------------+
| Is it about PRICE, COST, or WORTH? |
| (money, effort, value, transparency) |
+-----------------------------------------------+
| YES --> V (Value)
| NO |
v
+-----------------------------------------------+
| Is it about TRUST, PATTERNS, or LOYALTY? |
| (honesty over time, brand, recovery, bond) |
+-----------------------------------------------+
| YES --> R (Relationship)
| NO |
v
RE-READ SPAN -- may need splitting
```
**Memory Aid**: O-P-J-E-A-V-R = "**O**ffer **P**eople a **J**ourney in an **E**nvironment with **A**ccess to **V**alue and **R**elationship"
---
## 2. Domain Summary Table
| Code | Domain | Core Question | One-Line Definition | Example (+) | Example (-) |
|------|--------|---------------|---------------------|-------------|-------------|
| **O** | Offering | Does it work? | The core product/service delivered | "Perfectly cooked steak" | "Phone won't charge" |
| **P** | People | How did they treat me? | Human interactions and behavior | "So helpful and patient" | "Rude and dismissive" |
| **J** | Journey | Was it smooth? | Process, timing, friction | "Seated immediately" | "Waited 45 minutes" |
| **E** | Environment | Is the space okay? | Physical, digital, ambient context | "Beautiful clean space" | "App keeps crashing" |
| **A** | Access | Can I get it? | Availability, accessibility, inclusion | "Open 24/7" | "No wheelchair ramp" |
| **V** | Value | Is it worth it? | Cost, pricing, and worth | "Great bang for buck" | "Total rip-off" |
| **R** | Relationship | Can I trust them? | Trust, reliability, loyalty | "Always reliable" | "Shady company" |
### Default Owner Routing
| Domain | Default Owner |
|--------|---------------|
| O | Product / Operations |
| P | HR / Training |
| J | Operations / Process |
| E | Facilities / IT |
| A | Compliance / Design |
| V | Finance / Pricing |
| R | Leadership / CX |
---
## 3. Category Quick Reference
**28 Categories** - 4 per domain. Use this for URT-Core or to narrow down to subcodes.
### O - Offering (4 categories, 18 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **O1** | Function | Does it do what it's supposed to do? |
| **O2** | Quality | How well is it made or executed? |
| **O3** | Completeness | Is everything included that should be? |
| **O4** | Fit | Does it match customer's specific needs? |
### P - People (4 categories, 20 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **P1** | Attitude | Disposition, manner, emotional tone of staff |
| **P2** | Competence | Knowledge, skill, professional capability |
| **P3** | Responsiveness | Attentiveness, initiative, follow-through |
| **P4** | Communication | Quality of information exchange |
### J - Journey (4 categories, 20 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **J1** | Timing | Speed, punctuality, time management |
| **J2** | Ease | Effort required and friction encountered |
| **J3** | Reliability | Consistency and predictability of process |
| **J4** | Resolution | How problems are handled when they arise |
### E - Environment (4 categories, 20 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **E1** | Physical Space | Tangible environment attributes |
| **E2** | Digital Space | Online and application interface |
| **E3** | Ambiance | Intangible environmental qualities (mood, noise) |
| **E4** | Safety | Security and wellbeing factors |
### A - Access (4 categories, 20 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **A1** | Availability | Can you get it when you need it? |
| **A2** | Accessibility | Can everyone use it regardless of ability? |
| **A3** | Inclusivity | Does it work for diverse backgrounds? |
| **A4** | Convenience | Is it easy to reach and engage with? |
### V - Value (4 categories, 20 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **V1** | Price | The monetary cost itself |
| **V2** | Transparency | Clarity and honesty about costs |
| **V3** | Effort | Non-monetary costs (time, hassle, mental load) |
| **V4** | Worth | Overall value assessment |
### R - Relationship (4 categories, 20 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **R1** | Integrity | Honesty and ethical behavior |
| **R2** | Dependability | Consistency over time |
| **R3** | Recovery | Response to failures (ownership, apology, compensation) |
| **R4** | Loyalty | Investment in ongoing relationship |
---
## 3.5 High-Frequency Subcodes Quick Reference
These are the most commonly used subcodes across all domains. Memorize these first.
### Offering (O) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| O1.01 | Works/Doesn't Work | Basic functionality | "Software runs perfectly" | "Car won't start" |
| O1.02 | Performance Level | How well it operates | "Incredibly fast" | "Sluggish and laggy" |
| O1.05 | Outcome Achievement | Did customer accomplish goal? | "Passed my exam!" | "Treatment didn't work" |
| O2.01 | Materials/Inputs | Quality of components | "Real leather" | "Cheap plastic parts" |
| O2.03 | Presentation | Visual/aesthetic quality | "Gorgeous plating" | "Looked thrown together" |
| O2.05 | Condition at Delivery | State when received | "Still warm" | "Arrived damaged" |
| O4.01 | Specification Match | Matches what was ordered | "Exactly right" | "Wrong size delivered" |
### People (P) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| P1.01 | Warmth/Friendliness | Approachability | "So welcoming" | "Cold and unfriendly" |
| P1.02 | Respect | Treating with dignity | "Made me feel valued" | "Talked down to me" |
| P1.03 | Empathy | Understanding situation | "Really got my frustration" | "Couldn't care less" |
| P2.01 | Knowledge | Understanding of products | "Knew every detail" | "Couldn't answer basics" |
| P2.03 | Problem-Solving | Ability to address issues | "Found creative solution" | "Just said 'can't'" |
| P3.01 | Attentiveness | Awareness of needs | "Always checking in" | "Had to flag them down" |
| P3.03 | Availability | Present when needed | "Easy to find someone" | "Impossible to reach" |
| P4.01 | Clarity | Understandable info | "Explained clearly" | "Confusing jargon" |
### Journey (J) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| J1.01 | Wait Time | Time spent waiting | "Seated immediately" | "45 min past appointment" |
| J1.02 | Service Speed | Time for delivery | "Next day delivery" | "Took three weeks" |
| J1.03 | Response Time | Time to address inquiries | "Replied in minutes" | "Days for a response" |
| J2.01 | Simplicity | Straightforward processes | "So easy to do" | "Needlessly complicated" |
| J3.02 | Accuracy | Correct execution | "Order exactly right" | "Wrong items delivered" |
| J4.02 | Resolution Process | How problems handled | "Clear escalation path" | "Transferred in circles" |
| J4.03 | Resolution Speed | Time to fix | "Fixed same day" | "Took weeks to resolve" |
### Environment (E) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| E1.01 | Cleanliness | Hygiene and tidiness | "Spotless facilities" | "Filthy bathrooms" |
| E1.02 | Maintenance | Condition and upkeep | "Everything works" | "Broken equipment" |
| E2.01 | Interface Design | Visual/interaction quality | "Beautiful app" | "Cluttered mess" |
| E2.02 | Functionality | Features working | "Everything works" | "Buttons broken" |
| E3.01 | Atmosphere/Vibe | Overall mood | "Calm and relaxing" | "Stressful chaos" |
| E4.01 | Physical Safety | Protection from harm | "Felt completely safe" | "Dangerous conditions" |
### Access (A) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| A1.01 | Operating Hours | When accessible | "Open 24/7" | "Banker's hours only" |
| A1.02 | Booking Access | Ability to schedule | "Easy online booking" | "3 weeks for appointment" |
| A1.03 | Inventory/Capacity | Available | "Always in stock" | "Perpetually sold out" |
| A2.01 | Physical Accessibility | Mobility accommodations | "Wheelchair accessible" | "No ramps or elevators" |
| A3.01 | Language Support | Multiple languages | "Staff spoke my language" | "English only, no help" |
| A4.01 | Location | Physical accessibility | "Convenient location" | "Middle of nowhere" |
| A4.02 | Parking | Vehicle accommodation | "Easy free parking" | "Parking nightmare" |
### Value (V) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| V1.01 | Absolute Price | The actual cost | "Very affordable" | "Outrageously expensive" |
| V1.04 | Hidden Costs | Unexpected charges | "No hidden fees" | "Surprise $50 charge" |
| V2.01 | Pricing Clarity | Understanding costs | "Clear price list" | "Impossible to understand" |
| V2.03 | Advertising Accuracy | Marketing matches reality | "As advertised" | "Bait and switch" |
| V2.04 | Terms Fairness | Policy reasonableness | "Fair cancellation" | "Predatory contract" |
| V4.01 | Overall Value | Total assessment | "Excellent value" | "Total rip-off" |
| V4.02 | Quality-Price Ratio | What you get for payment | "Great for the price" | "Pay more, get less" |
### Relationship (R) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| R1.01 | Truthfulness | Accurate representations | "Everything as stated" | "They flat out lied" |
| R1.02 | Promise Keeping | Honoring commitments | "Always deliver on word" | "Never keep promises" |
| R1.04 | Ethics | Moral business conduct | "Ethical company" | "Shady business practices" |
| R2.02 | Consistency | Same experience over time | "Always great" | "Varies wildly" |
| R3.01 | Acknowledgment | Admitting failures | "Owned their mistake" | "Denied any wrongdoing" |
| R3.03 | Compensation | Making amends | "More than made up for it" | "Offered nothing" |
| R4.01 | Recognition | Acknowledging repeat customers | "They remember me" | "Stranger every time" |
---
## 4. Metadata Cheat Sheet
**7 dimensions, 24 total values**. Profile determines which are required.
### 4.1 Valence (V) - ALWAYS REQUIRED
| Code | Label | When to Use | Markers |
|------|-------|-------------|---------|
| **V+** | Positive | Praise, satisfaction | "great," "loved," "excellent" |
| **V-** | Negative | Complaint, dissatisfaction | "terrible," "hated," "worst" |
| **V0** | Neutral | Factual observation | "they have," "it was" |
| **V+-** | Mixed | Both positive and negative in same span | "good but," "despite," "however" |
**Rule**: Prefer splitting over V+- unless the positive and negative target the *same thing*.
### 4.2 Intensity (I) - ALWAYS REQUIRED
| Code | Label | When to Use | Markers |
|------|-------|-------------|---------|
| **I1** | Mild | Slight preference/concern | "a bit," "somewhat," "could be better" |
| **I2** | Moderate | Clear but not extreme | Standard adjectives, no intensifiers |
| **I3** | Strong | Emphatic, intense | "extremely," "worst ever," CAPS, !!!, profanity |
### 4.3 Specificity (S) - Standard+
| Code | Label | When to Use | Example |
|------|-------|-------------|---------|
| **S1** | Vague | General impression only | "Service was bad" |
| **S2** | Moderate | Some details or context | "Service was slow at dinner" |
| **S3** | Specific | Names, times, amounts | "Waiter John took 40 mins at 7pm Saturday" |
### 4.4 Actionability (A) - Standard+
| Code | Label | When to Use | Example |
|------|-------|-------------|---------|
| **A1** | Low | Feeling, no clear action path | "I just didn't like it" |
| **A2** | Medium | Suggests improvement area | "Food could be warmer" |
| **A3** | High | Specific implementable fix | "West bathroom stall lock is broken" |
**Note**: S and A are correlated but not identical. "John was rude" = S3 (named person) but A2 (action unclear).
### 4.5 Temporal Reference (T) - Standard+
| Code | Label | When to Use |
|------|-------|-------------|
| **TC** | Current | This specific visit/experience (default) |
| **TR** | Recent | Recent pattern of experiences |
| **TH** | Historical | Long-standing pattern |
| **TF** | Future | Expectations or predictions |
### 4.6 Evidence Type (E) - Standard+
| Code | Label | When to Use | Constraint |
|------|-------|-------------|------------|
| **ES** | Stated | Explicitly said by customer | Text directly says it (default) |
| **EI** | Inferred | Logically entailed by text | Must be directly deducible, not speculative |
| **EC** | Contextual | Requires surrounding text | Span uses "that," "they," "it" referencing earlier |
**EI Rule**: Only use when text *logically entails* the inference.
- OK: "Took 3 weeks to reply" = EI for slow response time
- NOT OK: "Seemed tired" = Cannot infer fatigue without more evidence
### 4.7 Comparative Reference (CR) - Standard+
| Code | Label | When to Use | Trigger Words |
|------|-------|-------------|---------------|
| **CR-N** | None | No comparison to past (DEFAULT - 90% of spans) | (no temporal language) |
| **CR-B** | Better | Explicit improvement | "better now," "improved," "finally fixed" |
| **CR-W** | Worse | Explicit decline | "worse now," "used to be good," "gone downhill" |
| **CR-S** | Same | Explicitly unchanged | "still," "as always," "same as before" |
**CR Rules**:
1. Only assign CR-B/W/S when customer *explicitly compares* to their own past
2. CR-N is default -- only change when comparison is explicit
3. Self-comparison only (not competitor comparison)
4. Implicit decline counts: "used to be good" = CR-W
### Metadata by Profile
| Dimension | Lite | Core | Standard | Full |
|-----------|:----:|:----:|:--------:|:----:|
| Valence | Required | Required | Required | Required |
| Intensity | Optional | Required | Required | Required |
| Specificity | -- | -- | Required | Required |
| Actionability | -- | -- | Required | Required |
| Temporal | -- | Optional | Required | Required |
| Evidence | -- | -- | Required | Required |
| Comparative | -- | Optional | Required | Required |
---
## 5. Common Mistakes to Avoid
### Top 10 Disambiguation Pitfalls
| # | Mistake | Correct Approach |
|---|---------|------------------|
| 1 | **Confusing O1.01 (Works/Doesn't Work) with J3.03 (System Uptime)** | O1.01 = product function. J3.03 = system/service availability. "Car won't start" = O1.01. "Website was down" = J3.03. |
| 2 | **Confusing P3.04 (Follow-Through) with R1.02 (Promise Keeping)** | P3.04 = specific interaction ("said they'd call back, didn't"). R1.02 = trust/pattern ("they never keep their word"). |
| 3 | **Confusing J2.01 (Simplicity) with E2.04 (Navigation)** | J2 = effort/friction in *process* ("too many steps"). E2 = qualities of *interface* ("buttons don't work"). |
| 4 | **Confusing J3.01 (Consistency) with O1.04 (Reliability)** | J3.01 = process delivers same result each time. O1.04 = product itself works consistently. "Pizza is always good" = J3.01. "Phone always connects" = O1.04. |
| 5 | **Confusing E2.03 (Interface Performance) with O1.02 (Product Performance)** | E2.03 = the app/interface is slow. O1.02 = the core product performs poorly ("car accelerates slowly"). |
| 6 | **Confusing A3.05 (Equal Treatment) with P1.02 (Respect)** | A3.05 = identity-based discrimination ("treated differently because of X"). P1.02 = general disrespect without identity framing. |
| 7 | **Confusing V2.xx with R1.xx** | V2 = pricing/terms/advertising specifically. R1 = trust/integrity/organizational character framing. |
| 8 | **Over-splitting cause-effect spans** | "Long wait because understaffed" = single span. Primary = J1.01, Secondary = A1.04. Only split if different sentences or genuinely different issues. |
| 9 | **Assigning CR-B/W/S without explicit comparison** | CR-N is default. "Service is slow" = CR-N. "Service is *still* slow" = CR-S. Must have explicit temporal language. |
| 10 | **Inventing causal codes** | Causal chain only when text explicitly states or logically entails. "Probably a training issue" = annotator speculation = NO causal code. |
### Quick Disambiguation Tests
| Question | If YES | If NO |
|----------|--------|-------|
| Does it mention the product/service itself? | Likely O | Check P-R |
| Does it name or describe a person? | Likely P | Check J-R |
| Does it mention time, steps, or waiting? | Likely J | Check E-R |
| Does it describe a place or interface? | Likely E | Check A-R |
| Does it mention accessibility or availability? | Likely A | Check V-R |
| Does it mention price, cost, or worth? | Likely V | Check R |
| Does it mention trust, patterns, or "always/never"? | Likely R | Re-read span |
---
## 6. V vs R Rule: Scam vs Rip-off
This is the most common confusion. Use this anchor:
### The Rule
| Feedback Type | Domain | Rationale |
|---------------|--------|-----------|
| **"This is a rip-off"** | **V4.01** | Complaint about the *exchange* (poor value) |
| **"This company is a scam"** | **R1.04** | Moral judgment about *intent* (they're dishonest) |
### Criminal Metaphor Mapping
| Word/Phrase | Domain | Reasoning |
|-------------|--------|-----------|
| "scam" | R1.04 | Character judgment |
| "fraud" | R1.04 | Character judgment |
| "crooks" | R1.04 | Character judgment |
| "rip-off" | V4.01 | Exchange complaint |
| "robbery" | V4.01 | Exchange complaint |
| "highway robbery" | V4.01 | Exchange complaint |
| "overpriced" | V1.01 or V4.02 | Price/value complaint |
### Decision Heuristic
**Ask**: Is the complaint about...
- **What I got vs. what I paid?** --> **V** (Value)
- **Their intent to deceive/harm me?** --> **R** (Relationship)
### Examples
| Span | Primary | Rationale |
|------|---------|-----------|
| "Total scam company" | R1.04 | Moral judgment about intent |
| "They charged me twice" | J3.02 | Process accuracy issue |
| "refused to refund" | R3.03 | Recovery/compensation failure |
| "This is robbery" | V4.01 | Exchange complaint |
| "They deliberately mislead people" | R1.01 | Organizational truthfulness |
| "Overpriced garbage" | V4.02 | Quality-price ratio |
---
## 7. J4 vs R3 Rule: Process vs Ownership
Both J4 and R3 deal with problems, but from different angles.
### The Distinction
| Domain | Focus | Key Question |
|--------|-------|--------------|
| **J4 (Resolution)** | *What they did* to fix it | How was the problem handled operationally? |
| **R3 (Recovery)** | *How they took responsibility* | Did they own it, apologize, make amends? |
### Category-Level Comparison
| J4 Category | R3 Category | Distinction |
|-------------|-------------|-------------|
| J4.01 Problem Acknowledgment | R3.01 Admitting Failures | J4.01 = recognized the issue exists. R3.01 = admitted wrongdoing/fault. |
| J4.02 Resolution Process | -- | J4.02 = the operational steps taken |
| J4.03 Resolution Speed | -- | J4.03 = how fast they fixed it |
| J4.04 Resolution Quality | R3.03 Compensation | J4.04 = was fix adequate? R3.03 = did they make amends beyond fixing? |
| J4.05 Prevention | R3.04 Improvement | J4.05 = process change. R3.04 = organizational commitment to change. |
### Decision Heuristic
**Ask**: Is the feedback about...
- **The mechanics of the fix?** --> **J4** (Journey: Resolution)
- **Their accountability and making things right?** --> **R3** (Relationship: Recovery)
### Examples
| Span | Primary | Rationale |
|------|---------|-----------|
| "They immediately sent a replacement" | J4.02 or R3.03 | If emphasizing speed/process = J4. If emphasizing making amends = R3. |
| "Took weeks to resolve" | J4.03 | Resolution speed |
| "They owned their mistake" | R3.01 | Admitting fault |
| "Sincere apology" | R3.02 | Expression of regret |
| "Offered nothing" | R3.03 | Compensation failure |
| "Changed their process" | J4.05 | Prevention action |
| "Blamed me instead" | R3.05 | Ownership failure |
---
## 8. Span Boundary Rules
### When to SPLIT
1. **At contrasting conjunctions**: but, however, although, despite, yet
2. **When subject changes**: location -> service -> food
3. **When valence changes**: positive -> negative
4. **When domain changes**: different aspect of experience
5. **When in different sentences** (unless tightly linked cause-effect)
### When to KEEP TOGETHER
1. **Cause-effect in same clause**: "Long wait because understaffed"
- Primary = impact (J1.01 Wait Time)
- Secondary = cause (A1.04 Staffing)
2. **Same target, mixed assessment**: "Good but overpriced" (same product)
3. **Elaboration of same point**: "Service was slow. Had to wait 40 minutes."
### Max Spans Guidance
| Sentence Type | Typical Spans |
|---------------|---------------|
| Simple sentence | 1-2 spans |
| Complex sentence | 2-3 spans |
| If tempted to create 4+ | Re-read; probably over-splitting |
### Cause-Effect Examples
| Span | Treatment | Primary | Secondary |
|------|-----------|---------|-----------|
| "Long wait because understaffed" | Single | J1.01 | A1.04 |
| "Food cold, had to send it back" | Single | O2.05 | J4.02 |
| "App crashed so I lost my data" | Single | E2.02 | E4.03 |
| "Rude waiter. Also the food was cold." | TWO spans | P1.02 / O2.05 | -- |
| "Support was slow because the system kept crashing" | Single | J1.03 | J3.03 |
### V+- vs Split Decision
| Feedback | Action |
|----------|--------|
| "Great product, terrible onboarding" | SPLIT: two different targets |
| "Good but overpriced" | KEEP: same target, use V+- |
| "Nice staff despite the chaos" | SPLIT if chaos is separate feedback |
**Test**: If the positive and negative parts have *different domain targets*, split them.
---
## 9. Worked Examples
### Example 1: Restaurant Review (Multiple Spans)
> "The pasta was perfectly al dente and beautifully plated, but our waiter disappeared for 20 minutes and we had to flag down someone else to get the check. Good value for a nice dinner out."
| Span | Primary | Secondary | V | I | S | A | T | E | CR |
|------|---------|-----------|---|---|---|---|---|---|-----|
| "pasta was perfectly al dente" | O1.02 | -- | V+ | I2 | S2 | A2 | TC | ES | CR-N |
| "beautifully plated" | O2.03 | -- | V+ | I2 | S2 | A2 | TC | ES | CR-N |
| "waiter disappeared for 20 minutes" | P3.03 | -- | V- | I2 | S3 | A3 | TC | ES | CR-N |
| "flag down someone else" | P3.01 | -- | V- | I2 | S2 | A2 | TC | ES | CR-N |
| "Good value for nice dinner" | V4.01 | -- | V+ | I1 | S1 | A1 | TC | ES | CR-N |
### Example 2: Software with Improvement Signal (CR-B)
> "Implementation took twice as long as promised, but once we went live, the system has been rock solid. Much more stable than the last version."
| Span | Primary | Secondary | V | I | CR | Note |
|------|---------|-----------|---|---|-----|------|
| "Implementation took twice as long" | J1.02 | R1.02 | V- | I3 | CR-N | No self-comparison |
| "system has been rock solid" | O1.04 | -- | V+ | I3 | CR-N | Current state |
| "Much more stable than last version" | O1.04 | -- | V+ | I2 | **CR-B** | Explicit improvement |
### Example 3: Persistent Problem (CR-S)
> "Still waiting forever for appointments. Third time I've complained about this and nothing has changed."
| Span | Primary | Secondary | V | I | CR | Note |
|------|---------|-----------|---|---|-----|------|
| "Still waiting forever" | A1.02 | J1.01 | V- | I3 | **CR-S** | "Still" = unchanged |
| "Third time I've complained" | J4.05 | R3.04 | V- | I2 | **CR-S** | Pattern, unchanged |
**Causal Chain** (Full profile only):
- MG-O (Oversight failure) -- Evidence: EI
- Justification: "Third time" + "nothing changed" logically entails oversight failure
### Example 4: Quality Decline (CR-W)
> "This used to be my favorite restaurant but the quality has really gone downhill since they changed chefs."
| Span | Primary | Secondary | V | I | CR | Note |
|------|---------|-----------|---|---|-----|------|
| "used to be my favorite" | R2.02 | -- | V- | I2 | **CR-W** | Past-positive implies decline |
| "quality has really gone downhill" | O2.02 | -- | V- | I3 | **CR-W** | Explicit decline |
### Example 5: Service Recovery (CR-B)
> "My package arrived damaged, which was frustrating. But they immediately sent a replacement and it arrived perfect. Great recovery!"
| Span | Primary | Secondary | V | I | CR |
|------|---------|-----------|---|---|-----|
| "package arrived damaged" | O2.05 | -- | V- | I2 | CR-N |
| "immediately sent replacement" | R3.03 | J4.03 | V+ | I2 | CR-N |
| "arrived perfect" | O2.05 | -- | V+ | I2 | **CR-B** |
| "Great recovery" | R3 | -- | V+ | I2 | CR-N |
### Example 6: Scam vs Rip-off (V vs R)
> "Total scam company. They charged me twice and refused to refund. This is robbery."
| Span | Primary | Secondary | V | I | CR |
|------|---------|-----------|---|---|-----|
| "Total scam company" | R1.04 | -- | V- | I3 | CR-N |
| "charged me twice" | J3.02 | -- | V- | I2 | CR-N |
| "refused to refund" | R3.03 | V2.04 | V- | I2 | CR-N |
| "This is robbery" | V4.01 | -- | V- | I3 | CR-N |
### Example 7: Accessibility Issue
> "No wheelchair ramp and the staff didn't seem to care."
| Span | Primary | Secondary | V | I | S | A | T | E | CR |
|------|---------|-----------|---|---|---|---|---|---|-----|
| "No wheelchair ramp" | A2.01 | -- | V- | I2 | S3 | A3 | TC | ES | CR-N |
| "staff didn't seem to care" | P1.03 | -- | V- | I2 | S1 | A2 | TC | EI | CR-N |
### Example 8: Digital Environment
> "App is beautiful but crashes constantly. Finally works on mobile though!"
| Span | Primary | Secondary | V | I | CR |
|------|---------|-----------|---|---|-----|
| "App is beautiful" | E2.01 | -- | V+ | I2 | CR-N |
| "crashes constantly" | E2.02 | -- | V- | I3 | CR-N |
| "Finally works on mobile" | E2.05 | -- | V+ | I2 | **CR-B** |
### Example 9: Cause-Effect (Single Span)
> "Support was painfully slow because their system kept going down."
| Span | Primary | Secondary | V | I | Note |
|------|---------|-----------|---|---|------|
| Full span | J1.03 | J3.03 | V- | I3 | Impact (slow) + cause (system down) |
### Example 10: Multi-Domain Complex Review
> "Great location and the room was clean, but checkout took forever and they tried to charge me for a minibar I never touched. Classic hotel scam."
| Span | Primary | Secondary | V | I | CR | Note |
|------|---------|-----------|---|---|-----|------|
| "Great location" | A4.01 | -- | V+ | I2 | CR-N | Convenience |
| "room was clean" | E1.01 | -- | V+ | I2 | CR-N | Physical space |
| "checkout took forever" | J1.01 | -- | V- | I3 | CR-N | Wait time |
| "tried to charge me for minibar I never touched" | V1.04 | J3.02 | V- | I2 | CR-N | Hidden cost + accuracy |
| "Classic hotel scam" | R1.04 | -- | V- | I3 | CR-N | Character judgment |
---
## 10. USN Quick Reference
URT String Notation for compact tagging.
### Format
```
URT:{Profile}:{codes}:{Valence}{Intensity}[:{metadata}][:{causal}]
```
### Profiles
| Code | Profile | Code Tier |
|------|---------|-----------|
| L | Lite | Domain (O, P, J...) |
| C | Core | Category (O1, P2...) |
| S | Standard | Subcode (O1.01, P2.03...) |
| F | Full | Subcode + Causal |
### Quick Examples
| Review | USN |
|--------|-----|
| "Food was cold" (Lite) | `URT:L:O:-2` |
| "Food was cold" (Core) | `URT:C:O2:-2` |
| "Food was cold, like last time" (Standard) | `URT:S:O2.05:-2:22TR.ES.S` |
| "Rude staff and unfair policy" (Standard) | `URT:S:P1.02+V2.04:-3:22TC.ES.N` |
| "Much better now!" (Standard) | `URT:S:V4.01:+2:11TC.ES.B` |
### Metadata Encoding (Standard+)
```
{Specificity}{Actionability}{Temporal}.{Evidence}.{Comparative}
Example: 22TC.ES.N
^^ = S2, A2
^^ = TC (current)
^^ = ES (stated)
^ = CR-N (no comparison)
```
### Valence Symbols
| Symbol | Meaning |
|--------|---------|
| + | Positive |
| - | Negative |
| 0 | Neutral |
| +- | Mixed |
---
## Quick Checklist Before Submitting
1. [ ] **One primary code per span** (no exceptions)
2. [ ] **Max 2 secondary codes** (prevents over-tagging)
3. [ ] **Checked "Don't Confuse With"** in spec for ambiguous codes
4. [ ] **Used decision tree** for domain selection
5. [ ] **Split at domain changes** (different aspects = different spans)
6. [ ] **CR-N unless explicit comparison** (default is no comparison)
7. [ ] **No invented causes** (causal chain only when text supports)
8. [ ] **Verified intensity markers** (I3 requires emphatic language)
9. [ ] **Cause-effect kept together** (impact = primary, cause = secondary)
10. [ ] **V vs R correctly applied** (exchange = V, character = R)
---
## 11. Billing/Returns/Refunds Decision Table
Special guidance for common billing and refund scenarios.
| Feedback Type | Primary Code | Secondary | Rationale |
|---------------|--------------|-----------|-----------|
| "Returns process was easy" | J4.02 | -- | Resolution process ease |
| "Returns process was a nightmare" | J4.02 | -- | Resolution process friction |
| "Refund took too long" | J4.03 | -- | Resolution speed |
| "Refund policy is unfair" | V2.04 | -- | Terms fairness |
| "They refused to refund me" (trust framing) | R3.03 | V2.04 | Compensation failure + policy |
| "Wrong amount refunded" | J3.02 | V1.04 | Accuracy + hidden costs |
| "They honored the warranty" | R2.05 | -- | Guarantee honor |
| "Worthless guarantee" | R2.05 | -- | Guarantee failure |
| "Charged me twice" | J3.02 | -- | Process accuracy |
| "Hidden fees at checkout" | V1.04 | V2.02 | Hidden costs + fee disclosure |
---
## 12. Data Privacy/Security Decision Table
| Feedback Type | Primary Code | Secondary | Rationale |
|---------------|--------------|-----------|-----------|
| "Security incident occurred" | E4.03 | -- | Security breach |
| "I don't trust them with my data" | R1.03 or R1.04 | E4.03 | Trust + security |
| "Data breach notification" | E4.03 | R1.03 | Security + transparency |
| "Privacy policy concerns" | V2.04 | R1.04 | Terms + ethics |
| "Account was hacked" | E4.03 | -- | Security failure |
| "They sell my data" | R1.04 | V2.04 | Ethics + terms |
---
## 13. CR Trigger Word Reference
Quick lookup for Comparative Reference assignment.
### CR-B (Better) - Improvement Signals
| Trigger Phrase | Example |
|----------------|---------|
| "better now" | "Service is much better now" |
| "improved" | "Quality has improved" |
| "finally fixed" | "They finally fixed the app!" |
| "much faster/cleaner/friendlier than before" | "Much faster than my last visit" |
| "they've turned it around" | "They've really turned things around" |
| "glad they addressed it" | "Glad they addressed the issue" |
| "not like last time" (positive context) | "Not like last time - actually worked!" |
### CR-W (Worse) - Decline Signals
| Trigger Phrase | Example |
|----------------|---------|
| "worse now" | "Quality is worse now" |
| "used to be good" | "This used to be my favorite restaurant" |
| "has gone downhill" | "Really gone downhill lately" |
| "not what it used to be" | "Just not what it used to be" |
| "declining" | "Quality is declining" |
| "deteriorating" | "Service has been deteriorating" |
| "they've really fallen off" | "They've really fallen off" |
| "was great before" | "Was great before the renovation" |
| "no longer what it was" | "No longer the place I loved" |
### CR-S (Same) - Persistence Signals
| Trigger Phrase | Example |
|----------------|---------|
| "still" | "Service is still slow" |
| "as always" | "Great as always" |
| "same as before" | "Same problems as before" |
| "nothing has changed" | "Complained twice, nothing has changed" |
| "yet again" | "Wrong order yet again" |
| "once again" | "Once again disappointed" |
| "as usual" | "Late as usual" |
| "same old" | "Same old problems" |
### CR-N (None) - No Comparison
- Default for ~90% of spans
- First-time customer reviews
- Competitor comparisons ("better than Brand X")
- No temporal language present
---
## 14. Causal Chain Quick Guide (Full Profile Only)
Only assign causal codes when text **explicitly states** or **logically entails** the cause. When in doubt, omit.
### Conditions Layer (CD-)
| Code | Name | When to Use | Example Text |
|------|------|-------------|--------------|
| CD-S | Staff State | Fatigue, training, motivation stated | "Server seemed exhausted" |
| CD-T | Team Dynamics | Handoffs, coordination issues stated | "They kept passing me around" |
| CD-E | Equipment | Malfunction stated | "The machine was broken" |
| CD-F | Facility | Maintenance, capacity stated | "Place was falling apart" |
| CD-O | Operational | Understaffing, demand surge stated | "Clearly understaffed" |
### Management Layer (MG-)
| Code | Name | When to Use | Example Text |
|------|------|-------------|--------------|
| MG-P | Planning | Scheduling, forecasting failure stated | "No one scheduled for weekend" |
| MG-T | Training | Training gap stated | "Clearly not trained properly" |
| MG-O | Oversight | Supervision failure inferable | "Third time reporting this" (EI) |
| MG-R | Resources | Resource allocation stated | "No supplies available" |
| MG-C | Communication | Policy relay failure stated | "Staff didn't know the policy" |
### Systemic Layer (SY-)
| Code | Name | When to Use | Example Text |
|------|------|-------------|--------------|
| SY-R | Resource Decisions | Budget, investment stated | "Cheap owners" |
| SY-P | Policy/Procedure | Bureaucracy, rules stated | "Company policy is the problem" |
| SY-C | Culture | Values, priorities stated | "Management doesn't care" |
| SY-S | Standards | Quality thresholds stated | "They've lowered their standards" |
| SY-H | Human Capital | Compensation, hiring stated | "Can't keep good employees" |
| SY-X | External | Market, regulatory stated | "Ever since COVID" |
### Causal Chain Validity Test
| Review Text | Causal Code? | Valid? | Why |
|-------------|--------------|--------|-----|
| "Long wait because they were short-staffed" | CD-O | Yes | Explicitly stated |
| "The machine was broken" | CD-E | Yes | Explicitly stated |
| "Fifth time I've reported this" | MG-O | Yes | Logically entailed (EI) |
| "They seem understaffed" | -- | No | "Seem" = speculation |
| "Probably a training issue" | -- | No | Annotator speculation |
| "Management doesn't care" (stated by customer) | SY-C | Yes | Customer explicitly states |
---
## 15. Inter-Annotator Agreement Rules
### Mandatory Conventions
1. **One primary code per span** - no exceptions
2. **Max 2 secondary codes** - prevents over-tagging
3. **Max 3 spans per sentence** - if more, reconsider boundaries
4. **Prefer splitting over V+-** - unless truly inseparable
5. **Check "Don't Confuse With"** - before finalizing subcode
6. **Evidence type required** - must justify EI inferences
7. **No invented causes** - causal_chain only when text supports
8. **CR-N is default** - only use CR-B/W/S when comparison is explicit
### Tie-Break Rules
When unsure between two codes:
| Disagreement Type | Resolution |
|-------------------|------------|
| Different domains | Use decision tree; first "YES" wins |
| Same category, different subcodes | Prefer more specific subcode |
| Different intensity | Defer to linguistic markers (see I3 markers) |
| Different actionability | A3 requires specific implementable action |
| Different CR | CR-N unless explicit temporal language present |
### Calibration Check Questions
Before submitting, ask yourself:
1. Would another annotator choose the same primary domain?
2. Is the secondary code genuinely distinct (different category)?
3. Is intensity justified by linguistic markers?
4. Is CR assignment supported by explicit comparison language?
5. Would I assign causal codes if I didn't know the industry?
---
## Where to Find More
| Document | Purpose |
|----------|---------|
| **URT-Specification-v5.1.md** | Full specification with all 140 subcodes |
| **URT Issue Lifecycle Framework** | Analytics layer for resolution tracking |
| **URT QA Protocol** | Calibration and audit procedures |
---
## Appendix: Profile Requirements Summary
### URT-Lite (Micro-business, quick triage)
```
Required: primary_code (domain), valence
Optional: intensity
Forbidden: secondary_codes, subcodes, S/A/E/CR/causal
```
### URT-Core (Small business, dashboards)
```
Required: primary_code (category), valence, intensity
Optional: secondary_codes (max 2, categories only), temporal, comparative
Forbidden: subcodes, S/A/E, causal
```
### URT-Standard (Operations, routing, analytics)
```
Required: primary_code (subcode), valence, intensity, specificity,
actionability, temporal, evidence, comparative
Optional: secondary_codes (max 2, subcodes only), causal_chain, confidence
Forbidden: domains, categories as primary
```
### URT-Full (Enterprise, root cause, audit)
```
Required: All Standard fields + causal_chain (when evidence supports),
linked_spans, confidence, annotator_notes
Optional: None - all fields should be completed
Forbidden: domains, categories as primary
```
---
*URT v5.1 Annotator Quickstart Guide | Track A: Training Materials*

476
urt-taxonomy/track-a-training/A2-QA-Protocol.md Normal file
View File

@@ -0,0 +1,476 @@
# A2: QA Protocol
## Universal Review Taxonomy (URT) v5.1
**Purpose**: Define quality assurance processes for URT annotation
**Version**: 5.1 | **Status**: Production Ready | **Date**: 2026-01-23
---
## Table of Contents
1. [Inter-Annotator Agreement (IAA) Metrics](#1-inter-annotator-agreement-iaa-metrics)
2. [Calibration Sessions](#2-calibration-sessions)
3. [Error Categories and Severity](#3-error-categories-and-severity)
4. [Audit Procedures](#4-audit-procedures)
5. [Gold Standard Management](#5-gold-standard-management)
6. [Quality Gates](#6-quality-gates)
7. [Feedback Loop](#7-feedback-loop)
8. [Metrics Dashboard](#8-metrics-dashboard)
---
## 1. Inter-Annotator Agreement (IAA) Metrics
### 1.1 Cohen's Kappa Thresholds by Code Tier
| Code Tier | Minimum Kappa | Target Kappa |
|-----------|---------------|--------------|
| **Domain (Tier 1)** | 0.80 | 0.90+ |
| **Category (Tier 2)** | 0.75 | 0.85+ |
| **Subcode (Tier 3)** | 0.70 | 0.80+ |
| **Valence** | 0.85 | 0.92+ |
| **Intensity** | 0.70 | 0.80+ |
| **Comparative Reference** | 0.75 | 0.85+ |
### 1.2 Krippendorff's Alpha (3+ Annotators)
| Scenario | Minimum Alpha | Target Alpha |
|----------|---------------|--------------|
| Initial Training | 0.67 | 0.75+ |
| Production Quality | 0.75 | 0.85+ |
| Gold Standard Creation | 0.85 | 0.90+ |
### 1.3 Agreement Interpretation Scale
| Range | Interpretation | Action |
|-------|----------------|--------|
| **0.90 - 1.00** | Almost Perfect | Maintain standards |
| **0.80 - 0.89** | Excellent | Minor calibration |
| **0.70 - 0.79** | Good | Schedule calibration |
| **0.60 - 0.69** | Moderate | Mandatory retraining |
| **< 0.60** | Poor | Suspend, reassess |
### 1.4 Profile-Specific Requirements
| Profile | Domain | Category | Subcode | Overall Target |
|---------|--------|----------|---------|----------------|
| URT-Lite | >= 0.85 | N/A | N/A | >= 0.85 |
| URT-Core | >= 0.85 | >= 0.80 | N/A | >= 0.80 |
| URT-Standard/Full | >= 0.85 | >= 0.80 | >= 0.75 | >= 0.78 |
---
## 2. Calibration Sessions
### 2.1 Session Frequency
| Team Size | Daily | Weekly | Monthly |
|-----------|-------|--------|---------|
| 1-3 annotators | -- | 30min | 2hr |
| 4-10 annotators | 15min | 1hr | 3hr |
| 11+ annotators | 15min | 2x 1hr | 4hr |
### 2.2 Weekly Session Structure (60 min)
| Time | Activity |
|------|----------|
| 0-5 min | Review IAA metrics from past week |
| 5-15 min | Discuss 3 highest-disagreement spans |
| 15-35 min | Group annotation exercise (5 spans) |
| 35-50 min | Compare results, discuss differences |
| 50-60 min | Document decisions, update guidance |
### 2.3 Materials Checklist
- [ ] 5-10 pre-selected review spans
- [ ] IAA metrics report
- [ ] Top disagreement patterns list
- [ ] Gold standard examples
- [ ] A1 Quickstart Guide
- [ ] Session notes template
### 2.4 Outcome Documentation
```
## Calibration Session: [DATE]
### Disagreement Patterns
1. Pattern: [Description]
Resolution: [Decision]
### Exercise Results
| Span | Consensus | Votes | Notes |
|------|-----------|-------|-------|
### Action Items
- [ ] Update A1 Section X
- [ ] Add gold standard example
```
---
## 3. Error Categories and Severity
### 3.1 Error Severity Matrix
| Severity | Weight | Description |
|----------|--------|-------------|
| **Critical** | 1.0 | Fundamentally wrong |
| **Major** | 0.5 | Significant deviation |
| **Minor** | 0.25 | Suboptimal but defensible |
| **Slip** | 0.1 | Typo/formatting |
### 3.2 Critical Errors (Weight: 1.0)
| Error Type | Example |
|------------|---------|
| Wrong Domain | "Rude waiter" coded as O instead of P |
| Wrong Valence | Complaint coded as V+ |
| Valence Omission | No valence assigned |
| Profile Violation | Subcode in URT-Lite |
### 3.3 Major Errors (Weight: 0.5)
| Error Type | Example |
|------------|---------|
| Wrong Category | J1 (Timing) vs J4 (Resolution) |
| Intensity Off by 2 | "TERRIBLE!!!" coded as I1 |
| Wrong CR Direction | "Gone downhill" coded as CR-B |
| Missed/Over Split | Two issues merged OR single split |
| J4/R3 Confusion | Process vs Ownership |
| V/R Confusion | "Total scam" coded as V4.01 |
### 3.4 Minor Errors (Weight: 0.25)
| Error Type | Example |
|------------|---------|
| Wrong Subcode (Same Category) | P1.01 vs P1.02 within P1 |
| Intensity Off by 1 | "pretty good" as I1 vs I2 |
| Borderline Secondary | Questionable secondary code |
### 3.5 Slips (Weight: 0.1)
Typos, formatting errors, boundary off by <5 chars
### 3.6 Error Severity Decision Tree
```
Is DOMAIN wrong? --> YES: CRITICAL
Is VALENCE wrong/missing? --> YES: CRITICAL
Is CATEGORY wrong? --> YES: MAJOR
Is INTENSITY off by 2? --> YES: MAJOR
Is SUBCODE wrong? --> YES: MINOR
Is it formatting/typo? --> YES: SLIP
```
### 3.7 Accuracy Calculation
```
Error Score = Sum(error_weight * count) / total_spans
Accuracy = 100% - Error Score
Thresholds:
> 95% = Excellent 85-90% = Acceptable
90-95% = Good < 85% = Below Standard
```
---
## 4. Audit Procedures
### 4.1 Sampling Methodology
**Random**: Equal probability selection for general monitoring
**Stratified**: Ensure representation across domains, annotators, edge cases
| Stratum | Minimum Sample |
|---------|----------------|
| Each Domain (O-R) | 5% of total |
| Each Annotator | 10% of output |
| High-Intensity (I3) | 15% of I3 spans |
| Non-default CR | 25% of CR-B/W/S |
### 4.2 Sample Size by Volume
| Daily Volume | Audit Rate |
|--------------|------------|
| < 100 spans | 30% |
| 100-500 | 20% |
| 500-2000 | 15% |
| 2000-10000 | 10% |
| > 10000 | 7% |
### 4.3 Audit Frequency
| Type | Frequency | Owner |
|------|-----------|-------|
| Spot Check | Daily | QA Lead |
| Sample Audit | Weekly | QA Team |
| Full Audit | Monthly | Senior QA |
| External | Quarterly | External |
### 4.4 Audit Workflow
```
[Daily Output] --> [Sample] --> [Blind Re-code]
|
v
[Compare] --> Match: [Log]
|
+--> Mismatch: [Classify Error] --> [Route] --> [Aggregate]
```
### 4.5 Escalation Paths
| Error Score | Level | Action |
|-------------|-------|--------|
| < 10% | 1 | Self-correction |
| 10-15% | 2 | QA Lead review |
| 15-20% | 3 | Team calibration |
| > 20% | 4 | Management escalation |
---
## 5. Gold Standard Management
### 5.1 Corpus Requirements
| Metric | Minimum | Target |
|--------|---------|--------|
| Total Spans | 500 | 1000+ |
| Per Domain | 50 | 100+ |
| Per Category | 10 | 25+ |
| Edge Cases | 100 | 200+ |
### 5.2 Creation Process
```
[Candidate] --> [3+ Annotators Classify] --> [Calculate Alpha]
|
Alpha >= 0.85: [Add to Gold]
Alpha < 0.85: [Discuss/Reject]
```
### 5.3 Gold Standard Documentation
```json
{
"gold_id": "GS-2026-001",
"span_text": "The waiter was incredibly rude",
"classification": {
"primary_code": "P1.02",
"valence": "V-",
"intensity": "I3"
},
"rationale": "Clear disrespect. 'Incredibly' indicates I3.",
"common_mistakes": ["P1.01 (Warmth)"],
"agreement_score": 0.92,
"version": "5.1",
"status": "active"
}
```
### 5.4 Version Control
| Change Type | Version Bump |
|-------------|--------------|
| Add example | Patch (5.1.1) |
| Fix error | Patch |
| Spec alignment | Minor (5.2) |
| Taxonomy change | Major (6.0) |
### 5.5 Retirement Criteria
- Spec change invalidates example
- Systematic confusion traced to example
- Industry shifts make obsolete
---
## 6. Quality Gates
### 6.1 New Annotator Qualification
```
Week 1: Training (A1 Guide, Spec, Videos)
Week 2: Supervised Practice (100 spans, daily feedback)
Week 3: Qualification Exam (50 gold spans, blind)
|
Pass (>= 85%): Production + 30-day probation
Fail: Remediation + retake
```
**Passing Criteria**:
- Overall Accuracy >= 85%
- Domain Accuracy >= 90%
- Critical Errors = 0
- Major Errors <= 3
### 6.2 Production Annotator Requirements
| Requirement | Frequency | Threshold |
|-------------|-----------|-----------|
| Accuracy Check | Weekly | >= 90% |
| Calibration | Weekly | 90% attendance |
| Gold Quiz | Monthly | >= 85% |
| IAA with Peers | Bi-weekly | Kappa >= 0.75 |
### 6.3 Annotator Tiers
| Tier | Accuracy | Audit Rate |
|------|----------|------------|
| Expert | >= 95% | 5% |
| Senior | 92-95% | 10% |
| Standard | 88-92% | 15% |
| Developing | 85-88% | 25% |
| Probation | < 85% | 50% |
### 6.4 Automated System Thresholds
| Metric | Minimum | Production | Best |
|--------|---------|------------|------|
| Domain Accuracy | 85% | 90% | 95% |
| Category Accuracy | 80% | 85% | 90% |
| Subcode Accuracy | 75% | 80% | 85% |
| Valence F1 | 0.88 | 0.92 | 0.96 |
### 6.5 Release Criteria
- [ ] All accuracy metrics meet thresholds
- [ ] Gold standard test documented
- [ ] Error analysis completed
- [ ] Rollback plan in place
- [ ] Stakeholder sign-off
---
## 7. Feedback Loop
### 7.1 Error Reporting
```
ERROR REPORT FIELDS:
- Reporter, Date, Span ID
- Type: [Spec Unclear | Gold Issue | Edge Case | Tool Bug]
- Description
- Suggested Resolution
- Urgency: [Critical | High | Medium | Low]
```
### 7.2 Triage Process
```
[Error Submitted] --> [QA Lead (24hr)]
|
+--> Spec Issue --> PM
+--> Gold Issue --> QA Team
+--> Tool Bug --> Engineering
+--> Training Gap --> QA Lead
```
### 7.3 Spec Clarification Process
```
[Ambiguity] --> Check A1 Guide --> Found: Apply
|
Not Found: Submit Request
|
PM + QA Review
|
Accept: Update A1/Spec
Reject: Document Rationale
```
### 7.4 Training Update Triggers
| Trigger | Action | Timeline |
|---------|--------|----------|
| IAA < 0.75 | Mandatory calibration | 48 hours |
| New error pattern (3+) | Targeted training | 1 week |
| Spec release | Full training | 2 weeks |
| Annotator < 85% | Individual coaching | Immediate |
### 7.5 Response SLAs
| Urgency | Response | Resolution |
|---------|----------|------------|
| Critical | 2 hours | 24 hours |
| High | 24 hours | 1 week |
| Medium | 48 hours | 2 weeks |
| Low | 1 week | Next sprint |
---
## 8. Metrics Dashboard
### 8.1 Key QA KPIs
| KPI | Target | Alert |
|-----|--------|-------|
| Overall Accuracy | >= 92% | < 88% |
| IAA (Kappa) | >= 0.80 | < 0.75 |
| Critical Error Rate | < 2/1K | >= 5/1K |
| Audit Coverage | >= 10% | < 7% |
| Calibration Attendance | >= 90% | < 80% |
| Error Resolution Time | < 5 days | > 10 days |
### 8.2 Reporting Frequency
| Report | Frequency | Audience |
|--------|-----------|----------|
| Daily Snapshot | Daily | QA Lead |
| Weekly Summary | Weekly | Team + Management |
| Monthly Deep Dive | Monthly | Leadership |
| Quarterly Review | Quarterly | Executives |
### 8.3 Alert Configuration
```
Green Yellow Red
Accuracy >= 92% 88-92% < 88%
IAA >= 0.80 0.75-0.80 < 0.75
Critical Err < 2/1K 2-5/1K > 5/1K
Coverage >= 12% 10-12% < 10%
Yellow: Notify QA Lead
Red: Escalate + Immediate Action
```
### 8.4 Dashboard Panels
1. **Accuracy Trend**: Line chart, 30-day rolling
2. **IAA Heatmap**: Annotator pairwise Kappa
3. **Error Distribution**: Stacked bar by severity
4. **Domain Performance**: Radar chart (O-P-J-E-A-V-R)
5. **Annotator Leaderboard**: Table with tiers
6. **Alert Status**: Traffic light indicators
### 8.5 Metric Formulas
```
Accuracy = 1 - (Sum(error_weight * count) / total_spans)
Cohen's Kappa = (Po - Pe) / (1 - Pe)
Po = Observed agreement
Pe = Expected agreement by chance
Krippendorff's Alpha = 1 - (Do / De)
Do = Observed disagreement
De = Expected disagreement
F1 = 2 * (Precision * Recall) / (Precision + Recall)
```
---
## Document References
| Document | Location |
|----------|----------|
| URT-Specification-v5.1.md | `/urt-taxonomy/spec/` |
| A1-Annotator-Quickstart.md | `/urt-taxonomy/track-a-training/` |
| Gold Standard Corpus | `/urt-taxonomy/gold-standard/` |
---
*URT v5.1 QA Protocol | Track A: Training Materials*

1925
urt-taxonomy/track-a-training/A3-Calibration-Test-Set.md Normal file
View File

File diff suppressed because it is too large Load Diff

4578
urt-taxonomy/track-a-training/A4-Full-Annotation-Manual.md Normal file
View File

File diff suppressed because it is too large Load Diff

1766
urt-taxonomy/track-b-engineering/B1-urt-codes.yaml Normal file
View File

File diff suppressed because it is too large Load Diff

1126
urt-taxonomy/track-b-engineering/B2-database-schema.sql Normal file
View File

File diff suppressed because it is too large Load Diff

627
urt-taxonomy/track-b-engineering/B3-owner-routing.md Normal file
View File

@@ -0,0 +1,627 @@
# B3: Owner Routing Matrix Documentation
## Universal Review Taxonomy v5.1
**Status**: Production Ready
**Release Date**: 2026-01-23
**Prerequisite**: B1-urt-codes.yaml (Code Registry)
**Configuration File**: B3-owner-routing.yaml
---
## Table of Contents
1. [Overview](#1-overview)
2. [Routing Resolution Logic](#2-routing-resolution-logic)
3. [SLA Definitions](#3-sla-definitions)
4. [Escalation Paths](#4-escalation-paths)
5. [On-Call Rotation](#5-on-call-rotation)
6. [Ticketing System Integration](#6-ticketing-system-integration)
7. [Configuration Examples](#7-configuration-examples)
8. [Multi-Location Override Patterns](#8-multi-location-override-patterns)
9. [Special Routing Rules](#9-special-routing-rules)
10. [Appendix: Domain-Team Mapping](#appendix-domain-team-mapping)
---
## 1. Overview
The Owner Routing Matrix maps URT codes to responsible teams and individuals. When feedback is classified using URT codes, the routing matrix determines:
- **Who owns the ticket** (primary team)
- **Who should be notified** (co-owners)
- **When to escalate** (SLA and intensity rules)
- **How quickly to respond** (priority-based SLA)
### Key Principles
1. **Specificity wins**: More specific routes override less specific ones
2. **Safety first**: E4 (Safety) issues always take priority
3. **Intensity drives urgency**: I3 (Strong) triggers auto-escalation
4. **Primary code determines owner**: Secondary codes add co-owners
---
## 2. Routing Resolution Logic
### Resolution Order
Routes are resolved in the following order (most specific first):
```
subcode_overrides > category_overrides > domain_defaults
(O1.01) (O1) (O)
```
### Resolution Algorithm
```
FUNCTION resolve_route(primary_code, secondary_codes, intensity, valence):
# Step 1: Check subcode-level override
IF primary_code IN subcode_overrides:
route = subcode_overrides[primary_code]
# Step 2: Check category-level override
ELSE IF category(primary_code) IN category_overrides:
route = category_overrides[category(primary_code)]
# Step 3: Fall back to domain default
ELSE:
route = domain_defaults[domain(primary_code)]
# Step 4: Apply intensity modifiers
IF intensity == "I3":
route.auto_escalate = TRUE
route.sla_hours = MIN(route.sla_hours, 4)
# Step 5: Add co-owners from secondary codes
FOR code IN secondary_codes:
co_owner = resolve_route(code, [], intensity, valence)
route.co_owners.append(co_owner.team)
# Step 6: Check cross-functional patterns
all_codes = [primary_code] + secondary_codes
FOR pattern IN cross_functional_patterns:
IF pattern.triggered_by(all_codes):
route = merge_routes(route, pattern.route)
RETURN route
```
### Example Resolutions
| Primary Code | Category | Domain | Resolution Path | Final Team |
|--------------|----------|--------|-----------------|------------|
| E2.02 | E2 | E | subcode_overrides["E2.02"] | engineering |
| O2.03 | O2 | O | category_overrides["O2"] | quality_control |
| R4.05 | R4 | R | category_overrides["R4"] | marketing |
| O3.02 | O3 | O | category_overrides["O3"] | fulfillment |
---
## 3. SLA Definitions
### Priority Levels
| Priority | Initial Response | Resolution Target | Update Frequency |
|----------|------------------|-------------------|------------------|
| **Critical** | 1 hour | 4 hours | Every hour |
| **High** | 4 hours | 24 hours | Every 4 hours |
| **Normal** | 24 hours | 72 hours | Daily |
| **Low** | 48 hours | 1 week | Every 2 days |
### Intensity-to-Priority Mapping
| Intensity | Valence | Priority | Auto-Escalate |
|-----------|---------|----------|---------------|
| I3 | V- | Critical | Yes |
| I3 | V+ | High | No |
| I2 | V- | High | No |
| I2 | V+ | Normal | No |
| I1 | V- | Normal | No |
| I1 | V+ | Low | No |
### SLA Override Rules
Certain subcodes have hardcoded SLA overrides that supersede intensity-based SLAs:
| Subcode | Override SLA | Reason |
|---------|--------------|--------|
| E4.01 | 1 hour | Physical safety |
| E4.02 | 2 hours | Health/hygiene |
| E4.03 | 1 hour | Security breach |
| E4.05 | 1 hour | Emergency readiness |
| A3.05 | 4 hours | Discrimination |
| R1.04 | 4 hours | Ethics violation |
---
## 4. Escalation Paths
### Escalation Hierarchy Diagram
```
+--------+
| CEO |
+--------+
|
+---------------------+---------------------+
| | |
+-------+ +-------+ +-------+
| CFO | | COO | | CTO |
+-------+ +-------+ +-------+
| | |
+---------------+ +---------------+ +---------------+
|Finance Director| |Ops Director | |Engineering Lead|
+---------------+ +---------------+ +---------------+
| | |
+---------------+ +---------------+ +---------------+
| Finance Mgr | | Ops Manager | | Tech Lead |
+---------------+ +---------------+ +---------------+
| | |
+---------------+ +---------------+ +---------------+
| Finance Team | | Ops Team | | Eng Team |
+---------------+ +---------------+ +---------------+
```
### Domain-Specific Escalation Paths
```
O (Offering):
Team -> Product Mgr -> Product Director -> VP Product -> COO -> CEO
P (People):
Team -> HR Manager -> HR Director -> CHRO -> CEO
J (Journey):
Team -> Ops Manager -> Ops Director -> COO -> CEO
E (Environment):
Physical: Team -> Facilities Mgr -> Facilities Dir -> COO -> CEO
Digital: Team -> Tech Lead -> Engineering Lead -> CTO -> CEO
A (Access):
Team -> Compliance Mgr -> Compliance Dir -> General Counsel -> CEO
V (Value):
Team -> Finance Mgr -> Finance Dir -> CFO -> CEO
R (Relationship):
Team -> CX Manager -> CX Director -> CMO -> CEO
(Integrity issues go directly to Leadership -> CEO)
```
### Auto-Escalation Triggers
| Trigger | Condition | Escalation Action |
|---------|-----------|-------------------|
| SLA Breach | >100% of SLA elapsed | Escalate +1 level |
| Repeated Issue | Same code 3x in 30 days | Escalate +1 level |
| VIP Customer | Flagged account | Notify director immediately |
| Safety Issue | E4.* with I2+ | Auto-escalate to safety officer |
| Ethics Issue | R1.04 | Auto-escalate to CEO |
---
## 5. On-Call Rotation
### Coverage Requirements
| Team | Coverage | Hours | Handles |
|------|----------|-------|---------|
| Support | 24/7 | All hours | I3, E4.*, security |
| Operations | Extended | 06:00-22:00 | J4, J1, high-priority ops |
| Engineering | On-call | After hours | E2.* critical, outages |
### On-Call Responsibilities
**Primary On-Call (Support)**:
- Monitor all incoming I3 (critical) tickets
- Triage E4 (safety) issues immediately
- Escalate security incidents (E4.03)
- Acknowledge all critical tickets within 15 minutes
**Secondary On-Call (Operations)**:
- Handle J4 (resolution) escalations
- Address J1 (timing) issues affecting active customers
- Coordinate with primary on-call for cross-functional issues
### Holiday Coverage
- Minimum: Support team on-call
- SLA multiplier: 1.5x (extended response times)
- Escalation: Only for critical/safety issues
- Other tickets: Queued for next business day
---
## 6. Ticketing System Integration
### Webhook Payload Structure
When integrating with ticketing systems, the routing matrix produces:
```json
{
"ticket": {
"urt_code": "P1.02",
"intensity": "I3",
"valence": "V-",
"routing": {
"primary_team": "hr",
"primary_email": "hr@example.com",
"escalation_contact": "hr_director",
"co_owners": ["training"],
"priority": "critical",
"sla_hours": 4,
"auto_escalate": true
},
"tags": ["people", "attitude", "respect", "critical"],
"metadata": {
"resolution_path": "subcode_override",
"pattern_matched": null
}
}
}
```
### Supported Integrations
| System | Integration Method | Notes |
|--------|-------------------|-------|
| Zendesk | Webhook + Triggers | Map teams to groups |
| Jira Service Management | REST API | Map to request types |
| Freshdesk | Webhook | Map teams to agent groups |
| Intercom | API | Map to team inboxes |
| Custom | Generic webhook | Use payload above |
### Field Mapping
| URT Field | Zendesk | Jira | Freshdesk |
|-----------|---------|------|-----------|
| primary_team | Group | Project | Group |
| priority | Priority | Priority | Priority |
| urt_code | Custom Field | Label | Tag |
| sla_hours | SLA Policy | SLA | SLA Policy |
| co_owners | CCs | Watchers | CCs |
---
## 7. Configuration Examples
### Restaurant Configuration
```yaml
# Restaurant-specific overrides
category_overrides:
O1:
team: "kitchen"
escalation: "head_chef"
notes: "Food function issues go to kitchen"
O2:
team: "kitchen"
escalation: "head_chef"
notes: "Food quality is kitchen responsibility"
E1:
team: "front_of_house"
escalation: "foh_manager"
notes: "Physical space is FOH concern"
P1:
team: "front_of_house"
escalation: "foh_manager"
notes: "Server attitude is FOH responsibility"
subcode_overrides:
J1.01: # Wait time
team: "host_station"
escalation: "foh_manager"
J1.05: # Pacing
team: "kitchen"
escalation: "head_chef"
notes: "Pacing often kitchen-driven"
```
### SaaS Company Configuration
```yaml
# SaaS-specific overrides
category_overrides:
O1:
team: "engineering"
escalation: "engineering_lead"
notes: "Product function is eng responsibility"
E2:
team: "product"
escalation: "product_director"
notes: "Digital experience is product"
J3:
team: "devops"
escalation: "sre_lead"
notes: "Reliability is SRE concern"
subcode_overrides:
E2.03: # Performance
team: "sre"
escalation: "sre_lead"
auto_escalate: true
notes: "Performance issues often infra"
J3.03: # System availability
team: "sre"
escalation: "sre_lead"
auto_escalate: true
sla_override_hours: 1
```
### Healthcare Provider Configuration
```yaml
# Healthcare-specific overrides
category_overrides:
O1:
team: "clinical"
escalation: "chief_medical_officer"
notes: "Treatment outcomes are clinical"
P2:
team: "clinical"
escalation: "chief_medical_officer"
notes: "Clinical competence"
E4:
team: "compliance"
escalation: "compliance_director"
auto_escalate: true
notes: "All safety issues are compliance"
subcode_overrides:
A3.03: # Dietary/Medical accommodations
team: "clinical"
escalation: "chief_nursing_officer"
notes: "Medical accommodations are clinical"
E4.02: # Health/Hygiene
team: "infection_control"
escalation: "chief_medical_officer"
auto_escalate: true
sla_override_hours: 1
```
### Retail Store Configuration
```yaml
# Retail-specific overrides
category_overrides:
O1:
team: "merchandising"
escalation: "merchandising_director"
O3:
team: "inventory"
escalation: "inventory_manager"
A1:
team: "store_ops"
escalation: "store_director"
P1:
team: "store_ops"
escalation: "store_manager"
subcode_overrides:
A1.03: # Inventory
team: "inventory"
escalation: "inventory_manager"
J1.01: # Wait time (checkout)
team: "store_ops"
escalation: "store_manager"
```
---
## 8. Multi-Location Override Patterns
### Location-Specific Routing
For businesses with multiple locations, use location prefixes:
```yaml
# Location-specific team definitions
teams:
store_001_ops:
name: "Store 001 Operations"
email: "store001-ops@example.com"
manager: "store_001_manager"
location: "store_001"
store_002_ops:
name: "Store 002 Operations"
email: "store002-ops@example.com"
manager: "store_002_manager"
location: "store_002"
# Location override rules
location_routing:
enabled: true
# Default: route to location team if exists
resolution_order:
1: "location_team" # e.g., store_001_ops
2: "regional_team" # e.g., west_region_ops
3: "central_team" # e.g., operations
# Escalation: location -> regional -> central
escalation_path:
location: "regional"
regional: "central"
central: "executive"
```
### Regional Aggregation
```yaml
regional_routing:
west_region:
locations: ["store_001", "store_002", "store_003"]
regional_team: "west_ops"
regional_director: "west_director"
east_region:
locations: ["store_004", "store_005", "store_006"]
regional_team: "east_ops"
regional_director: "east_director"
# Pattern: systemic issues across multiple locations escalate regionally
multi_location_patterns:
same_issue_multiple_locations:
threshold: 3
window_days: 7
action: "escalate_to_regional"
notes: "Same issue at 3+ locations triggers regional review"
```
### Franchise Model
```yaml
franchise_routing:
# Franchisee handles local issues
franchisee_owned:
- "P1" # Staff attitude
- "P2" # Staff competence
- "E1" # Physical space
- "J1" # Timing
# Franchisor handles brand/system issues
franchisor_owned:
- "O1" # Product function (brand standards)
- "O2" # Product quality (brand standards)
- "R1" # Integrity (brand reputation)
- "V2" # Transparency (policy)
# Shared responsibility
shared:
- "E4" # Safety (both)
- "A3" # Inclusivity (both)
```
---
## 9. Special Routing Rules
### Multi-Code Span Routing
When a span has multiple codes:
```
Primary: P1.02 (Respect) - I3
Secondary: V2.04 (Terms Fairness)
Resolution:
1. Primary team: HR (from P1.02)
2. Co-owner: Legal (from V2.04)
3. Priority: Critical (from I3)
4. Escalation: HR Director + General Counsel notified
```
**Priority Domain Order** (for conflict resolution):
1. E4 (Safety) - always first
2. R1 (Integrity/Ethics)
3. A3 (Discrimination)
4. P (People)
5. V (Value)
6. J (Journey)
7. O (Offering)
8. E (non-safety)
9. A (non-discrimination)
10. R (non-integrity)
### Valence Differentiation
| Valence | Routing Behavior |
|---------|------------------|
| V+ | Route to recognition workflow, lower priority |
| V- | Standard routing, priority by intensity |
| V0 | Log only, no ticket created |
| V+- | Route as V-, note positive elements |
**V+ Special Handling**:
- Positive feedback about staff (P domain, V+) routes to HR for recognition
- Positive feedback about product (O domain, V+) routes to marketing for testimonial potential
- SLA: 48 hours (acknowledgment, not resolution)
### Causal Code Routing
Causal codes indicate root cause and route to strategic teams:
| Layer | Codes | Typical Route | Purpose |
|-------|-------|---------------|---------|
| Conditions | CD-* | Operational teams | Fix immediate cause |
| Management | MG-* | Management | Address management gap |
| Systemic | SY-* | Leadership | Strategic intervention |
**Example**:
```
Review: "Staff seemed exhausted and made mistakes"
Primary: J3.05 (Error Rate)
Causal: CD-S (Staff State) -> MG-P (Planning) -> SY-H (Human Capital)
Routing:
- Ticket: Operations (for immediate errors)
- Root cause ticket: HR (staffing/wellness)
- Strategic flag: Leadership (compensation/retention review)
```
### Cross-Functional Triggers
Certain code combinations automatically involve multiple teams:
| Pattern | Trigger Codes | Teams Involved |
|---------|---------------|----------------|
| Security Breach | E4.03 + R1.04 | Security, Legal, Leadership |
| Discrimination | A3.05 + P1.02 | HR, Legal, DEI |
| Billing Dispute | V1.04 + V2.02 | Billing, Legal, CX |
| Product Failure (Critical) | O1.01 (I3) | Engineering, QA, CX |
| Systemic Issue | SY-* codes | Leadership, Operations |
---
## Appendix: Domain-Team Mapping
### Complete Domain-to-Team Matrix
| Domain | Category | Default Team | Escalation Path |
|--------|----------|--------------|-----------------|
| **O** | O1 Function | product_engineering | eng_lead -> product_dir |
| | O2 Quality | quality_control | qa_mgr -> product_dir |
| | O3 Completeness | fulfillment | fulfillment_mgr -> ops_dir |
| | O4 Fit | product | product_mgr -> product_dir |
| **P** | P1 Attitude | hr_training | hr_dir |
| | P2 Competence | training | training_mgr -> hr_dir |
| | P3 Responsiveness | operations | ops_mgr -> ops_dir |
| | P4 Communication | training | training_mgr -> hr_dir |
| **J** | J1 Timing | operations | ops_mgr -> ops_dir |
| | J2 Ease | process_improvement | ops_dir |
| | J3 Reliability | operations | ops_mgr -> ops_dir |
| | J4 Resolution | customer_service | cx_dir |
| **E** | E1 Physical | facilities | facilities_mgr -> facilities_dir |
| | E2 Digital | it | tech_lead -> cto |
| | E3 Ambiance | facilities | facilities_mgr -> facilities_dir |
| | E4 Safety | safety | safety_officer -> coo |
| **A** | A1 Availability | operations | ops_mgr -> ops_dir |
| | A2 Accessibility | accessibility | accessibility_lead -> compliance_dir |
| | A3 Inclusivity | dei | dei_dir -> hr_dir |
| | A4 Convenience | operations | ops_mgr -> ops_dir |
| **V** | V1 Price | pricing | pricing_mgr -> finance_dir |
| | V2 Transparency | legal | legal_counsel -> general_counsel |
| | V3 Effort | cx | cx_mgr -> cx_dir |
| | V4 Worth | leadership | cx_dir -> cmo -> ceo |
| **R** | R1 Integrity | leadership | legal -> ceo |
| | R2 Dependability | operations | ops_dir -> coo |
| | R3 Recovery | cx | cx_mgr -> cx_dir |
| | R4 Loyalty | marketing | marketing_mgr -> cmo |
---
## Version History
| Version | Date | Changes |
|---------|------|---------|
| 1.0 | 2026-01-23 | Initial release with URT v5.1 |
---
*This document is part of Track B (Engineering/Implementation) for the Universal Review Taxonomy v5.1.*

692
urt-taxonomy/track-b-engineering/B3-owner-routing.yaml Normal file
View File

@@ -0,0 +1,692 @@
# =============================================================================
# B3: Owner Routing Matrix
# Universal Review Taxonomy v5.1
# =============================================================================
#
# Maps URT codes to responsible teams/individuals for ticket routing.
# Resolution order: subcode_overrides > category_overrides > domain_defaults
#
# Status: Production Ready
# Release Date: 2026-01-23
# Prerequisite: B1-urt-codes.yaml (Code Registry)
# =============================================================================
version: "5.1"
status: "production"
release_date: "2026-01-23"
# =============================================================================
# ROUTING RULES
# =============================================================================
routing_rules:
# ---------------------------------------------------------------------------
# DOMAIN DEFAULTS
# Baseline routing for each of the 7 experience domains
# ---------------------------------------------------------------------------
domain_defaults:
O:
team: "product"
escalation: "product_director"
description: "Core product/service delivery issues"
typical_response: "Product review, quality control audit"
P:
team: "hr"
escalation: "hr_director"
description: "Personnel behavior and performance issues"
typical_response: "Staff coaching, training intervention"
J:
team: "operations"
escalation: "operations_director"
description: "Process, timing, and operational flow issues"
typical_response: "Process improvement, workflow optimization"
E:
team: "facilities"
escalation: "facilities_director"
description: "Physical and digital environment issues"
typical_response: "Maintenance dispatch, IT ticket"
A:
team: "compliance"
escalation: "compliance_director"
description: "Availability, accessibility, and inclusivity issues"
typical_response: "Accessibility audit, policy review"
V:
team: "finance"
escalation: "finance_director"
description: "Pricing, value, and cost-related issues"
typical_response: "Pricing review, refund authorization"
R:
team: "leadership"
escalation: "ceo"
description: "Trust, relationship, and organizational integrity issues"
typical_response: "Executive review, strategic response"
# ---------------------------------------------------------------------------
# CATEGORY-LEVEL OVERRIDES
# More specific routing for category-level patterns
# ---------------------------------------------------------------------------
category_overrides:
# Offering categories
O1:
team: "product_engineering"
escalation: "engineering_lead"
notes: "Function issues often require technical investigation"
O2:
team: "quality_control"
escalation: "qa_manager"
notes: "Quality issues route to QC team"
O3:
team: "fulfillment"
escalation: "fulfillment_manager"
notes: "Completeness issues often involve shipping/delivery"
O4:
team: "product"
escalation: "product_manager"
notes: "Fit issues inform product development"
# People categories
P1:
team: "hr_training"
escalation: "hr_director"
notes: "Attitude issues require coaching/culture work"
P2:
team: "training"
escalation: "training_manager"
notes: "Competence issues indicate training gaps"
P3:
team: "operations"
escalation: "operations_manager"
notes: "Responsiveness often tied to operational capacity"
P4:
team: "training"
escalation: "training_manager"
notes: "Communication skills are trainable"
# Journey categories
J1:
team: "operations"
escalation: "operations_manager"
notes: "Timing issues require process optimization"
J2:
team: "process_improvement"
escalation: "operations_director"
notes: "Ease/friction issues benefit from process redesign"
J3:
team: "operations"
escalation: "operations_manager"
notes: "Reliability issues need root cause analysis"
J4:
team: "customer_service"
escalation: "cx_director"
notes: "Resolution issues are CX team primary focus"
# Environment categories
E1:
team: "facilities"
escalation: "facilities_manager"
notes: "Physical space issues"
E2:
team: "it"
escalation: "cto"
notes: "Digital space issues route to IT/Engineering"
E3:
team: "facilities"
escalation: "facilities_manager"
notes: "Ambiance is facility management concern"
E4:
team: "safety"
escalation: "safety_officer"
notes: "Safety issues require immediate attention"
# Access categories
A1:
team: "operations"
escalation: "operations_director"
notes: "Availability is operational capacity"
A2:
team: "accessibility"
escalation: "compliance_director"
notes: "Accessibility requires specialized audit"
A3:
team: "dei"
escalation: "dei_director"
notes: "Inclusivity issues route to DEI team"
A4:
team: "operations"
escalation: "operations_manager"
notes: "Convenience is operational concern"
# Value categories
V1:
team: "pricing"
escalation: "finance_director"
notes: "Price issues need pricing team review"
V2:
team: "legal"
escalation: "general_counsel"
notes: "Transparency issues may have legal implications"
V3:
team: "cx"
escalation: "cx_director"
notes: "Effort issues are customer experience concern"
V4:
team: "leadership"
escalation: "ceo"
notes: "Worth/satisfaction is strategic concern"
# Relationship categories
R1:
team: "leadership"
escalation: "ceo"
notes: "Integrity issues are executive concern"
R2:
team: "operations"
escalation: "coo"
notes: "Dependability reflects operational excellence"
R3:
team: "cx"
escalation: "cx_director"
notes: "Recovery is core CX competency"
R4:
team: "marketing"
escalation: "cmo"
notes: "Loyalty is marketing/retention concern"
# ---------------------------------------------------------------------------
# SUBCODE-LEVEL OVERRIDES
# Most specific routing for individual subcodes
# ---------------------------------------------------------------------------
subcode_overrides:
# Critical safety subcodes - always escalate
E4.01:
team: "safety"
escalation: "safety_officer"
auto_escalate: true
sla_override_hours: 1
notes: "Physical safety is always critical"
E4.02:
team: "safety"
escalation: "safety_officer"
auto_escalate: true
sla_override_hours: 2
notes: "Health/hygiene requires immediate attention"
E4.05:
team: "safety"
escalation: "safety_officer"
auto_escalate: true
sla_override_hours: 1
notes: "Emergency readiness is critical"
# Digital/IT specific subcodes
E2.01:
team: "ux_design"
escalation: "ux_lead"
notes: "Interface design issues route to UX"
E2.02:
team: "engineering"
escalation: "engineering_lead"
notes: "Digital functionality is engineering concern"
E2.03:
team: "devops"
escalation: "cto"
notes: "Performance issues may be infrastructure"
E2.04:
team: "ux_design"
escalation: "ux_lead"
notes: "Navigation is UX concern"
E2.05:
team: "mobile_engineering"
escalation: "mobile_lead"
notes: "Mobile experience routes to mobile team"
# Discrimination - always escalate
A3.05:
team: "hr"
escalation: "hr_director"
auto_escalate: true
sla_override_hours: 4
co_owner: "legal"
notes: "Equal treatment issues require HR and legal"
# Accessibility compliance
A2.01:
team: "facilities"
escalation: "compliance_director"
co_owner: "accessibility"
notes: "Physical accessibility may require ADA compliance"
A2.02:
team: "accessibility"
escalation: "compliance_director"
notes: "Visual accessibility is WCAG concern"
A2.03:
team: "accessibility"
escalation: "compliance_director"
notes: "Hearing accessibility is compliance concern"
A2.04:
team: "ux_design"
escalation: "compliance_director"
notes: "Cognitive accessibility affects design"
A2.05:
team: "engineering"
escalation: "compliance_director"
notes: "Digital accessibility requires eng work"
# Security breaches
E4.03:
team: "security"
escalation: "cso"
auto_escalate: true
sla_override_hours: 1
notes: "Security incidents require immediate response"
# Ethics/integrity - executive level
R1.04:
team: "leadership"
escalation: "ceo"
auto_escalate: true
sla_override_hours: 4
co_owner: "legal"
notes: "Ethics issues are executive priority"
# Billing/refund specific
V1.04:
team: "billing"
escalation: "finance_director"
notes: "Hidden costs need billing team review"
V2.02:
team: "billing"
escalation: "finance_director"
co_owner: "legal"
notes: "Fee disclosure may have compliance implications"
V2.04:
team: "legal"
escalation: "general_counsel"
notes: "Terms fairness is legal concern"
# Staff-specific issues
P1.02:
team: "hr"
escalation: "hr_director"
notes: "Respect issues need HR intervention"
P2.04:
team: "hr"
escalation: "hr_director"
notes: "Professionalism is HR concern"
# ---------------------------------------------------------------------------
# INTENSITY-BASED ESCALATION
# SLA and escalation rules by sentiment intensity
# ---------------------------------------------------------------------------
intensity_escalation:
I3:
auto_escalate: true
sla_hours: 4
notification_level: "director"
priority: "critical"
description: "Strong negative - requires immediate attention"
I2:
auto_escalate: false
sla_hours: 24
notification_level: "manager"
priority: "high"
description: "Moderate negative - standard priority"
I1:
auto_escalate: false
sla_hours: 72
notification_level: "team"
priority: "normal"
description: "Mild negative - normal queue"
# ---------------------------------------------------------------------------
# VALENCE-BASED ROUTING MODIFIERS
# Different handling for positive vs negative feedback
# ---------------------------------------------------------------------------
valence_routing:
"V+":
action: "acknowledge"
sla_hours: 48
team_modifier: null
notes: "Positive feedback routes to recognition workflow"
"V-":
action: "resolve"
sla_hours: null # Uses intensity-based SLA
team_modifier: null
notes: "Negative feedback uses standard routing"
"V0":
action: "log"
sla_hours: null # No SLA for neutral
team_modifier: null
notes: "Neutral observations logged for analytics"
"V+-":
action: "resolve"
sla_hours: null
team_modifier: null
notes: "Mixed feedback prioritizes negative element"
# ---------------------------------------------------------------------------
# CAUSAL CODE ROUTING
# Routes for root cause analysis codes
# ---------------------------------------------------------------------------
causal_routing:
# Conditions layer
"CD-S":
team: "hr"
escalation: "hr_director"
notes: "Staff state issues - HR/wellness concern"
"CD-T":
team: "operations"
escalation: "operations_manager"
notes: "Team dynamics - operational leadership"
"CD-E":
team: "facilities"
escalation: "facilities_manager"
notes: "Equipment issues - facilities/IT"
"CD-F":
team: "facilities"
escalation: "facilities_director"
notes: "Facility issues - maintenance"
"CD-O":
team: "operations"
escalation: "operations_director"
notes: "Operational conditions - ops leadership"
# Management layer
"MG-P":
team: "operations"
escalation: "operations_director"
notes: "Planning failures - ops management"
"MG-T":
team: "training"
escalation: "hr_director"
notes: "Training gaps - L&D team"
"MG-O":
team: "operations"
escalation: "operations_director"
notes: "Oversight failures - management review"
"MG-R":
team: "procurement"
escalation: "finance_director"
notes: "Resource issues - procurement/finance"
"MG-C":
team: "leadership"
escalation: "coo"
notes: "Communication failures - leadership"
# Systemic layer
"SY-R":
team: "leadership"
escalation: "cfo"
notes: "Resource decisions - executive finance"
"SY-P":
team: "leadership"
escalation: "coo"
notes: "Policy issues - executive operations"
"SY-C":
team: "leadership"
escalation: "ceo"
notes: "Culture issues - executive leadership"
"SY-S":
team: "leadership"
escalation: "coo"
notes: "Standards issues - executive ops"
"SY-H":
team: "hr"
escalation: "chro"
notes: "Human capital - HR leadership"
"SY-X":
team: "leadership"
escalation: "ceo"
notes: "External factors - executive strategy"
# =============================================================================
# SCHEDULING AND ON-CALL
# =============================================================================
scheduling:
business_hours:
start: "09:00"
end: "17:00"
timezone: "UTC"
days: ["monday", "tuesday", "wednesday", "thursday", "friday"]
on_call_teams:
- team: "support"
coverage: "24/7"
handles:
- "I3" # Critical intensity
- "E4" # Safety issues
- "E4.03" # Security
- team: "operations"
coverage: "extended"
hours: "06:00-22:00"
handles:
- "J4" # Resolution issues
- "J1" # Timing issues
holiday_coverage:
minimum_teams: ["support"]
escalation_delay_multiplier: 1.5
# =============================================================================
# MULTI-CODE SPAN ROUTING
# =============================================================================
multi_code_rules:
# When a span has multiple codes, determine primary owner
priority_resolution:
method: "highest_priority_team"
tiebreaker: "primary_code_wins"
# Domain priority order (for conflicts)
domain_priority:
1: "E4" # Safety always first
2: "R1" # Integrity/ethics
3: "A3" # Discrimination/inclusivity
4: "P" # People issues
5: "V" # Value issues
6: "J" # Journey issues
7: "O" # Offering issues
8: "E" # Environment (non-safety)
9: "A" # Access (non-discrimination)
10: "R" # Relationship (non-integrity)
# Secondary code handling
secondary_code_behavior:
notify_co_owners: true
add_to_ticket_tags: true
require_sign_off: false
# =============================================================================
# CROSS-FUNCTIONAL ROUTING PATTERNS
# =============================================================================
cross_functional:
# Patterns that always involve multiple teams
patterns:
security_breach:
trigger_codes: ["E4.03", "R1.04"]
teams: ["security", "legal", "leadership"]
escalation: "ceo"
sla_hours: 1
discrimination_complaint:
trigger_codes: ["A3.05", "P1.02"]
teams: ["hr", "legal", "dei"]
escalation: "hr_director"
sla_hours: 4
billing_dispute:
trigger_codes: ["V1.04", "V2.02", "V2.04"]
teams: ["billing", "legal", "cx"]
escalation: "finance_director"
sla_hours: 24
product_failure:
trigger_codes: ["O1.01", "O1.04"]
conditions:
intensity: "I3"
teams: ["product_engineering", "qa", "cx"]
escalation: "product_director"
sla_hours: 4
systemic_issue:
trigger_codes: ["SY-C", "SY-P", "SY-R"]
teams: ["leadership", "operations"]
escalation: "ceo"
sla_hours: 24
# =============================================================================
# TEAM DEFINITIONS
# =============================================================================
teams:
product:
name: "Product Team"
email: "product@example.com"
slack: "#product-team"
manager: "product_director"
hr:
name: "Human Resources"
email: "hr@example.com"
slack: "#hr-team"
manager: "hr_director"
operations:
name: "Operations"
email: "ops@example.com"
slack: "#operations"
manager: "operations_director"
facilities:
name: "Facilities Management"
email: "facilities@example.com"
slack: "#facilities"
manager: "facilities_director"
it:
name: "IT/Engineering"
email: "it@example.com"
slack: "#engineering"
manager: "cto"
compliance:
name: "Compliance"
email: "compliance@example.com"
slack: "#compliance"
manager: "compliance_director"
finance:
name: "Finance"
email: "finance@example.com"
slack: "#finance"
manager: "finance_director"
leadership:
name: "Executive Leadership"
email: "leadership@example.com"
slack: "#leadership"
manager: "ceo"
cx:
name: "Customer Experience"
email: "cx@example.com"
slack: "#customer-experience"
manager: "cx_director"
safety:
name: "Safety & Security"
email: "safety@example.com"
slack: "#safety"
manager: "safety_officer"
legal:
name: "Legal"
email: "legal@example.com"
slack: "#legal"
manager: "general_counsel"
training:
name: "Learning & Development"
email: "training@example.com"
slack: "#training"
manager: "training_manager"
accessibility:
name: "Accessibility"
email: "accessibility@example.com"
slack: "#accessibility"
manager: "accessibility_lead"
dei:
name: "Diversity, Equity & Inclusion"
email: "dei@example.com"
slack: "#dei"
manager: "dei_director"
# =============================================================================
# ESCALATION HIERARCHY
# =============================================================================
escalation_hierarchy:
levels:
1: "team_lead"
2: "manager"
3: "director"
4: "vp"
5: "c_suite"
auto_escalation_rules:
sla_breach:
after_hours: 2
escalate_to: "+1 level"
repeated_issue:
threshold: 3
window_days: 30
escalate_to: "+1 level"
vip_customer:
always_notify: "director"
sla_multiplier: 0.5
# =============================================================================
# SLA DEFINITIONS
# =============================================================================
sla:
response_times:
critical:
initial_response_hours: 1
resolution_hours: 4
update_frequency_hours: 1
high:
initial_response_hours: 4
resolution_hours: 24
update_frequency_hours: 4
normal:
initial_response_hours: 24
resolution_hours: 72
update_frequency_hours: 24
low:
initial_response_hours: 48
resolution_hours: 168 # 1 week
update_frequency_hours: 48
breach_actions:
warning:
at_percent: 75
action: "notify_manager"
critical:
at_percent: 100
action: "escalate_and_notify_director"
overdue:
at_percent: 150
action: "escalate_to_vp"

2895
urt-taxonomy/track-b-engineering/B4-api-contract.yaml Normal file
View File

File diff suppressed because it is too large Load Diff

1204
urt-taxonomy/track-c-analytics/C1-Issue-Lifecycle-Framework.md Normal file
View File

File diff suppressed because it is too large Load Diff

1148
urt-taxonomy/track-c-analytics/C2-KPI-Mapping-Guide.md Normal file
View File

File diff suppressed because it is too large Load Diff

1905
urt-taxonomy/track-d-integration/D1-Dashboard-Specification.md Normal file
View File

File diff suppressed because it is too large Load Diff