alezmad/whyrating-engine-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
whyrating-engine-legacy/urt-taxonomy/track-a-training/A1-Annotator-Quickstart.md
Alejandro Gutiérrez 3eda9bdbfa Add complete URT v5.1 taxonomy framework (11 artifacts) 
Universal Review Taxonomy v5.1 implementation with:
- Track A (Training): A1 Quickstart, A2 QA Protocol, A3 Calibration Set, A4 Full Manual
- Track B (Engineering): B1 Code Registry, B2 Database Schema, B3 Owner Routing, B4 API Contract
- Track C (Analytics): C1 Issue Lifecycle, C2 KPI Mapping Guide
- Track D (Integration): D1 Dashboard Specification

Covers 7 domains, 28 categories, 138 subcodes, 16 causal codes, and 7 metadata dimensions.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 10:51:41 +00:00

926 lines
39 KiB
Markdown
Raw Permalink Blame History

# A1: Annotator Quickstart Guide
## Universal Review Taxonomy (URT) v5.1
**Purpose**: Quick reference for human annotators to classify review spans
**Version**: 5.1 | **Status**: Production Ready | **Date**: 2026-01-23
---
## Table of Contents
1. [Quick Decision Tree](#1-quick-decision-tree)
2. [Domain Summary Table](#2-domain-summary-table)
3. [Category Quick Reference (28 Categories)](#3-category-quick-reference)
- [3.5 High-Frequency Subcodes](#35-high-frequency-subcodes-quick-reference)
4. [Metadata Cheat Sheet](#4-metadata-cheat-sheet)
5. [Common Mistakes to Avoid](#5-common-mistakes-to-avoid)
6. [V vs R Rule: Scam vs Rip-off](#6-v-vs-r-rule-scam-vs-rip-off)
7. [J4 vs R3 Rule: Process vs Ownership](#7-j4-vs-r3-rule-process-vs-ownership)
8. [Span Boundary Rules](#8-span-boundary-rules)
9. [Worked Examples](#9-worked-examples)
10. [USN Quick Reference](#10-usn-quick-reference)
11. [Billing/Returns/Refunds Decision Table](#11-billingreturnsrefunds-decision-table)
12. [Data Privacy/Security Decision Table](#12-data-privacysecurity-decision-table)
13. [CR Trigger Word Reference](#13-cr-trigger-word-reference)
14. [Causal Chain Quick Guide](#14-causal-chain-quick-guide-full-profile-only)
15. [Inter-Annotator Agreement Rules](#15-inter-annotator-agreement-rules)
---
## 1. Quick Decision Tree
Use this flowchart for every span. Go top-to-bottom; first "YES" wins.
```
START: Read the span
|
v
+-----------------------------------------------+
| Is it about the THING they bought/received? |
| (product, service, treatment, outcome) |
+-----------------------------------------------+
| YES --> O (Offering)
| NO |
v
+-----------------------------------------------+
| Is it about HOW PEOPLE behaved? |
| (staff attitude, skill, communication) |
+-----------------------------------------------+
| YES --> P (People)
| NO |
v
+-----------------------------------------------+
| Is it about TIME, STEPS, or FRICTION? |
| (waiting, process, ease, resolution steps) |
+-----------------------------------------------+
| YES --> J (Journey)
| NO |
v
+-----------------------------------------------+
| Is it about the SPACE or INTERFACE? |
| (physical place, app/website, ambiance) |
+-----------------------------------------------+
| YES --> E (Environment)
| NO |
v
+-----------------------------------------------+
| Is it about ABILITY TO ACCESS or INCLUSION? |
| (availability, disability, language, bias) |
+-----------------------------------------------+
| YES --> A (Access)
| NO |
v
+-----------------------------------------------+
| Is it about PRICE, COST, or WORTH? |
| (money, effort, value, transparency) |
+-----------------------------------------------+
| YES --> V (Value)
| NO |
v
+-----------------------------------------------+
| Is it about TRUST, PATTERNS, or LOYALTY? |
| (honesty over time, brand, recovery, bond) |
+-----------------------------------------------+
| YES --> R (Relationship)
| NO |
v
RE-READ SPAN -- may need splitting
```
**Memory Aid**: O-P-J-E-A-V-R = "**O**ffer **P**eople a **J**ourney in an **E**nvironment with **A**ccess to **V**alue and **R**elationship"
---
## 2. Domain Summary Table
| Code | Domain | Core Question | One-Line Definition | Example (+) | Example (-) |
|------|--------|---------------|---------------------|-------------|-------------|
| **O** | Offering | Does it work? | The core product/service delivered | "Perfectly cooked steak" | "Phone won't charge" |
| **P** | People | How did they treat me? | Human interactions and behavior | "So helpful and patient" | "Rude and dismissive" |
| **J** | Journey | Was it smooth? | Process, timing, friction | "Seated immediately" | "Waited 45 minutes" |
| **E** | Environment | Is the space okay? | Physical, digital, ambient context | "Beautiful clean space" | "App keeps crashing" |
| **A** | Access | Can I get it? | Availability, accessibility, inclusion | "Open 24/7" | "No wheelchair ramp" |
| **V** | Value | Is it worth it? | Cost, pricing, and worth | "Great bang for buck" | "Total rip-off" |
| **R** | Relationship | Can I trust them? | Trust, reliability, loyalty | "Always reliable" | "Shady company" |
### Default Owner Routing
| Domain | Default Owner |
|--------|---------------|
| O | Product / Operations |
| P | HR / Training |
| J | Operations / Process |
| E | Facilities / IT |
| A | Compliance / Design |
| V | Finance / Pricing |
| R | Leadership / CX |
---
## 3. Category Quick Reference
**28 Categories** - 4 per domain. Use this for URT-Core or to narrow down to subcodes.
### O - Offering (4 categories, 18 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **O1** | Function | Does it do what it's supposed to do? |
| **O2** | Quality | How well is it made or executed? |
| **O3** | Completeness | Is everything included that should be? |
| **O4** | Fit | Does it match customer's specific needs? |
### P - People (4 categories, 20 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **P1** | Attitude | Disposition, manner, emotional tone of staff |
| **P2** | Competence | Knowledge, skill, professional capability |
| **P3** | Responsiveness | Attentiveness, initiative, follow-through |
| **P4** | Communication | Quality of information exchange |
### J - Journey (4 categories, 20 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **J1** | Timing | Speed, punctuality, time management |
| **J2** | Ease | Effort required and friction encountered |
| **J3** | Reliability | Consistency and predictability of process |
| **J4** | Resolution | How problems are handled when they arise |
### E - Environment (4 categories, 20 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **E1** | Physical Space | Tangible environment attributes |
| **E2** | Digital Space | Online and application interface |
| **E3** | Ambiance | Intangible environmental qualities (mood, noise) |
| **E4** | Safety | Security and wellbeing factors |
### A - Access (4 categories, 20 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **A1** | Availability | Can you get it when you need it? |
| **A2** | Accessibility | Can everyone use it regardless of ability? |
| **A3** | Inclusivity | Does it work for diverse backgrounds? |
| **A4** | Convenience | Is it easy to reach and engage with? |
### V - Value (4 categories, 20 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **V1** | Price | The monetary cost itself |
| **V2** | Transparency | Clarity and honesty about costs |
| **V3** | Effort | Non-monetary costs (time, hassle, mental load) |
| **V4** | Worth | Overall value assessment |
### R - Relationship (4 categories, 20 subcodes)
| Code | Category | Definition |
|------|----------|------------|
| **R1** | Integrity | Honesty and ethical behavior |
| **R2** | Dependability | Consistency over time |
| **R3** | Recovery | Response to failures (ownership, apology, compensation) |
| **R4** | Loyalty | Investment in ongoing relationship |
---
## 3.5 High-Frequency Subcodes Quick Reference
These are the most commonly used subcodes across all domains. Memorize these first.
### Offering (O) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| O1.01 | Works/Doesn't Work | Basic functionality | "Software runs perfectly" | "Car won't start" |
| O1.02 | Performance Level | How well it operates | "Incredibly fast" | "Sluggish and laggy" |
| O1.05 | Outcome Achievement | Did customer accomplish goal? | "Passed my exam!" | "Treatment didn't work" |
| O2.01 | Materials/Inputs | Quality of components | "Real leather" | "Cheap plastic parts" |
| O2.03 | Presentation | Visual/aesthetic quality | "Gorgeous plating" | "Looked thrown together" |
| O2.05 | Condition at Delivery | State when received | "Still warm" | "Arrived damaged" |
| O4.01 | Specification Match | Matches what was ordered | "Exactly right" | "Wrong size delivered" |
### People (P) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| P1.01 | Warmth/Friendliness | Approachability | "So welcoming" | "Cold and unfriendly" |
| P1.02 | Respect | Treating with dignity | "Made me feel valued" | "Talked down to me" |
| P1.03 | Empathy | Understanding situation | "Really got my frustration" | "Couldn't care less" |
| P2.01 | Knowledge | Understanding of products | "Knew every detail" | "Couldn't answer basics" |
| P2.03 | Problem-Solving | Ability to address issues | "Found creative solution" | "Just said 'can't'" |
| P3.01 | Attentiveness | Awareness of needs | "Always checking in" | "Had to flag them down" |
| P3.03 | Availability | Present when needed | "Easy to find someone" | "Impossible to reach" |
| P4.01 | Clarity | Understandable info | "Explained clearly" | "Confusing jargon" |
### Journey (J) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| J1.01 | Wait Time | Time spent waiting | "Seated immediately" | "45 min past appointment" |
| J1.02 | Service Speed | Time for delivery | "Next day delivery" | "Took three weeks" |
| J1.03 | Response Time | Time to address inquiries | "Replied in minutes" | "Days for a response" |
| J2.01 | Simplicity | Straightforward processes | "So easy to do" | "Needlessly complicated" |
| J3.02 | Accuracy | Correct execution | "Order exactly right" | "Wrong items delivered" |
| J4.02 | Resolution Process | How problems handled | "Clear escalation path" | "Transferred in circles" |
| J4.03 | Resolution Speed | Time to fix | "Fixed same day" | "Took weeks to resolve" |
### Environment (E) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| E1.01 | Cleanliness | Hygiene and tidiness | "Spotless facilities" | "Filthy bathrooms" |
| E1.02 | Maintenance | Condition and upkeep | "Everything works" | "Broken equipment" |
| E2.01 | Interface Design | Visual/interaction quality | "Beautiful app" | "Cluttered mess" |
| E2.02 | Functionality | Features working | "Everything works" | "Buttons broken" |
| E3.01 | Atmosphere/Vibe | Overall mood | "Calm and relaxing" | "Stressful chaos" |
| E4.01 | Physical Safety | Protection from harm | "Felt completely safe" | "Dangerous conditions" |
### Access (A) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| A1.01 | Operating Hours | When accessible | "Open 24/7" | "Banker's hours only" |
| A1.02 | Booking Access | Ability to schedule | "Easy online booking" | "3 weeks for appointment" |
| A1.03 | Inventory/Capacity | Available | "Always in stock" | "Perpetually sold out" |
| A2.01 | Physical Accessibility | Mobility accommodations | "Wheelchair accessible" | "No ramps or elevators" |
| A3.01 | Language Support | Multiple languages | "Staff spoke my language" | "English only, no help" |
| A4.01 | Location | Physical accessibility | "Convenient location" | "Middle of nowhere" |
| A4.02 | Parking | Vehicle accommodation | "Easy free parking" | "Parking nightmare" |
### Value (V) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| V1.01 | Absolute Price | The actual cost | "Very affordable" | "Outrageously expensive" |
| V1.04 | Hidden Costs | Unexpected charges | "No hidden fees" | "Surprise $50 charge" |
| V2.01 | Pricing Clarity | Understanding costs | "Clear price list" | "Impossible to understand" |
| V2.03 | Advertising Accuracy | Marketing matches reality | "As advertised" | "Bait and switch" |
| V2.04 | Terms Fairness | Policy reasonableness | "Fair cancellation" | "Predatory contract" |
| V4.01 | Overall Value | Total assessment | "Excellent value" | "Total rip-off" |
| V4.02 | Quality-Price Ratio | What you get for payment | "Great for the price" | "Pay more, get less" |
### Relationship (R) - Top Subcodes
| Code | Name | Definition | + Example | - Example |
|------|------|------------|-----------|-----------|
| R1.01 | Truthfulness | Accurate representations | "Everything as stated" | "They flat out lied" |
| R1.02 | Promise Keeping | Honoring commitments | "Always deliver on word" | "Never keep promises" |
| R1.04 | Ethics | Moral business conduct | "Ethical company" | "Shady business practices" |
| R2.02 | Consistency | Same experience over time | "Always great" | "Varies wildly" |
| R3.01 | Acknowledgment | Admitting failures | "Owned their mistake" | "Denied any wrongdoing" |
| R3.03 | Compensation | Making amends | "More than made up for it" | "Offered nothing" |
| R4.01 | Recognition | Acknowledging repeat customers | "They remember me" | "Stranger every time" |
---
## 4. Metadata Cheat Sheet
**7 dimensions, 24 total values**. Profile determines which are required.
### 4.1 Valence (V) - ALWAYS REQUIRED
| Code | Label | When to Use | Markers |
|------|-------|-------------|---------|
| **V+** | Positive | Praise, satisfaction | "great," "loved," "excellent" |
| **V-** | Negative | Complaint, dissatisfaction | "terrible," "hated," "worst" |
| **V0** | Neutral | Factual observation | "they have," "it was" |
| **V+-** | Mixed | Both positive and negative in same span | "good but," "despite," "however" |
**Rule**: Prefer splitting over V+- unless the positive and negative target the *same thing*.
### 4.2 Intensity (I) - ALWAYS REQUIRED
| Code | Label | When to Use | Markers |
|------|-------|-------------|---------|
| **I1** | Mild | Slight preference/concern | "a bit," "somewhat," "could be better" |
| **I2** | Moderate | Clear but not extreme | Standard adjectives, no intensifiers |
| **I3** | Strong | Emphatic, intense | "extremely," "worst ever," CAPS, !!!, profanity |
### 4.3 Specificity (S) - Standard+
| Code | Label | When to Use | Example |
|------|-------|-------------|---------|
| **S1** | Vague | General impression only | "Service was bad" |
| **S2** | Moderate | Some details or context | "Service was slow at dinner" |
| **S3** | Specific | Names, times, amounts | "Waiter John took 40 mins at 7pm Saturday" |
### 4.4 Actionability (A) - Standard+
| Code | Label | When to Use | Example |
|------|-------|-------------|---------|
| **A1** | Low | Feeling, no clear action path | "I just didn't like it" |
| **A2** | Medium | Suggests improvement area | "Food could be warmer" |
| **A3** | High | Specific implementable fix | "West bathroom stall lock is broken" |
**Note**: S and A are correlated but not identical. "John was rude" = S3 (named person) but A2 (action unclear).
### 4.5 Temporal Reference (T) - Standard+
| Code | Label | When to Use |
|------|-------|-------------|
| **TC** | Current | This specific visit/experience (default) |
| **TR** | Recent | Recent pattern of experiences |
| **TH** | Historical | Long-standing pattern |
| **TF** | Future | Expectations or predictions |
### 4.6 Evidence Type (E) - Standard+
| Code | Label | When to Use | Constraint |
|------|-------|-------------|------------|
| **ES** | Stated | Explicitly said by customer | Text directly says it (default) |
| **EI** | Inferred | Logically entailed by text | Must be directly deducible, not speculative |
| **EC** | Contextual | Requires surrounding text | Span uses "that," "they," "it" referencing earlier |
**EI Rule**: Only use when text *logically entails* the inference.
- OK: "Took 3 weeks to reply" = EI for slow response time
- NOT OK: "Seemed tired" = Cannot infer fatigue without more evidence
### 4.7 Comparative Reference (CR) - Standard+
| Code | Label | When to Use | Trigger Words |
|------|-------|-------------|---------------|
| **CR-N** | None | No comparison to past (DEFAULT - 90% of spans) | (no temporal language) |
| **CR-B** | Better | Explicit improvement | "better now," "improved," "finally fixed" |
| **CR-W** | Worse | Explicit decline | "worse now," "used to be good," "gone downhill" |
| **CR-S** | Same | Explicitly unchanged | "still," "as always," "same as before" |
**CR Rules**:
1. Only assign CR-B/W/S when customer *explicitly compares* to their own past
2. CR-N is default -- only change when comparison is explicit
3. Self-comparison only (not competitor comparison)
4. Implicit decline counts: "used to be good" = CR-W
### Metadata by Profile
| Dimension | Lite | Core | Standard | Full |
|-----------|:----:|:----:|:--------:|:----:|
| Valence | Required | Required | Required | Required |
| Intensity | Optional | Required | Required | Required |
| Specificity | -- | -- | Required | Required |
| Actionability | -- | -- | Required | Required |
| Temporal | -- | Optional | Required | Required |
| Evidence | -- | -- | Required | Required |
| Comparative | -- | Optional | Required | Required |
---
## 5. Common Mistakes to Avoid
### Top 10 Disambiguation Pitfalls
| # | Mistake | Correct Approach |
|---|---------|------------------|
| 1 | **Confusing O1.01 (Works/Doesn't Work) with J3.03 (System Uptime)** | O1.01 = product function. J3.03 = system/service availability. "Car won't start" = O1.01. "Website was down" = J3.03. |
| 2 | **Confusing P3.04 (Follow-Through) with R1.02 (Promise Keeping)** | P3.04 = specific interaction ("said they'd call back, didn't"). R1.02 = trust/pattern ("they never keep their word"). |
| 3 | **Confusing J2.01 (Simplicity) with E2.04 (Navigation)** | J2 = effort/friction in *process* ("too many steps"). E2 = qualities of *interface* ("buttons don't work"). |
| 4 | **Confusing J3.01 (Consistency) with O1.04 (Reliability)** | J3.01 = process delivers same result each time. O1.04 = product itself works consistently. "Pizza is always good" = J3.01. "Phone always connects" = O1.04. |
| 5 | **Confusing E2.03 (Interface Performance) with O1.02 (Product Performance)** | E2.03 = the app/interface is slow. O1.02 = the core product performs poorly ("car accelerates slowly"). |
| 6 | **Confusing A3.05 (Equal Treatment) with P1.02 (Respect)** | A3.05 = identity-based discrimination ("treated differently because of X"). P1.02 = general disrespect without identity framing. |
| 7 | **Confusing V2.xx with R1.xx** | V2 = pricing/terms/advertising specifically. R1 = trust/integrity/organizational character framing. |
| 8 | **Over-splitting cause-effect spans** | "Long wait because understaffed" = single span. Primary = J1.01, Secondary = A1.04. Only split if different sentences or genuinely different issues. |
| 9 | **Assigning CR-B/W/S without explicit comparison** | CR-N is default. "Service is slow" = CR-N. "Service is *still* slow" = CR-S. Must have explicit temporal language. |
| 10 | **Inventing causal codes** | Causal chain only when text explicitly states or logically entails. "Probably a training issue" = annotator speculation = NO causal code. |
### Quick Disambiguation Tests
| Question | If YES | If NO |
|----------|--------|-------|
| Does it mention the product/service itself? | Likely O | Check P-R |
| Does it name or describe a person? | Likely P | Check J-R |
| Does it mention time, steps, or waiting? | Likely J | Check E-R |
| Does it describe a place or interface? | Likely E | Check A-R |
| Does it mention accessibility or availability? | Likely A | Check V-R |
| Does it mention price, cost, or worth? | Likely V | Check R |
| Does it mention trust, patterns, or "always/never"? | Likely R | Re-read span |
---
## 6. V vs R Rule: Scam vs Rip-off
This is the most common confusion. Use this anchor:
### The Rule
| Feedback Type | Domain | Rationale |
|---------------|--------|-----------|
| **"This is a rip-off"** | **V4.01** | Complaint about the *exchange* (poor value) |
| **"This company is a scam"** | **R1.04** | Moral judgment about *intent* (they're dishonest) |
### Criminal Metaphor Mapping
| Word/Phrase | Domain | Reasoning |
|-------------|--------|-----------|
| "scam" | R1.04 | Character judgment |
| "fraud" | R1.04 | Character judgment |
| "crooks" | R1.04 | Character judgment |
| "rip-off" | V4.01 | Exchange complaint |
| "robbery" | V4.01 | Exchange complaint |
| "highway robbery" | V4.01 | Exchange complaint |
| "overpriced" | V1.01 or V4.02 | Price/value complaint |
### Decision Heuristic
**Ask**: Is the complaint about...
- **What I got vs. what I paid?** --> **V** (Value)
- **Their intent to deceive/harm me?** --> **R** (Relationship)
### Examples
| Span | Primary | Rationale |
|------|---------|-----------|
| "Total scam company" | R1.04 | Moral judgment about intent |
| "They charged me twice" | J3.02 | Process accuracy issue |
| "refused to refund" | R3.03 | Recovery/compensation failure |
| "This is robbery" | V4.01 | Exchange complaint |
| "They deliberately mislead people" | R1.01 | Organizational truthfulness |
| "Overpriced garbage" | V4.02 | Quality-price ratio |
---
## 7. J4 vs R3 Rule: Process vs Ownership
Both J4 and R3 deal with problems, but from different angles.
### The Distinction
| Domain | Focus | Key Question |
|--------|-------|--------------|
| **J4 (Resolution)** | *What they did* to fix it | How was the problem handled operationally? |
| **R3 (Recovery)** | *How they took responsibility* | Did they own it, apologize, make amends? |
### Category-Level Comparison
| J4 Category | R3 Category | Distinction |
|-------------|-------------|-------------|
| J4.01 Problem Acknowledgment | R3.01 Admitting Failures | J4.01 = recognized the issue exists. R3.01 = admitted wrongdoing/fault. |
| J4.02 Resolution Process | -- | J4.02 = the operational steps taken |
| J4.03 Resolution Speed | -- | J4.03 = how fast they fixed it |
| J4.04 Resolution Quality | R3.03 Compensation | J4.04 = was fix adequate? R3.03 = did they make amends beyond fixing? |
| J4.05 Prevention | R3.04 Improvement | J4.05 = process change. R3.04 = organizational commitment to change. |
### Decision Heuristic
**Ask**: Is the feedback about...
- **The mechanics of the fix?** --> **J4** (Journey: Resolution)
- **Their accountability and making things right?** --> **R3** (Relationship: Recovery)
### Examples
| Span | Primary | Rationale |
|------|---------|-----------|
| "They immediately sent a replacement" | J4.02 or R3.03 | If emphasizing speed/process = J4. If emphasizing making amends = R3. |
| "Took weeks to resolve" | J4.03 | Resolution speed |
| "They owned their mistake" | R3.01 | Admitting fault |
| "Sincere apology" | R3.02 | Expression of regret |
| "Offered nothing" | R3.03 | Compensation failure |
| "Changed their process" | J4.05 | Prevention action |
| "Blamed me instead" | R3.05 | Ownership failure |
---
## 8. Span Boundary Rules
### When to SPLIT
1. **At contrasting conjunctions**: but, however, although, despite, yet
2. **When subject changes**: location -> service -> food
3. **When valence changes**: positive -> negative
4. **When domain changes**: different aspect of experience
5. **When in different sentences** (unless tightly linked cause-effect)
### When to KEEP TOGETHER
1. **Cause-effect in same clause**: "Long wait because understaffed"
- Primary = impact (J1.01 Wait Time)
- Secondary = cause (A1.04 Staffing)
2. **Same target, mixed assessment**: "Good but overpriced" (same product)
3. **Elaboration of same point**: "Service was slow. Had to wait 40 minutes."
### Max Spans Guidance
| Sentence Type | Typical Spans |
|---------------|---------------|
| Simple sentence | 1-2 spans |
| Complex sentence | 2-3 spans |
| If tempted to create 4+ | Re-read; probably over-splitting |
### Cause-Effect Examples
| Span | Treatment | Primary | Secondary |
|------|-----------|---------|-----------|
| "Long wait because understaffed" | Single | J1.01 | A1.04 |
| "Food cold, had to send it back" | Single | O2.05 | J4.02 |
| "App crashed so I lost my data" | Single | E2.02 | E4.03 |
| "Rude waiter. Also the food was cold." | TWO spans | P1.02 / O2.05 | -- |
| "Support was slow because the system kept crashing" | Single | J1.03 | J3.03 |
### V+- vs Split Decision
| Feedback | Action |
|----------|--------|
| "Great product, terrible onboarding" | SPLIT: two different targets |
| "Good but overpriced" | KEEP: same target, use V+- |
| "Nice staff despite the chaos" | SPLIT if chaos is separate feedback |
**Test**: If the positive and negative parts have *different domain targets*, split them.
---
## 9. Worked Examples
### Example 1: Restaurant Review (Multiple Spans)
> "The pasta was perfectly al dente and beautifully plated, but our waiter disappeared for 20 minutes and we had to flag down someone else to get the check. Good value for a nice dinner out."
| Span | Primary | Secondary | V | I | S | A | T | E | CR |
|------|---------|-----------|---|---|---|---|---|---|-----|
| "pasta was perfectly al dente" | O1.02 | -- | V+ | I2 | S2 | A2 | TC | ES | CR-N |
| "beautifully plated" | O2.03 | -- | V+ | I2 | S2 | A2 | TC | ES | CR-N |
| "waiter disappeared for 20 minutes" | P3.03 | -- | V- | I2 | S3 | A3 | TC | ES | CR-N |
| "flag down someone else" | P3.01 | -- | V- | I2 | S2 | A2 | TC | ES | CR-N |
| "Good value for nice dinner" | V4.01 | -- | V+ | I1 | S1 | A1 | TC | ES | CR-N |
### Example 2: Software with Improvement Signal (CR-B)
> "Implementation took twice as long as promised, but once we went live, the system has been rock solid. Much more stable than the last version."
| Span | Primary | Secondary | V | I | CR | Note |
|------|---------|-----------|---|---|-----|------|
| "Implementation took twice as long" | J1.02 | R1.02 | V- | I3 | CR-N | No self-comparison |
| "system has been rock solid" | O1.04 | -- | V+ | I3 | CR-N | Current state |
| "Much more stable than last version" | O1.04 | -- | V+ | I2 | **CR-B** | Explicit improvement |
### Example 3: Persistent Problem (CR-S)
> "Still waiting forever for appointments. Third time I've complained about this and nothing has changed."
| Span | Primary | Secondary | V | I | CR | Note |
|------|---------|-----------|---|---|-----|------|
| "Still waiting forever" | A1.02 | J1.01 | V- | I3 | **CR-S** | "Still" = unchanged |
| "Third time I've complained" | J4.05 | R3.04 | V- | I2 | **CR-S** | Pattern, unchanged |
**Causal Chain** (Full profile only):
- MG-O (Oversight failure) -- Evidence: EI
- Justification: "Third time" + "nothing changed" logically entails oversight failure
### Example 4: Quality Decline (CR-W)
> "This used to be my favorite restaurant but the quality has really gone downhill since they changed chefs."
| Span | Primary | Secondary | V | I | CR | Note |
|------|---------|-----------|---|---|-----|------|
| "used to be my favorite" | R2.02 | -- | V- | I2 | **CR-W** | Past-positive implies decline |
| "quality has really gone downhill" | O2.02 | -- | V- | I3 | **CR-W** | Explicit decline |
### Example 5: Service Recovery (CR-B)
> "My package arrived damaged, which was frustrating. But they immediately sent a replacement and it arrived perfect. Great recovery!"
| Span | Primary | Secondary | V | I | CR |
|------|---------|-----------|---|---|-----|
| "package arrived damaged" | O2.05 | -- | V- | I2 | CR-N |
| "immediately sent replacement" | R3.03 | J4.03 | V+ | I2 | CR-N |
| "arrived perfect" | O2.05 | -- | V+ | I2 | **CR-B** |
| "Great recovery" | R3 | -- | V+ | I2 | CR-N |
### Example 6: Scam vs Rip-off (V vs R)
> "Total scam company. They charged me twice and refused to refund. This is robbery."
| Span | Primary | Secondary | V | I | CR |
|------|---------|-----------|---|---|-----|
| "Total scam company" | R1.04 | -- | V- | I3 | CR-N |
| "charged me twice" | J3.02 | -- | V- | I2 | CR-N |
| "refused to refund" | R3.03 | V2.04 | V- | I2 | CR-N |
| "This is robbery" | V4.01 | -- | V- | I3 | CR-N |
### Example 7: Accessibility Issue
> "No wheelchair ramp and the staff didn't seem to care."
| Span | Primary | Secondary | V | I | S | A | T | E | CR |
|------|---------|-----------|---|---|---|---|---|---|-----|
| "No wheelchair ramp" | A2.01 | -- | V- | I2 | S3 | A3 | TC | ES | CR-N |
| "staff didn't seem to care" | P1.03 | -- | V- | I2 | S1 | A2 | TC | EI | CR-N |
### Example 8: Digital Environment
> "App is beautiful but crashes constantly. Finally works on mobile though!"
| Span | Primary | Secondary | V | I | CR |
|------|---------|-----------|---|---|-----|
| "App is beautiful" | E2.01 | -- | V+ | I2 | CR-N |
| "crashes constantly" | E2.02 | -- | V- | I3 | CR-N |
| "Finally works on mobile" | E2.05 | -- | V+ | I2 | **CR-B** |
### Example 9: Cause-Effect (Single Span)
> "Support was painfully slow because their system kept going down."
| Span | Primary | Secondary | V | I | Note |
|------|---------|-----------|---|---|------|
| Full span | J1.03 | J3.03 | V- | I3 | Impact (slow) + cause (system down) |
### Example 10: Multi-Domain Complex Review
> "Great location and the room was clean, but checkout took forever and they tried to charge me for a minibar I never touched. Classic hotel scam."
| Span | Primary | Secondary | V | I | CR | Note |
|------|---------|-----------|---|---|-----|------|
| "Great location" | A4.01 | -- | V+ | I2 | CR-N | Convenience |
| "room was clean" | E1.01 | -- | V+ | I2 | CR-N | Physical space |
| "checkout took forever" | J1.01 | -- | V- | I3 | CR-N | Wait time |
| "tried to charge me for minibar I never touched" | V1.04 | J3.02 | V- | I2 | CR-N | Hidden cost + accuracy |
| "Classic hotel scam" | R1.04 | -- | V- | I3 | CR-N | Character judgment |
---
## 10. USN Quick Reference
URT String Notation for compact tagging.
### Format
```
URT:{Profile}:{codes}:{Valence}{Intensity}[:{metadata}][:{causal}]
```
### Profiles
| Code | Profile | Code Tier |
|------|---------|-----------|
| L | Lite | Domain (O, P, J...) |
| C | Core | Category (O1, P2...) |
| S | Standard | Subcode (O1.01, P2.03...) |
| F | Full | Subcode + Causal |
### Quick Examples
| Review | USN |
|--------|-----|
| "Food was cold" (Lite) | `URT:L:O:-2` |
| "Food was cold" (Core) | `URT:C:O2:-2` |
| "Food was cold, like last time" (Standard) | `URT:S:O2.05:-2:22TR.ES.S` |
| "Rude staff and unfair policy" (Standard) | `URT:S:P1.02+V2.04:-3:22TC.ES.N` |
| "Much better now!" (Standard) | `URT:S:V4.01:+2:11TC.ES.B` |
### Metadata Encoding (Standard+)
```
{Specificity}{Actionability}{Temporal}.{Evidence}.{Comparative}
Example: 22TC.ES.N
^^ = S2, A2
^^ = TC (current)
^^ = ES (stated)
^ = CR-N (no comparison)
```
### Valence Symbols
| Symbol | Meaning |
|--------|---------|
| + | Positive |
| - | Negative |
| 0 | Neutral |
| +- | Mixed |
---
## Quick Checklist Before Submitting
1. [ ] **One primary code per span** (no exceptions)
2. [ ] **Max 2 secondary codes** (prevents over-tagging)
3. [ ] **Checked "Don't Confuse With"** in spec for ambiguous codes
4. [ ] **Used decision tree** for domain selection
5. [ ] **Split at domain changes** (different aspects = different spans)
6. [ ] **CR-N unless explicit comparison** (default is no comparison)
7. [ ] **No invented causes** (causal chain only when text supports)
8. [ ] **Verified intensity markers** (I3 requires emphatic language)
9. [ ] **Cause-effect kept together** (impact = primary, cause = secondary)
10. [ ] **V vs R correctly applied** (exchange = V, character = R)
---
## 11. Billing/Returns/Refunds Decision Table
Special guidance for common billing and refund scenarios.
| Feedback Type | Primary Code | Secondary | Rationale |
|---------------|--------------|-----------|-----------|
| "Returns process was easy" | J4.02 | -- | Resolution process ease |
| "Returns process was a nightmare" | J4.02 | -- | Resolution process friction |
| "Refund took too long" | J4.03 | -- | Resolution speed |
| "Refund policy is unfair" | V2.04 | -- | Terms fairness |
| "They refused to refund me" (trust framing) | R3.03 | V2.04 | Compensation failure + policy |
| "Wrong amount refunded" | J3.02 | V1.04 | Accuracy + hidden costs |
| "They honored the warranty" | R2.05 | -- | Guarantee honor |
| "Worthless guarantee" | R2.05 | -- | Guarantee failure |
| "Charged me twice" | J3.02 | -- | Process accuracy |
| "Hidden fees at checkout" | V1.04 | V2.02 | Hidden costs + fee disclosure |
---
## 12. Data Privacy/Security Decision Table
| Feedback Type | Primary Code | Secondary | Rationale |
|---------------|--------------|-----------|-----------|
| "Security incident occurred" | E4.03 | -- | Security breach |
| "I don't trust them with my data" | R1.03 or R1.04 | E4.03 | Trust + security |
| "Data breach notification" | E4.03 | R1.03 | Security + transparency |
| "Privacy policy concerns" | V2.04 | R1.04 | Terms + ethics |
| "Account was hacked" | E4.03 | -- | Security failure |
| "They sell my data" | R1.04 | V2.04 | Ethics + terms |
---
## 13. CR Trigger Word Reference
Quick lookup for Comparative Reference assignment.
### CR-B (Better) - Improvement Signals
| Trigger Phrase | Example |
|----------------|---------|
| "better now" | "Service is much better now" |
| "improved" | "Quality has improved" |
| "finally fixed" | "They finally fixed the app!" |
| "much faster/cleaner/friendlier than before" | "Much faster than my last visit" |
| "they've turned it around" | "They've really turned things around" |
| "glad they addressed it" | "Glad they addressed the issue" |
| "not like last time" (positive context) | "Not like last time - actually worked!" |
### CR-W (Worse) - Decline Signals
| Trigger Phrase | Example |
|----------------|---------|
| "worse now" | "Quality is worse now" |
| "used to be good" | "This used to be my favorite restaurant" |
| "has gone downhill" | "Really gone downhill lately" |
| "not what it used to be" | "Just not what it used to be" |
| "declining" | "Quality is declining" |
| "deteriorating" | "Service has been deteriorating" |
| "they've really fallen off" | "They've really fallen off" |
| "was great before" | "Was great before the renovation" |
| "no longer what it was" | "No longer the place I loved" |
### CR-S (Same) - Persistence Signals
| Trigger Phrase | Example |
|----------------|---------|
| "still" | "Service is still slow" |
| "as always" | "Great as always" |
| "same as before" | "Same problems as before" |
| "nothing has changed" | "Complained twice, nothing has changed" |
| "yet again" | "Wrong order yet again" |
| "once again" | "Once again disappointed" |
| "as usual" | "Late as usual" |
| "same old" | "Same old problems" |
### CR-N (None) - No Comparison
- Default for ~90% of spans
- First-time customer reviews
- Competitor comparisons ("better than Brand X")
- No temporal language present
---
## 14. Causal Chain Quick Guide (Full Profile Only)
Only assign causal codes when text **explicitly states** or **logically entails** the cause. When in doubt, omit.
### Conditions Layer (CD-)
| Code | Name | When to Use | Example Text |
|------|------|-------------|--------------|
| CD-S | Staff State | Fatigue, training, motivation stated | "Server seemed exhausted" |
| CD-T | Team Dynamics | Handoffs, coordination issues stated | "They kept passing me around" |
| CD-E | Equipment | Malfunction stated | "The machine was broken" |
| CD-F | Facility | Maintenance, capacity stated | "Place was falling apart" |
| CD-O | Operational | Understaffing, demand surge stated | "Clearly understaffed" |
### Management Layer (MG-)
| Code | Name | When to Use | Example Text |
|------|------|-------------|--------------|
| MG-P | Planning | Scheduling, forecasting failure stated | "No one scheduled for weekend" |
| MG-T | Training | Training gap stated | "Clearly not trained properly" |
| MG-O | Oversight | Supervision failure inferable | "Third time reporting this" (EI) |
| MG-R | Resources | Resource allocation stated | "No supplies available" |
| MG-C | Communication | Policy relay failure stated | "Staff didn't know the policy" |
### Systemic Layer (SY-)
| Code | Name | When to Use | Example Text |
|------|------|-------------|--------------|
| SY-R | Resource Decisions | Budget, investment stated | "Cheap owners" |
| SY-P | Policy/Procedure | Bureaucracy, rules stated | "Company policy is the problem" |
| SY-C | Culture | Values, priorities stated | "Management doesn't care" |
| SY-S | Standards | Quality thresholds stated | "They've lowered their standards" |
| SY-H | Human Capital | Compensation, hiring stated | "Can't keep good employees" |
| SY-X | External | Market, regulatory stated | "Ever since COVID" |
### Causal Chain Validity Test
| Review Text | Causal Code? | Valid? | Why |
|-------------|--------------|--------|-----|
| "Long wait because they were short-staffed" | CD-O | Yes | Explicitly stated |
| "The machine was broken" | CD-E | Yes | Explicitly stated |
| "Fifth time I've reported this" | MG-O | Yes | Logically entailed (EI) |
| "They seem understaffed" | -- | No | "Seem" = speculation |
| "Probably a training issue" | -- | No | Annotator speculation |
| "Management doesn't care" (stated by customer) | SY-C | Yes | Customer explicitly states |
---
## 15. Inter-Annotator Agreement Rules
### Mandatory Conventions
1. **One primary code per span** - no exceptions
2. **Max 2 secondary codes** - prevents over-tagging
3. **Max 3 spans per sentence** - if more, reconsider boundaries
4. **Prefer splitting over V+-** - unless truly inseparable
5. **Check "Don't Confuse With"** - before finalizing subcode
6. **Evidence type required** - must justify EI inferences
7. **No invented causes** - causal_chain only when text supports
8. **CR-N is default** - only use CR-B/W/S when comparison is explicit
### Tie-Break Rules
When unsure between two codes:
| Disagreement Type | Resolution |
|-------------------|------------|
| Different domains | Use decision tree; first "YES" wins |
| Same category, different subcodes | Prefer more specific subcode |
| Different intensity | Defer to linguistic markers (see I3 markers) |
| Different actionability | A3 requires specific implementable action |
| Different CR | CR-N unless explicit temporal language present |
### Calibration Check Questions
Before submitting, ask yourself:
1. Would another annotator choose the same primary domain?
2. Is the secondary code genuinely distinct (different category)?
3. Is intensity justified by linguistic markers?
4. Is CR assignment supported by explicit comparison language?
5. Would I assign causal codes if I didn't know the industry?
---
## Where to Find More
| Document | Purpose |
|----------|---------|
| **URT-Specification-v5.1.md** | Full specification with all 140 subcodes |
| **URT Issue Lifecycle Framework** | Analytics layer for resolution tracking |
| **URT QA Protocol** | Calibration and audit procedures |
---
## Appendix: Profile Requirements Summary
### URT-Lite (Micro-business, quick triage)
```
Required: primary_code (domain), valence
Optional: intensity
Forbidden: secondary_codes, subcodes, S/A/E/CR/causal
```
### URT-Core (Small business, dashboards)
```
Required: primary_code (category), valence, intensity
Optional: secondary_codes (max 2, categories only), temporal, comparative
Forbidden: subcodes, S/A/E, causal
```
### URT-Standard (Operations, routing, analytics)
```
Required: primary_code (subcode), valence, intensity, specificity,
actionability, temporal, evidence, comparative
Optional: secondary_codes (max 2, subcodes only), causal_chain, confidence
Forbidden: domains, categories as primary
```
### URT-Full (Enterprise, root cause, audit)
```
Required: All Standard fields + causal_chain (when evidence supports),
linked_spans, confidence, annotator_notes
Optional: None - all fields should be completed
Forbidden: domains, categories as primary
```
---
*URT v5.1 Annotator Quickstart Guide | Track A: Training Materials*
Reference in New Issue View Git Blame Copy Permalink