|
| 1 | +# Flexible Categorization Engine |
| 2 | + |
| 3 | +## The Problem |
| 4 | + |
| 5 | +All four DLQ commands (`dump-dlq`, `purge-dlq`, `resubmit-dlq`, `diagnose-dlq`) group messages into categories for |
| 6 | +interactive selection. Previously, categorization was hardcoded to two properties: `Subject` (Label) and |
| 7 | +`DeadLetterReason`. This was baked into every layer — from `DlqCategoryKey(string Label, string DeadLetterReason)` |
| 8 | +through the scan session, display table, merge algorithm, and message filtering. |
| 9 | + |
| 10 | +This worked for simple cases: |
| 11 | + |
| 12 | +```text |
| 13 | +╭───┬──────────────────┬──────────────────────────┬───────╮ |
| 14 | +│ # │ Label │ DeadLetterReason │ Count │ |
| 15 | +├───┼──────────────────┼──────────────────────────┼───────┤ |
| 16 | +│ 1 │ OrderProcessor │ MaxDeliveryCountExceeded │ 847 │ |
| 17 | +│ 2 │ PaymentHandler │ TTLExpiredException │ 412 │ |
| 18 | +╰───┴──────────────────┴──────────────────────────┴───────╯ |
| 19 | +``` |
| 20 | + |
| 21 | +But real-world Service Bus usage often demands different grouping strategies: |
| 22 | + |
| 23 | +- **Group by error code in the message body** — when `Subject` is generic but the JSON payload contains a structured |
| 24 | + `errorCode` field. |
| 25 | +- **Group by deployment context** — when messages carry `environment` or `region` metadata in their body that matters |
| 26 | + more than the dead-letter reason. |
| 27 | +- **Group by custom application headers** — when teams set application-specific properties like `TenantId` or |
| 28 | + `ProcessorVersion` on messages. |
| 29 | +- **Single-dimension grouping** — sometimes only `DeadLetterReason` matters, and `Subject` just adds noise. |
| 30 | + |
| 31 | +With hardcoded categorization, none of this was possible without modifying source code. |
| 32 | + |
| 33 | +## The Solution: `--categorize-by` |
| 34 | + |
| 35 | +The `--categorize-by` option lets users define which properties form the category dimensions, using a prefix syntax: |
| 36 | + |
| 37 | +| Prefix | Source | Example | |
| 38 | +|-----------------|------------------------------------------------|-------------------------------------------------| |
| 39 | +| `#PropertyName` | System property on `ServiceBusReceivedMessage` | `#Subject`, `#DeadLetterReason`, `#ContentType` | |
| 40 | +| `$PropertyName` | JSON body property (deserialized) | `$errorCode`, `$tier` | |
| 41 | +| `$Nested.Path` | Nested JSON body property via dot notation | `$error.severity`, `$context.region` | |
| 42 | + |
| 43 | +```bash |
| 44 | +# Default (backward-compatible, same as before) |
| 45 | +dump-dlq -n ns.servicebus.windows.net -q myqueue -i |
| 46 | + |
| 47 | +# Single dimension: just the dead-letter reason |
| 48 | +dump-dlq -n ns.servicebus.windows.net -q myqueue -i --categorize-by "#DeadLetterReason" |
| 49 | + |
| 50 | +# Mixed: system property + JSON body property |
| 51 | +dump-dlq -n ns.servicebus.windows.net -q myqueue -i --categorize-by "#DeadLetterReason,$errorCode" |
| 52 | + |
| 53 | +# Nested body property |
| 54 | +dump-dlq -n ns.servicebus.windows.net -q myqueue -i --categorize-by "#Subject,$error.severity" |
| 55 | + |
| 56 | +# Three dimensions |
| 57 | +dump-dlq -n ns.servicebus.windows.net -q myqueue -i --categorize-by "$tier,#Subject,#DeadLetterReason" |
| 58 | +``` |
| 59 | + |
| 60 | +The table headers update dynamically: |
| 61 | + |
| 62 | +```text |
| 63 | +╭───┬──────────────────────────┬─────────────────┬───────╮ |
| 64 | +│ # │ #DeadLetterReason │ $error.severity │ Count │ |
| 65 | +├───┼──────────────────────────┼─────────────────┼───────┤ |
| 66 | +│ 1 │ MaxDeliveryCountExceeded │ critical │ 312 │ |
| 67 | +│ 2 │ MaxDeliveryCountExceeded │ warning │ 198 │ |
| 68 | +│ 3 │ TTLExpiredException │ info │ 47 │ |
| 69 | +╰───┴──────────────────────────┴─────────────────┴───────╯ |
| 70 | +``` |
| 71 | + |
| 72 | +## Architecture |
| 73 | + |
| 74 | +### Core Types |
| 75 | + |
| 76 | +The engine is built on three new types in `Application/DeadLetters/Common/`: |
| 77 | + |
| 78 | +**`CategoryPropertyRef`** — A parsed reference to a single property. Knows whether it's a system or body property and |
| 79 | +carries the dot-separated path. |
| 80 | + |
| 81 | +```csharp |
| 82 | +public enum PropertySource { System, Body } |
| 83 | + |
| 84 | +public sealed record CategoryPropertyRef(PropertySource Source, string PropertyPath) |
| 85 | +{ |
| 86 | + public string DisplayName => Source == PropertySource.System |
| 87 | + ? $"#{PropertyPath}" : $"${PropertyPath}"; |
| 88 | + |
| 89 | + public static CategoryPropertyRef Parse(string reference); |
| 90 | + // "#Subject" → (System, "Subject") |
| 91 | + // "$error.code" → (Body, "error.code") |
| 92 | +} |
| 93 | +``` |
| 94 | + |
| 95 | +**`CategorizationSchema`** — An ordered list of property references that defines the categorization dimensions. Provides |
| 96 | +a static `Default` that preserves backward compatibility. |
| 97 | + |
| 98 | +```csharp |
| 99 | +public sealed class CategorizationSchema |
| 100 | +{ |
| 101 | + public static readonly CategorizationSchema Default = new([ |
| 102 | + new(PropertySource.System, "Subject"), |
| 103 | + new(PropertySource.System, "DeadLetterReason") |
| 104 | + ]); |
| 105 | + |
| 106 | + public IReadOnlyList<CategoryPropertyRef> Properties { get; } |
| 107 | + public int DimensionCount => Properties.Count; |
| 108 | + public bool UsesBodyProperties { get; } // cached flag for optimization |
| 109 | +
|
| 110 | + public static CategorizationSchema Parse(IEnumerable<string>? references); |
| 111 | + // null/empty → Default |
| 112 | +} |
| 113 | +``` |
| 114 | + |
| 115 | +**`CategoryPropertyResolver`** — Resolves a `CategoryPropertyRef` against a `ServiceBusReceivedMessage` to produce a |
| 116 | +string value. Handles system property dispatch, JSON body deserialization with caching, and dot-path navigation. |
| 117 | + |
| 118 | +### From 2D to N-dimensional |
| 119 | + |
| 120 | +The key structural change was evolving `DlqCategoryKey` from a two-field record to an N-dimensional key: |
| 121 | + |
| 122 | +```text |
| 123 | +Before: sealed record DlqCategoryKey(string Label, string DeadLetterReason) |
| 124 | +After: sealed class DlqCategoryKey(ImmutableArray<string> Values) + IEquatable |
| 125 | +``` |
| 126 | + |
| 127 | +The same transformation applied to `DlqCategory`. Both types retain backward-compatible `Label` and `DeadLetterReason` |
| 128 | +convenience properties that index into `Values[0]` and `Values[1]`, so existing code that only uses the default schema |
| 129 | +continues to work unchanged. |
| 130 | + |
| 131 | +Custom `IEquatable<DlqCategoryKey>` and `GetHashCode()` implementations were necessary because `ImmutableArray<T>` does |
| 132 | +not provide structural equality — the default record equality would compare by reference, breaking dictionary lookups |
| 133 | +and grouping. |
| 134 | + |
| 135 | +### Property Resolution |
| 136 | + |
| 137 | +System properties resolve via a switch expression over known `ServiceBusReceivedMessage` property names: |
| 138 | + |
| 139 | +```text |
| 140 | +Subject, DeadLetterReason, ContentType, CorrelationId, |
| 141 | +MessageId, SessionId, ReplyTo, To, DeadLetterErrorDescription |
| 142 | +``` |
| 143 | + |
| 144 | +Unrecognized names fall through to `message.ApplicationProperties`, enabling categorization by custom headers without a |
| 145 | +special syntax. If nothing matches, the value is `"(none)"`. |
| 146 | + |
| 147 | +Body properties use the existing `MessageBodyDecoder.Decode()` to get a `JsonNode`, then navigate the dot-separated path |
| 148 | +segment by segment. A `ConcurrentDictionary<long, JsonNode?>` keyed by `SequenceNumber` caches decoded bodies — |
| 149 | +important because the reactive scanning architecture rebuilds category snapshots every second from the same cached |
| 150 | +messages. |
| 151 | + |
| 152 | +### Integration with Existing Features |
| 153 | + |
| 154 | +**`--merge-similar`** — The LCS-based category merger was generalized from 2 hardcoded dimensions (label frame + reason |
| 155 | +frame) to N dimensions. The `TokenizedCategory` type changed from `(string[] LabelTokens, string[] ReasonTokens)` to |
| 156 | +`string[][] DimensionTokens`. Scoring computes per-dimension LCS scores and requires all to meet the 0.5 threshold. The |
| 157 | +core LCS, scoring, and template rendering algorithms are unchanged. |
| 158 | + |
| 159 | +**Reactive scanning** — The `StreamDlq` command, `DlqScanSession`, and `DlqCategoryScanner` all accept optional |
| 160 | +`CategorizationSchema` and `CategoryPropertyResolver` parameters. When omitted, they fall back to the default schema. |
| 161 | +The resolver's body cache integrates naturally with the reactive architecture — bodies are decoded once and reused |
| 162 | +across snapshot rebuilds. |
| 163 | + |
| 164 | +**Interactive display** — `DlqCategoryDisplay.GenerateTableData()` generates column headers dynamically from |
| 165 | +`schema.Properties.Select(p => p.DisplayName)` instead of hardcoded `"Label"` / `"DeadLetterReason"` strings. The table |
| 166 | +adapts to any number of dimensions. |
| 167 | + |
| 168 | +## Data Flow |
| 169 | + |
| 170 | +```text |
| 171 | +CLI option Application layer Display |
| 172 | +──────────── ───────────────── ─────── |
| 173 | +
|
| 174 | +--categorize-by CategorizationSchema.Parse() |
| 175 | +"#DeadLetterReason,$tier" → Schema { Properties: [ → Table headers: |
| 176 | + (System, "DeadLetterReason"), "#DeadLetterReason", "$tier" |
| 177 | + (Body, "tier") |
| 178 | + ]} |
| 179 | + │ |
| 180 | + ▼ |
| 181 | + DlqCategoryKey.FromMessage() |
| 182 | + resolver.ResolveProperty(msg, prop) |
| 183 | + │ |
| 184 | + ▼ |
| 185 | + DlqCategoryKey(["MaxDelivery..", "1"]) |
| 186 | + │ |
| 187 | + ▼ |
| 188 | + GroupBy key → DlqCategory(values, count) |
| 189 | + │ |
| 190 | + ▼ |
| 191 | + CategoryMerger.Merge() (if --merge-similar) |
| 192 | + │ |
| 193 | + ▼ |
| 194 | + Interactive selection → ExpandKeys → Filter |
| 195 | +``` |
| 196 | + |
| 197 | +## Design Decisions |
| 198 | + |
| 199 | +**Sealed class over record for `DlqCategoryKey`** — Records generate equality based on field values, but |
| 200 | +`ImmutableArray<T>` has reference equality semantics. A sealed class with explicit `IEquatable` implementation gives |
| 201 | +correct structural equality for dictionary keys and LINQ grouping. |
| 202 | + |
| 203 | +**ApplicationProperties fallback for `#` syntax** — Rather than requiring a separate prefix for custom headers, |
| 204 | +unrecognized `#PropertyName` values fall through to `message.ApplicationProperties`. This means `#Diagnostic-Id` |
| 205 | +resolves a custom header, while `#Subject` resolves the built-in property. One syntax covers both. |
| 206 | + |
| 207 | +**Body cache keyed by SequenceNumber** — Each message in a Service Bus peek has a unique, stable sequence number. Using |
| 208 | +this as the cache key (rather than MessageId) avoids issues with duplicate message IDs and aligns with how the reactive |
| 209 | +cache identifies messages. |
| 210 | + |
| 211 | +**`"(none)"` for unresolved values** — When a property doesn't exist on a message (wrong path, binary body, null value), |
| 212 | +the resolver returns `"(none)"` rather than throwing. This groups all unresolvable messages together in one category, |
| 213 | +which is the most useful behavior for interactive exploration. |
| 214 | + |
| 215 | +**Default schema for backward compatibility** — When `--categorize-by` is not specified, |
| 216 | +`CategorizationSchema.Parse(null)` returns `CategorizationSchema.Default` (`#Subject,#DeadLetterReason`). Every code |
| 217 | +path that previously hardcoded these two properties now passes `schema ?? CategorizationSchema.Default`, producing |
| 218 | +identical behavior. |
| 219 | + |
| 220 | +## Files |
| 221 | + |
| 222 | +3 new files in `Application/DeadLetters/Common/` (`CategoryPropertyRef`, `CategorizationSchema`, |
| 223 | +`CategoryPropertyResolver`), 6 modified core types (`DlqCategoryKey`, `DlqCategory`, `DlqCategorySnapshot`, |
| 224 | +`DlqCategoryScanner`, `DlqCategoryDisplay`, `CategoryMerger`), 6 modified infrastructure files (`DlqScanSession`, |
| 225 | +`DlqMessageService`, `StreamDlq`, `CategorySelection`, `DlqScanSessionExtensions`, `StreamDlqCategories`), and 8 CLI |
| 226 | +files across all 4 commands (CLI option + handler parse call each). |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?v=4&size=40){kind=avatar}
0 commit comments