B2B SaaS / Customer Success · AI/ML Engineering · BUILDSTR
Challenge
Find the right human when AI runs out
AI now handles a lot of tier-one support for companies, but the 25–40% of tickets it can’t answer are where things go wrong. At that point, the AI either invents an answer, dead-ends the customer, or hands the ticket to whoever is next in line with no context. Culturora is building the intelligence layer for that handoff, for B2B SaaS companies of 200–2,000 people, sold to the VP of Customer Success. Culturora believes that the answer is already in the ticket data: who closes billing disputes cleanly, whose fixes hold, and who gets work escalated to them versus who escalates it away.
- Escalations went to whoever was next in line, not whoever knew the answer
- Knowing who knows what, and how to approach them, lived in team leads’ heads and walked out the door with them
- Surveys and self-reported skill profiles go stale and rarely capture this honestly
- The key question: could ordinary ticket history alone, with no surveys or skills forms, produce a ranking worth trusting?
Solution
Infer expertise from the tickets themselves
BUILDSTR built an AWS-native system that reads closed-ticket history, builds a profile for each support engineer, and recommends the best available expert for each new ticket. It works in three stages: build profiles, route tickets, learn from outcomes. A coordinator agent on AgentCore Runtime, built with the Strands SDK, handles the language work. Everything numeric happens in code.
- Build profiles: the agent reads one engineer’s de-identified ticket history and infers their topics, helper archetype, and communication style. Each result is scored against a withheld answer key and only saved if it passes a 60% agreement gate.
- Score in code: expertise per topic combines four signals from ordinary ticket fields: volume, quality (CSAT and reopens), speed, and escalation patterns. The model classifies the ticket and writes the rationale but never produces a number, so every ranking can be explained.
- Route: a new ticket returns the top three experts ranked by fit, which weighs expertise, topic match, and current workload. Each comes with its supporting evidence and plain-English advice on how to approach that person.
- Respect capacity: load-aware routing in Amazon DynamoDB automatically removes overloaded experts from the ranking
- Learn: CSAT, handle time, and resolved/reopened/escalated outcomes feed back into each expert’s scores, so profiles stay current
- Remember: AgentCore Memory holds agent context across sessions
- Test fairly: five Bedrock models ran head to head on identical data and prompts: 320 routing runs plus 35 profile derivations, 355 live calls with zero failures and zero invented candidates
- Hand over: a three-tab demo UI (Derive Profile, Expert Graph, Route Ticket), two AWS CDK stacks, 192 automated tests, and an offline evaluation harness
Result
A model for each job, and a path to scale
The project validated the method end to end against 200 synthetic Freshdesk-style tickets across seven engineers, scored against a hidden answer key the models never saw. It also showed that routing and profiling need different models. Routing runs on every ticket, so speed and cost matter. Profiling runs once per expert, so judgment matters. Culturora left with a defensible model choice for each.
- Amazon Nova Pro for routing: 98.4% topic accuracy, 53% cheaper and 40% faster than the default model (Claude Haiku)
- Claude Sonnet for profiles: the only model to rebuild all seven experts accurately, blind
- Low run costs: routing at about $4 per 1,000 tickets on Nova Pro, and profiling all seven experts for under $1
- A sized path from about 50 AI-assisted sessions a day to a projected 13,900–37,350 at scale
- Proved the pattern and paved the path to bringing that same intelligence layer to Zendesk
