The situation
A growing SaaS company with customers in three languages and a two-person support team. Most tickets were “how do I” questions already answered in the documentation. The founders had tried a generic chatbot plugin; it answered confidently and wrongly, and was switched off within a month.
What we built
A support copilot embedded in the product. It retrieves from documentation, resolved tickets and release notes, answers with the source linked, and opens a ticket with the conversation attached when it cannot help. An evaluation suite of 400 real questions runs on every change, so accuracy is a number the team watches, not a feeling.
The retrieval layer respects plan tiers: users only see answers about features they have.
The result
Tickets reaching human agents fell by 55% in the first quarter. Accuracy on the evaluation set is 91% and tracked weekly. The support team now writes documentation for the gaps the copilot reports.
What mattered
The evaluation set was built before the model was chosen. Every architecture decision was made against measured accuracy and cost per answer, which is why the second attempt worked where the plugin did not.