Trust-Calibrated Enterprise Service Copilots: Cost-Aware Agentic RAG for Customer-Support Knowledge Management

Abstract

Enterprise service copilots must retrieve relevant knowledge, provide traceable answers, control unsupported responses, and operate within practical latency and context budgets. This study evaluated these requirements on WixQA using the complete 6,221-article knowledge base, 6,221 Synthetic questions, 200 ExpertWritten questions, and 200 Simulated questions. The pipeline compared Okapi BM25, a 256-dimensional Dense-LSA retriever, reciprocal-rank fusion, a supervised logistic reranker, a top-1 article-match confidence gate, budget-aware stopping, and deterministic extractive evidence cards. Learned components used grouped five-fold out-of-fold evaluation on Synthetic, followed by frozen external evaluation. On the pooled external sets, Recall@5 was 0.4704 for BM25, 0.3704 for Dense-LSA, 0.4946 for hybrid retrieval, and 0.4838 after reranking. The adaptive policy answered 56.5% of ExpertWritten and 48.5% of Simulated requests; answered-case token F1 was 0.3051 and 0.2927, respectively. Median retrieved context fell to 71 and 63 analysis tokens. The Synthetic gate attained 0.0999 selective top-1 error at 0.8883 coverage, but frozen external error rose to 0.5664 and 0.6907. All tested financial scenarios produced negative ROI, although referral yielded the least negative base-case ratio. Hybrid retrieval improved external evidence recall, but synthetic confidence and benchmark efficiency did not support autonomous service. Representative calibration, explicit referral, and operational measurement remain necessary.