How do you test an LLM before it goes live in a regulated business?
Test a language model the way an auditor will question it: on your own cases, against written pass criteria, with attacks on its permissions and data, and with a record of every result. RBI's Governor has asked banks to red-team and stress-test AI before deployment and periodically after; five test sets are where that starts.
Intelligent systems2 min read
Testing a large language model for a regulated business means building a fixed set of your own cases with known right answers, scoring the model against written pass criteria, attacking it the way a misuse would, and keeping the results so you can show them later. A demo that goes well is not a test. The same prompt can produce a different answer tomorrow, so a single good run proves little.
Write the pass criteria before testing
Agree with the business owner and risk what counts as a pass for each task, in writing. For a customer reply it might be: correct facts from the policy, no promise the bank cannot keep, the right disclosure, the right language. For a document extraction it might be field-level accuracy above a set level on each document type. Criteria written after the results tend to fit the results.
Five test sets
- 01Typical cases drawn from real past work, labeled by the people who do it, covering every product and language the system will see.
- 02Hard cases: rare products, incomplete documents, ambiguous requests, the ones staff escalate today.
- 03Cases where the right answer is to refuse or hand over to a person, such as legal threats, complaints and questions outside scope.
- 04Attacks: prompts that try to make it ignore instructions, reveal other customers' data, act beyond its permissions or call tools it should not.
- 05Fairness checks: the same case with names, regions or languages changed, to see whether the answer changes when it should not.
Run every set on every change
A new model version, a revised prompt, a new document source or a vendor update can each change behavior. Run all five sets on every change and compare against the version in production. Keep a small set from each that runs automatically before any release, and the full set before a major one.
Test the system, not just the model
Many failures in production come from around the model: retrieval that returns an outdated policy, a tool that writes to the wrong record, a log that misses the prompt. Test end to end, through the real integration, with the same permissions the live system will have. Confirm that a person can see what the model was given, what it said and what it did.
Keep the evidence
Store each test run with the model version, prompts, data sources, scores and who signed off. When an auditor, a regulator or your own risk team asks how you know the system is safe to run, the answer is this record. Our December 2025 article, Evaluation harnesses for models you cannot roll back, covers how to keep these tests running after launch.
Questions
- How do you evaluate an LLM for a bank or insurer?
- Build fixed test sets from your own cases with known right answers, score the model against pass criteria agreed in writing, include attack and fairness tests, and keep every result.
- What should LLM red-teaming cover?
- Attempts to make the model ignore instructions, reveal other customers' data, act beyond its permissions or call tools it should not, tested through the real integration.
- How often should an LLM be retested?
- On every change: a new model version, a prompt revision, a new data source or a vendor update. Run a quick subset automatically and the full set before major releases.
Sources
- RBI Governor Sanjay Malhotra Sets Out AI Playbook for Indian Banks, The Kanal, August 13, 2026
More on intelligent systems
All writingNovember 2026
CERT-In's AI security audit guidelines: what an auditor will check
CERT-In's AI Security Auditing Guidelines, dated November 4, 2026, sort AI systems into four risk tiers and audit them across nine domains. For AI agents, auditors will test whether an agent can exceed its permissions or bypass approval. Here is what to prepare.
October 2026
Multilingual voice AI for Indian call centers: a build checklist
A voice AI line in Indian languages succeeds when it resolves the calls that make up most of the volume, hands the rest to a person with context, and logs every call in the core systems. A build checklist, with results from a line we built that answers in 11 languages.
October 2026
What drives the cost of an enterprise AI project?
The model is rarely the biggest cost in an enterprise AI project. Integration with core systems, data work, controls and the cost of running it after launch usually decide the budget. Here is how to estimate each before you sign.
Tell us which system the AI has to run inside
If that system is not ready for AI yet, tell us that too. An engineer will reply and say plainly whether we are the right firm for it.