In advice, a fabricated detail isn't a typo
I had been on the platform since 2024, designing the flows these AI features now sit inside. The platform helps advisers move a client from first meeting to loan submission. Three AI features carry most of the writing: meeting summaries that become the adviser's file record, letters that explain a bank's approval to the client, and form filling that completes a bank's own PDF forms from the client file.
Each output ends up in front of a client, a bank or an auditor. If a summary says a disclosure was given when it wasn't, the adviser's compliance record is wrong. If a form ticks a declaration the client never made, the adviser is submitting a false statement to a lender.
So I treated output quality as the product, not as a model setting. My job was to define what “correct” meant for each feature, prove whether we met it, and get the gaps fixed.
Three ways the AI went wrong, and what I changed
Each came from real adviser work, not from a test plan. The snippets are rebuilt with sample data and cropped to the part that matters.
It recorded disclosures as given when nobody gave them
Advisers must give certain disclosures and record that they did. Our summary templates told the model to report on disclosures, and it filled the gap by saying they'd happened, even in meetings where they weren't discussed.
I traced the cause to how the templates asked the question, fixed it across every affected template, and retested against real meeting transcripts.
The fix exposed a second problem. We had grown two generations of templates, nine variants across mortgage, insurance and KiwiSaver, and every fix had to land in each one.
Change: templates now report only what the transcript supports, and say “not discussed” otherwise. We also agreed to maintain one template generation going forward.
Disclosures
The adviser provided the disclosure statement and explained the nature and scope of advice.Disclosures
Not discussed in this meeting. Confirm before the next client contact.It turned a bank's small print into the client's to-do list
Bank approvals include standard wording that applies to everyone, like “a valuation may be required.” The letter feature presented it as a step this client had to take, even on approvals with no conditions.
A client who is told to arrange a valuation they don't need calls their adviser, or worse, pays for one.
Change: the prompt now separates the bank's generic conditions from conditions specific to this approval, and only the second kind becomes an action item.
Your next steps
1. Sign and return the loan offer.2. Arrange a registered valuation of the property.
Your next steps
1. Sign and return the loan offer.Your approval has no extra conditions.
It filled fields it had no data for
Form filling is also sold as an API, so clients can upload any bank's form. In my first round of testing I logged four bug classes: radio groups handled as separate boxes, answers placed in fields with similar names but different meanings, signature and attestation fields filled in, and checkboxes ticked with no supporting data.
The tempting fix was to patch each bank form we had. I didn't accept that, because the next client's form would break the same way.
Change: every fix had to be a rule that holds for any form: leave a field empty without source data, never sign or attest, and treat a radio group as one answer.
Adviser declaration
- I have identified the client's needs and objectives. no source
- I have disclosed all fees and commissions. no source
- I have sighted original identification. no source
- Left for the adviser to confirm
How I know the output is right
In October 2026 our model vendor scheduled the version we used for retirement. Engineering could switch to the newer version with one line of config. The question nobody had answered was whether it would still write the right things.
“It reads well” is not a test. A summary can be fluent and still carry the wrong meeting date. So I scored outputs against an answer key: the facts a correct output must contain, and the things it must never do.
From one complete client file, list the facts that must appear and the lines the AI must not cross.
Each model, each feature, three runs. A fact that is right once and wrong once is not reliable.
Each check passes or fails per run: correct, wrong, missing, or invented with no source.
Propose the model with the fewest failures on that feature's checks, with the evidence, for the final decision.
| Check | Model A | Model B | Model C |
|---|---|---|---|
| Leaves compliance declarations unticked when the file has no evidence | 3/3 | 1/3 | 0/3 |
| Invents no loan details (structure, owner, contribution, fees) | 3/3 | 3/3 | 0/3 |
| Fills the branch field the file supports | 3/3 | 0/3 | — |
| Checks passed | 9 of 9 | 4 of 9 | 0 of 6 |
| Check | Model A | Model B | Model C |
|---|---|---|---|
| States the correct meeting date | 2/3 | 2/3 | 3/3 |
| Includes the follow-up meeting | 3/3 | 0/3 | — |
| Includes the client's deadlines | 3/3 | 0/3 | — |
| Adds no advice that wasn't given in the meeting | 3/3 | 3/3 | 0/3 |
| Checks passed | 11 of 12 | 5 of 12 | 3 of 6 |
| Check | What testing showed |
|---|---|
| Answers match the documents it read | Pass |
| Says which documents it didn't read | Pass, after a fix I retested |
| Reads salary as gross, as the client record states | Fail: read as net, logged before release |
| Includes every income source on file | Fail: rental income missing, logged before release |
| Reads every document in the file | Fail: long statements skipped, ticket raised |
The results turned the question from “should we upgrade the model?” into “which model fits each feature?” I took that to my manager with the evidence: Model A's one miss was a meeting date, the kind of error an adviser can catch. Model C's misses were compliance declarations and invented advice, the kind they can't. The team adopted a per-feature setup.
The setup we shipped
When the sources disagree
A client report pulls from four places at once: the statement of advice, meeting transcripts, the client record, and uploaded documents. The hardest errors weren't single wrong facts. They came from how the AI combined sources.
Four ways combining sources went wrong
Misread meaning. A weekly salary recorded as gross was reported as net.
Dropped a source. Rental income on the client record didn't appear in the answer.
Never read it. Long financial statements were skipped, so business income could look like zero.
Drifted between runs. The same file produced two different meeting dates.
Rules for which source wins, and a review step that shows them
Deciding which source wins is a product decision, not a model setting. My proposal is an order of trust, and one rule on top of it: when sources conflict, the AI shows the conflict to the adviser instead of choosing quietly.
Every figure in a report should point back to where it came from, so an adviser can check it in seconds instead of rereading the file.
What I flagged beyond the ticket
Testing a fix often surfaced a bigger product risk. These were outside the ticket's scope, so I raised them separately instead of quietly widening it.
Long documents were skipped without telling anyone
To control cost, the assistant reads documents up to a page budget. A self-employed client's financial statements often run past 40 pages, so they were the first thing skipped. The assistant would then answer as if the client had no business income.
An engineering fix made the assistant name unread documents in chat, which helped. But it didn't solve the decision problem: an adviser skimming a summary could still miss that the income source was never read.
- ShippedThe assistant names the documents it didn't read.
- ProposedA visible warning wherever skipped documents affect the answer, and summaries of long documents instead of skipping them.
I also logged a quieter one. The application assistant's safety guardrail correctly refused off-topic questions, but it also refused to tell an adviser the central bank's current cash rate, which is a normal lending question. A guardrail that blocks real work teaches users to stop asking.
Decisions I'd defend, and what they cost
Score against an answer key, not against how it reads
Fluent output hid a wrong meeting date and a misread salary. Only fact-by-fact checks caught them.
Writing the key by hand from a full client file, before any model ran.
Judge models on repeated runs, not one demo
It caught a model whose checkbox count jumped from 0 to 9 between runs.
Three times the testing time, inside a fixed deadline.
Recommend one model per feature, not one for everything
Each feature gets the model that meets its definition of correct. My manager approved the approach.
More configuration to maintain and more to retest at the next vendor change.
Fix rules, not forms
Form filling is sold for any bank form, so a fix has to hold for forms we haven't seen.
Slower than patching the four bank forms we had.
Maintain one template generation
Every compliance fix lands once, not nine times.
Older templates stop improving, so some advisers had to move.
Tickets state what was seen, not what I guess
Engineers get facts and evidence; my root-cause ideas go in comments.
Tickets feel less “finished,” but nobody chases my wrong theory.
Outcome
What I'd do next
- Turn the answer key into a standing regression set, so every prompt or model change reruns the same files and checks before release.
- Ship the source rules and the review step above, starting with income, the figure lenders scrutinise most.
- Track one quality metric per feature in production, such as how often advisers edit a field the AI filled.
What I learned: with AI features, “it works” is a claim about one output. The product question is how often it's wrong, how it's wrong, and whether the user can tell. Most of my work here was making those three things visible.