Month-end variance commentary, automated

Sign. Variance = actual minus plan. For costs a positive variance is over plan (unfavorable) and a negative one is under plan (favorable).

Materiality. A variance is material when, for one department and report line, the month's variance is at least $50,000 and at least 5% of that month's plan, or the year-to-date variance is at least $150,000 and at least 3% of the year-to-date plan.

One-time, timing or persistent. Timing: this month the line has an entry that books something to another period and will reverse (an accrual adjustment for an earlier month, a reclass between expense and prepaid, or an invoice for a service period longer than a month posted straight to expense), or, for commissions, the month's accrual was calculated on new ARR more than 5% away from the new ARR booked in the CRM that month. One-time: not timing, and the line has an invoice this month (or, for a year-to-date variance, earlier this year) from a vendor not seen on this line before in the year, at least half the size of the variance and in the same direction. Persistent: everything else (headcount, a recurring vendor's rate, usage).

Reforecast. Persistent: this month's variance repeats in each remaining month of FY2027, so the full-year impact on cost is this month's variance times the number of months left in the year (6 after June, for example). One-time and timing: no further impact (zero) on the remaining months.

April, sales commissions. The month's ledger had a $72,000 commission accrual adjustment for three deals, with the entry type “accrual adjustment”. The model read it as a cost that will repeat, labelled the variance persistent and carried $544,730 into the rest of the year. Left in, that would have overstated FY2027 cost by the same amount.

Draft 1 (model call 7): Commissions are $68,091.30 over plan for April ($275,552.70 actual vs $207,461.40 plan, 32.8%), an unfavorable variance. The overage comes from a $72,000 commission accrual adjustment on SO-1187, SO-1191 and SO-1192, partly offset by the base accrual calculated on new ARR of $2,035,500 against $2,074,600 planned. The line is classified as persistent, with the adjustment's basis still to be confirmed by Sales Operations.

C6 label: labelled persistent, but the written rule says timing: the line has an entry that books to another period (E-00183) · C7 reforecast: reforecast impact $544,730, but the rule gives timing: no further impact, $0

Draft 2 (model call 8): Commissions are $68,091.30 over plan for April ($275,552.70 actual vs $207,461.40 plan, 32.8%), an unfavorable variance. The overage is driven by a $72,000.00 commission accrual adjustment on SO-1187, SO-1191 and SO-1192, while the base accrual of $203,552.70 was calculated on new ARR of $2,035,500, matching the CRM booking. The line is classified as timing, so the reforecast impact is zero, pending Sales Operations confirmation that the adjustment reverses.

Each check is also fed a known-bad draft and has to go red, and fed the phrases it once got wrong and has to stay quiet: 23 of 23 selftest cases behave as expected.

The checks only test what can be tested against the books. In the final run the score against the answer key found nothing they let through: every main cause and every second cause was named, and no cause was invented. The previous version of the test, in the call log, shows what they miss. Its May R&D commentary explained the shortfall by late hiring and left out the smaller cause on the same line, pay per head above plan after the April merit round; every number in it was right, so no check could object. Its low-effort run listed a half-percent difference in support pay per head among the reasons: a real figure, so it passed, but in the books it is noise.

That version also had a problem the pipeline created itself. Its prompt rounded the plan, the actual and the variance to the dollar one by one, so the figures it showed did not subtract ($275,553 − $207,461 is $68,092; the variance shown was $68,091). The fixed April draft then told Sales operations to resolve a $1 gap that is not in the books, and no check could see it, because the checks compare the text with the ledger, not with the prompt. The prompt now shows every figure to the cent, every draft was redone, and a ninth check sends back commentary in the pipeline's own words (that same next step had cited an entry's “accrual_adjustment type”).

So the checks hold the numbers, labels and reforecast to the ledger and the written rules; they do not judge what matters, and they cannot catch a mistake built into what the pipeline hands the model. That is why the score exists here, and why a real close still needs a person who knows the business to read the pack.

The pack comes from the command line's default setting, where the model reasons before it answers. The same 19 prompts were also run with --effort low, which skips that step and cost about 46% as much per call.

On this ledger the two settings ended up the same. Both made the same April mistake, labelling the commission catch-up persistent, and both fixed it when the checks sent it back; the low-effort draft also talked about the pipeline's “timing or one-time tests”, which the wording check caught. One run each is too few to say which setting is better in general.

Across the 142 model calls made for this project ($17.10 in all), earlier versions of the checks sent back 22 drafts that were right. The cause check read “the price held at $0.80”, “leavers matched plan” and “average pay per FTE was close to plan” as claims that those things caused the variance, read the law firm Marlow Price LLP as a claim about price, and read a sentence saying it was not yet known whether a change was rate or volume as a claim about volume. The number check took the “-31” out of a date, did not know the 5% threshold quoted from the written rule, and was stricter than the books on rounding. Every fix is now a selftest case, and the call log keeps each false alarm: the re-draft prompts quote the flags as they were sent.

The first version of the test was also too easy. It gave the model one ledger line at a time, with memos that spelled out the cause: that a contract had renewed at a new list rate, which deals the commission cut-off had left out. In the default run every first draft named the right cause, and across both settings the checks found nothing beyond a stray $1 rounding figure, so the test could not show anything. The second version gave it the department's whole month with short memos, the way a ledger looks, and mixed a second cause into three lines; the final version keeps that and fixes the rounding in the prompt described above. All three versions are in the call log.

Tillbrook is made up, and so is its ledger: the planted causes are cleaner than real ones, and the statistics a real team would have to chase are all at hand. The checks test what can be tested mechanically; they cannot tell whether a next step is a good one, or whether a cause the evidence supports is the main one. The score against the answer key is how this demo measures that, and a real close has no answer key. The reforecast rule is deliberately simple: a persistent variance repeats at this month's size. Revenue has no planted cause. One model and one run per setting: another run would draft differently, which is why every prompt and reply is logged.

Tillbrook Software is a made-up company. The pack's commentary was drafted by claude-haiku-5-5 between 2026-10-24 04:34 and 2026-10-24 04:41 UTC; those calls cost $3.23.