Category: Systems & Operations

After I published Knowledge Is Cheap Now, Murray MacRae PhD asked me on LinkedIn how you identify an AI hallucination. My answer was quick, and I still think it was right: if you’re an expert in the subject you can see it, and if you’re not, you mostly can’t. His question stayed with me, and in the weeks that followed I ended up on both ends of it, and the uncomfortable end was the one where I didn’t see it at all.

The first time ran the other way round. On 9 September I was running an audit of the files that hold my writing system, 5 phases, one Claude model doing the evaluation and another carrying out the changes, and one finding said the Te Pūkenga restructure that ended my last role had been invented. It beat itself up about it, stating that my redundancy and my early retirement in November 2024 had been fabricated and hallucinated, and should come out. It only backed down when it went back to The Coach Said Ageism is a Myth, where I wrote about that restructure in June, and found the context it had been missing. I was made redundant. I retired early. I was there. Nobody else reading that audit would have known which way round the truth ran; I knew because it’s my life, and there’s no one more expert in it than me.

Consistent, and wrong

That audit checked my writing against what it had in front of it, and what it had in front of it was partial. I don’t write my Strategic Kata pieces in chronological order, some run ahead of when they happened and some reach back years, and the system that helps me edit holds its own selective memory of my career. Against that partial record, a restructure it couldn’t place looked invented. Internally consistent, externally wrong, and delivered with complete confidence.

OpenAI’s researchers published a paper in September 2025 on why language models hallucinate,¹ and their explanation is plainer than I expected. The models are trained and scored like students sitting a multiple-choice exam. If a model doesn’t know someone’s birthday, a guess has a 1 in 365 chance of being right, while “I don’t know” scores zero every time, so over thousands of questions the model that guesses looks better on the scoreboard than the one that admits it isn’t sure. So it guesses, and it guesses fluently. The way I read it, hallucination will always be there when the model hasn’t got the right context. I’m seeing less of it in my own work than when I started with AI a couple of years ago, and I’m not saying it’s gone, because September gave me three of them.

Some hallucinations you can see from across the room. When I trained Gemini to draw my comics, it kept giving mouths to characters who don’t have mouths. Anyone could spot that. The ones that cost you are written in well-formed, confident sentences about something you don’t know well enough to argue with, and that’s where Murray’s question bites.

Where I could see it

On my own ground I catch it, and I was catching the human version of it long before AI arrived. At IRD, BNZ and Stats NZ nearly everything ran on Microsoft, and I read the vendor reports, the release notes and the updates because that was part of the job. When a supplier brought in a product person from headquarters, flown in from Europe or Australia or the US to present the new release, they tended to assume the room knew less than they did. Often the room knew more. Some of my team had been running the beta for weeks because they loved to tinker, and they knew what the product actually did, so every now and then what the presenter claimed didn’t match what we had seen with our own hands. The presenter believed every word, and was fluent about a product they hadn’t used the way we had.

At ABN AMRO, when we moved into a new building, the data centre went into office space on the top three floors, and the project manager handed me the specification for the floor where the AS/400 and the other servers would sit. There was no raised floor in it. It had simply been left out, with no budget argument or short cut behind it. A raised floor is where the cooling air moves, where the power cables, data cables and ventilation run safely out of the way, and where the engineers get at all of it for maintenance, even on an office floor. I only saw what was missing because before ABN AMRO I had run a data centre at a local bank, in a different role. The raised floor went in, after some objection and a few rounds of discussion with leadership and finance. The specification had been wrong by omission, the same way AI can be: incomplete, with nothing on the page to tell you something is missing.

A couple of years ago I asked ChatGPT to help revise my CV. It inflated my percentages, it translated my New Zealand job titles into what they would mean in the US, and it turned work that was done locally, and done well, into global impact. I laughed at it, because it’s a very good wordsmith with a strong sense of marketing. I have never sent that CV to anyone. I couldn’t stand behind evidence it had made up.

On my own ground I can even see what I don’t know. In September, while I was drafting Does the Exit Interview Work? about our body corporate changing managers, Claude wrote that the outgoing manager ran 40 other buildings, and presented it as fact. I had no idea how many buildings they ran. It wasn’t my business, and they never told us. That’s how I knew the number was made up: I never had one to give, and nothing in the material said it. It came out before the piece went live.

Where I couldn’t

Now the other half. The same article had a second one in it, and that one I didn’t see. The evidence in it was a 2016 Harvard Business Review study by Everett Spain and Boris Groysberg.² Claude compressed the study’s method into “188 executives across 210 organisations”, which can’t be right, since you can’t spread 188 people across 210 organisations. They had surveyed 188 executives and interviewed 32 senior leaders, who between them represented 210. The draft also left out where the authors land: exit interviews can work if a second-line manager runs them. I knew that. I had said the same about the concept in my first blog post on exit interviews in 2022, and I part company with them only because in practice I have never seen one work. Claude reported the citation as verified. I read it, and I published it.

The next day a reader who knew the study wrote to me privately and told me it was wrong, twice. The correction sits at the foot of that piece, dated 24 September. I had read and scanned the study, and I never checked whether the numbers on my own page added up. The reader knew the study well enough to see they couldn’t. On that paper, I was the non-expert.

Big firms miss it too. Australia’s Department of Employment and Workplace Relations paid Deloitte AU$440,000 for a 237-page review of its welfare compliance system.³ The report cited academic work that doesn’t exist and a quote from a Federal Court judgment that isn’t in the judgment. Deloitte had used GPT-4o to help draft parts of it. A firm full of professionals put its name on it. Chris Rudge, a University of Sydney researcher in health and welfare law, read it and saw the problem, because he knew the literature it was citing. Deloitte refunded the final instalment of the contract in October 2025.

Closer to home, in February this year the New Zealand Supreme Court dealt with an applicant representing himself in a Family Court matter. His submissions cited cases that looked real: real case names, attached to citations that pointed somewhere else. The court said they appeared to have been hallucinated by an AI application,⁴ and warned that relying on false citations, including unverified AI output, may in serious cases amount to obstruction of justice or contempt of court. He had no lawyer, and little chance of seeing it, and the warning landed on him anyway.

What a non-expert can do

Back to Murray’s question. My first instinct, when I dictated the draft of this piece, was that a non-expert could paste the paragraph back into AI and ask whether it holds up. I have talked myself out of that. It’s like asking the con artist to tell you he’s honest, and if you can’t judge the original, you can’t judge the verdict either. It will catch some things, a dead link, a case that doesn’t exist at all. It’s much less reliable on a real study with its method bent out of shape, which is exactly the kind that got past me.

What caught all three was a person who knew the original well. My reader knew Spain and Groysberg’s numbers. Chris Rudge knew the work Deloitte claimed to be citing. The Supreme Court had the law reports. So this is what I do now when I’m outside my own ground. I open the source myself, the actual paper or judgment or release note the summary was made from, and check that the sentence carrying the weight says what I’ve been told it says. And when it matters, I borrow an expert, someone who has walked that ground, and ask them to read it before I rely on it.

Niklas Gruhn wrote a blog post in August about the meat proxy,⁵ the person who forwards whatever the AI produced without reading it, understanding it or checking it. The burden of verification sits with whoever puts their name on the output, and for Strategic Kata that’s me. My process already had checks in it: the Wispr Flow voice draft, the excavation, the back and forth, an audit against what I’ve published before and what’s online. It still let one through. So every citation now gets two rounds before it goes out: I open the link to see that it lands where it says it does, and I search the source for the passage I’m leaning on and read it there. It’s a quick read. It’s also the price of my name on the page.

Murray was right. I just didn’t expect the non-expert to be me.

1. Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang, Why Language Models Hallucinate, OpenAI, September 2025: https://arxiv.org/abs/2509.04664 ↩︎
2. Everett Spain and Boris Groysberg, “Making Exit Interviews Count”, Harvard Business Review, April 2016: https://hbr.org/2016/04/making-exit-interviews-count ↩︎
3. “Deloitte to partially refund Australian government for report with apparent AI-generated errors”, Associated Press via Fortune, 7 October 2025: https://fortune.com/2025/10/07/deloitte-ai-australia-government-report-hallucinations-technology-290000-refund ↩︎
4. Jones v Family Court at Whangārei [2026] NZSC 1, 11 February 2026, at [7]: https://www.courtsofnz.govt.nz/assets/cases/2026/2026-NZSC-1.pdf ↩︎
5. Niklas Gruhn, blog post on the meat proxy, 3 August 2026: https://gruhn.me/blog/2026-08-03/ ↩︎

On 22 July, at 10.44 in the morning, the hospital systems across the Waikato stopped answering. Waikato, Thames, Te Kūiti, Tokoroa and Taumarunui, five hospitals back on pen and paper for about 11 hours, with 68 appointments and 15 theatre cases deferred at Waikato Hospital alone. The cause was connectivity failure between its network and main data center,¹ which means the systems themselves were running perfectly well and nobody could reach them, and for some time afterwards Health NZ could say which part had failed without being able to say why.

I live in Hamilton, so this one happened close to home, and it took me straight back to Inland Revenue and the 33 priority one incidents we logged in 2013 on infrastructure IBM had stopped supporting. I’ve written about what the whiteboards did to that number, 33 to 7 to 3, in Two Kinds of Visibility, and about the 2am reboots we scheduled off our own data in Arcane by Necessity, and I promised in the first of those to come back one day to how we sweated those assets. This is that piece, and it turns on a tool I asked for and didn’t get.

The question I put to GKC

In 2014, our second year, we took a Splunk subscription. The New Zealand supplier was GKC, and David Anso and Glen Patrick spent a good deal of time with us, setting up the 45 inch monitoring TV our Director for IT Operations had bought, getting us through the training, teaching us to scrape and analyse the logs coming off the estate. What came out of it was a dashboard on that TV showing the WebSphere servers, the load, and whatever else we could feed it. Good piece of work, and genuinely useful.

The limit was data ingestion. Ingestion cost money, the budget didn’t stretch, so we were capped on how much we could take in at any one time, and when we hit the cap we culled what we were feeding in to protect the trends we cared about most. We were rationing our own visibility, which tells you something about the era we were operating in.

Somewhere during that training I asked the question that has stayed with me for 12 years. Why can’t we use this data to forecast the time to impact? The logs already hold the behaviour, we already know the shape of a server heading for trouble, so why can’t the system tell us what will fail, and when, before it does?

The answer was polite and it was correct. Great idea, they said, and it’s possible, but the technology and the algorithms are limited right now, and you’d have to supply the algorithms yourselves. Not one of us had a major in maths or statistics. Nobody in that team could have written them. So the idea died there, filed under too early, and we went back to doing the job by hand: benchmarks for when each server needed its cache cleared, thresholds watched by people, restarts scheduled for the hours the country was asleep.

Teru, my Senior IT Consultant, carried a spreadsheet that did the analytical half of this. Everything the tool would have done automatically, he did in columns.

Give 2014 the tool it asked for

The what if sets itself up. Drop today’s capability into that room. AI ingesting the logs without a cap, taking over Teru’s spreadsheet, doing the maths none of us could do, producing the sequence of failures instead of a picture of the present.

Splunk ships this now. IT Service Intelligence runs machine learning over historical service data to predict a service’s health score, and the horizon it offers is roughly 30 minutes, given a service with more than 5 good indicators and more than a week of history. The thing I sketched out loud in 2014 is a product feature in 2026. Hold onto that 30 minutes, because it matters more than it looks.

So: if we’d held that tool at Inland Revenue in 2014, would we have kept sweating those assets, and would the $1.5 billion transformation still have gone ahead?

We’d have sweated them longer

My answer surprised me, because I wanted the tool badly at the time and I assumed I’d say it would have made us braver about replacing things. I think the opposite.

Look at where we sat. A government department spends taxpayers’ money, and the obligation running underneath everything, the one my boss quoted whenever things went wrong, was the integrity of the tax system under the Tax Administration Act. Nowhere in that obligation does it say the servers have to be new. If the estate keeps running and the operating cost stays low, the return to the taxpayer improves, and a forecast that tells you precisely when each machine needs attention makes that arithmetic better. Sweating stops being a nervous gamble and becomes a scheduled, costed decision. We were already running a manual, low resolution version of exactly that, and holding 99.9 percent availability through tax peak season while we did it.

Which is worth putting next to what the transformation eventually reported: 7 of 10 measures achieved, digital uptake at 99 percent, and system availability of 99.9 percent.² The same availability number a team of 11 to 12 was already holding by hand, on kit two versions behind, three years before the programme started. Availability was never the argument for spending $1.5 billion. If forecasting had been the deciding variable, the case for replacement would have got weaker.

Better forecasting, longer sweating. That’s the half of this that’s true.

And the world wouldn’t have let us

Here’s the other half, and it took me some time to see it properly.

A forecast tells you when the thing you have will fail. It’s silent on when you have to replace it, because that clock sits outside your system, set by people who aren’t consulting your dashboard.

New Zealanders understand this through houses better than through servers. In Japan a house has an expected lifespan, and at the end of it you demolish and rebuild, so the clock is explicit. Here you maintain instead, and the maintenance isn’t optional, because the exterior paint goes, the piping goes, the wiring ages, the timber can rot, moisture gets in, appliances die, and the building code keeps moving underneath all of it, so what counted as a sound house 20 years ago needs work today to still count as one. The weather, the wear and the standard set that timing, and the house has no say in it.

Our estate had the same problem, and I could see it while we were congratulating ourselves on the reboot schedule. WebSphere was two versions behind and out of vendor support. We had an Enterprise Content Management System that had to be upgraded whether it was healthy or not, because without it the documentation and the correspondence Inland Revenue generates would eventually stop working with everything around it. Demand kept climbing, the agents’ bots hammered the portal harder every peak, and the expectations of what a tax administration should do online kept moving. None of that shows up on a failure curve. A perfect time-to-impact forecast would have told us the servers were fine until Thursday, and nothing at all about the debt we were building out of knowledge that lived only in the heads of the people running it.

That last part is the expensive bit. Sweat something long enough and the replacement bill is mostly the reconstruction of everything the old system knew and nobody wrote down. Transformation programmes are partly that invoice arriving.

The other failure mode

There’s a mirror version of this in the news here, and both belong in the same frame, because I don’t want to be read as saying that spending is the problem.

MBIE’s Biometric Capability Upgrade ran for seven years and was stopped in December 2025 having delivered nothing usable. By September 2026 the ministry had confirmed $50.7 million lost across it and a related project,³ and Parliament’s Privileges Committee had found that the ministry deliberately misled a select committee about it, a contempt of Parliament. Ministers in two governments were, on the Minister’s account, managed rather than informed, and that detail is the tell that this isn’t a story about one party or one minister.

The root of it looks structural to me. A programme of work that never fixes its answer to two questions, who is this for and what is it for, will drift for as long as the money lasts. Business process improvement, cost reduction, something the public actually needs: pick one and hold it. If the specification moves every time the market ships something newer, there’s no outcome to measure against, only a target that keeps walking away, and years pass, and everyone involved can say honestly that they worked hard the whole time.

So one organisation deferred replacement for a decade under an absolute obligation, spent $1.5 billion, and delivered most of what it promised. Another spent tens of millions chasing a capability and delivered nothing. Neither outcome was decided by how well anybody could forecast a failure.

What the 30 minutes tells you

Which brings me back to Splunk’s horizon.

30 minutes is an operational number and a good one. It’s enough to move traffic, warn the service desk, get somebody to a keyboard, take a controlled restart instead of wearing an uncontrolled outage. In 2014 it would have made us better operators, saved Teru some late nights, and possibly taken 3 priority ones down to 1.

The same 30 minutes is silent on whether to spend $1.5 billion, on when the vendor stops patching, on when the regulation changes, and on when the last person who understands the batch schedule retires. Nobody reading that dashboard learns anything about the decision that actually mattered.

I asked for the tool in 2014 because I thought better visibility would improve the decision. Twelve years on, I think it improves the operation, and the decision was always being made on a different set of facts, most of which sit outside the building.

What I do with my own

I should admit that I sweat my own assets thoroughly. I’m running a 10th generation i5 laptop well past the 3 years the industry likes to quote, and when I replace it I’ll be looking at a refurbished 13th generation i7 rather than a new machine, because the value is better and the dollar stretches further. There’s a 2012 Mac Mini in the house doing duty as a NAS under Linux, more than 10 years old and not complaining. I keep an old laptop purely because it has a DVD writer and I haven’t digitised the discs yet. Linux on hardware Windows has given up on buys years, and for the work I do the performance difference is nothing I can feel.

I can do all of that because when my laptop dies, the consequence is my own inconvenience. No tax return, no surgery, no statutory obligation. The moment other people depend on it, the calculation changes, and the calculation was never about the hardware.

Everything has a best before date and an expiry date, and the gap between them is where the sweating happens.

I’d still have taken the tool in 2014.

1. Waikato hospitals’ outage of 22 July 2026, its duration and the appointments and theatre cases deferred: RNZ, 23 July 2026, RNZ. ↩︎
2. Inland Revenue’s final report on its business transformation outcomes: 7 of 10 measures achieved, digital uptake 99%, availability 99.9%: Inland Revenue Annual Report 2023–24, Reseller News. ↩︎
3. Correction, 27 September 2026: the paragraph on MBIE has been corrected. The first version reported $33 million spent and a further $6 million surfacing afterwards. The cost is later confirmed by MBIE’s chief executive on 23 September. MBIE’s Biometric Capability Upgrade, the Privileges Committee’s finding of contempt, and the cost confirmed in September 2026: Privileges Committee report, 25 August 2026, MBIE, RNZ. ↩︎

Four times a year, when Stats NZ releases the CPI, I open an Excel spreadsheet and type the number in by hand. Unemployment when it comes out, the exchange rate, dairy prices, GDP, US inflation; each one found on a different website, copied across, checked against the previous quarter. I have kept this spreadsheet going for some time now, and an early-warning panel that only warns when I remember to feed it is not much of an early-warning panel.

Some context on why I bother. Years ago, while I was still at ABN AMRO, I took the Certified Financial Planner course and passed the certification, never to practise it in public; I learned it to manage my own money, and the knowledge served me well. Now, as an early retiree, that decision pays for itself, because it pays to manage your own portfolio well rather than paying someone else to do it, and I am running mine through decumulation, where the macro regime matters more than any single stock pick, because the regime decides how much risk I should be carrying at all. What I watch for are the three conditions that hurt a retiree’s portfolio in different ways: inflation, deflation, and stagflation. Over the years of manual tracking I settled on six indicators with hard triggers: NZ unemployment above 5.5%, US Core PCE inflation above 3.5%, the NZD/USD below 0.58, a 15% year-on-year drop in the Global Dairy Trade index, NZ CPI above 4%, and a negative quarter of NZ GDP. The signal I care about sits in the pattern across all six.

Some time back I tried to automate this myself in Python. I struggled, honestly. Part of it was the programming; the bigger part was discovering how many variables I hadn’t accounted for: which API carries which series, how each source dates its observations, what happens when a feed goes quiet. I gave up, and the spreadsheet won by default. Tedious, but it worked, in the way that typing numbers into cells every quarter “works.”

Then this year, with everyone talking about vibe coding, I decided the idea deserved a second attempt, this time with Claude Code doing the typing.

Under two hours

I wrote the request the way I would brief a contractor. The six indicators and their exact trigger thresholds. Two layers of logic: per-indicator status with a small history chart and a trend arrow, and above that a headline regime call (stagflation risk, inflation risk, deflation risk, watch, normal) driven by confirmed trends across three axes, where a confirmed trend means 3 consecutive quarters moving the same way. The hosting constraints: my WordPress site runs on Hostinger shared hosting, so the whole thing had to be one PHP script on a scheduled cron writing a data file, plus a static page reading it; no database, no server process, a folder I copy into public_html and it just works. I attached an image of how I wanted the dashboard laid out. And I defined done: the fetch runs end to end against live sources, the page renders against real data, the regime logic is proven by tests, and one data source gets deliberately broken to show the failure handling works before anyone tells me it’s finished.

I ran the prompt, revised it once, and end to end, from idea to a working dashboard tested against live data, we spent more than an hour but less than two. And the constraint was me. The machine was rarely waiting on itself; it was waiting on me, because at every fork I was validating and verifying with my own eyeballs before letting it move on. Two hours still surprises me when I type it, given what the same idea cost me in Python.

The two hours themselves are the part worth describing. Claude Code narrates as it works, a running commentary of what it’s checking and why, and some of it needs my answer, so I was reading the whole way through while keeping one eye on the token meter to see what all this thinking was costing me. And the commentary showed it thinking past the literal instruction. Before writing a line of code it verified every data source was actually reachable, because the whole architecture hinged on that. The Global Dairy Trade index has no public API and loads its numbers by JavaScript, so it read the site’s own app code, found the static JSON feed the charts call, and wired the scraper to that, built to shout if the feed’s shape ever changes rather than print a wrong number. When it found that the Stats NZ API needs a subscription key I hadn’t supplied, it declined to ship an untested adapter with guessed identifiers, saying plainly that claiming untested code works would be lying, and documented the upgrade path instead. It drew a distinction I hadn’t asked for, between a fetch that fails (mark the indicator Stale, keep the last good value, force the headline to Watch) and a source that succeeds but lags at origin (show the true data date and let me see the age). And at the end it broke the GDP feed on purpose, confirmed the Stale badge, the forced Watch and the error exit code all fired, then restored a clean run. The final test suite passed 23 out of 23.

Where we got stuck

One indicator refused to cooperate: NZ CPI. The US Federal Reserve’s FRED database carries nearly everything, including the NZ series via an OECD re-cut, and unemployment and GDP came through current. But FRED’s NZ CPI lagged 4 quarters behind, showing 2.5% for Q1 2025 when the real figure was 3.1% at Q1 2026. On a panel whose whole purpose is early warning about inflation, a year of lag on the inflation number is the one gap that matters; 3.1 sits a lot closer to my 4% trigger than 2.5 does.

So we hunted together. I suggested the Reserve Bank of New Zealand’s key indicators page, and we tested it in the browser side by side: the data is there, fresh, with history back to 1926. But the page sits behind F5 bot defence, so a PHP cron job gets served a JavaScript challenge instead of the numbers. Lovely data, unreachable by automation. Then I pointed at Stats NZ’s Infoshare, and it drove the site all the way to the export form before reporting back: the direct export needs a 42 KB form-state blob round-tripped on every call, an uploaded query file, and the full period list posted back, all behind Cloudflare. Technically scriptable, and it would snap the first time Stats NZ touched that page.

The irony is not lost on me. I worked at Stats NZ. Their numbers are the authoritative ones, and theirs was the single source we could not automate cleanly, because the proper API wants a subscription key and, worse, wants you to already know which of their scattered data products you’re requesting. That discovery problem put me off registering a key at all; it’s now a question I’m taking to my network contacts, to find the right API path for CPI and maybe GDP.

Where we landed is almost embarrassingly simple: a manual quarterly entry. CPI comes out 4 times a year, and each release I add one line to a config file, the same number I’ll be reading in the news anyway, while FRED’s older history fills in behind it for the chart. Mistype it and the card goes Stale, loudly. Thirty seconds a quarter, no key, no fragile scraper. For a quarterly series, that’s a better fit, and it replaced a spreadsheet’s worth of typing with one line.

What it actually felt like

The closest comparison I have is pair programming: one person at the keyboard, the other watching, questioning, supplying context the typist doesn’t have. I sat with 2 browser windows open, one reading the commentary and watching the tokens tick, the other checking the same data sources myself, verifying what it told me against what I could see. When it hit a fork it stopped, named the fork, and gave a recommendation; I made the call and it kept moving.

And the division of labour is what made 2 hours possible. The machine did the engineering I had already proven I couldn’t do alone; that Python attempt settled the question years ago. I did the parts it couldn’t: knowing that RBNZ and Infoshare existed as alternatives, judging that a manual entry is acceptable for a quarterly series, and above all knowing which six numbers matter and where the triggers sit, which came from the long stretch of doing the job by hand in Excel. The code took less than 2 hours. The specification took years of quarterly typing to earn. I suspect that ratio holds for most people trying this: the prompt worked on the first serious pass because I could define done precisely, and I could define done precisely because I had been the system I was replacing.

I briefed it like a contractor, with constraints and done-criteria. I attached a picture of the outcome. I stayed in the room the whole time, reading the commentary, because the questions it asked were exactly the ones worth answering well. And I checked its claims independently while it worked, the same courtesy I’d extend to any capable colleague I’d only just met.

The first real run of the finished dashboard came back Watch: dairy prices falling into rising US inflation, the dollar leaning on 0.58, nothing red on its own. Which is precisely the framing line I asked to be kept visible on the page: “None of these are red individually. Collectively, they paint a picture worth watching.”

The dashboard is now live beside Strategic Kata. The cron runs on the 1st and 15th of each month, timed to the Global Dairy Trade auctions, the fastest-moving of the six feeds; the quarterly series refresh as their releases land, and the one CPI line each quarter is the whole of my remaining manual labour.

The spreadsheet is retired. I guess that makes two of us.

At 2:00 am, just after the bots from the tax refund agents finished scraping the portal, we rebooted the servers.

The servers were healthy. That was the point. By the time an out-of-support WebSphere Application Server tells you it’s unhealthy, you’re already inside a priority one incident, and a system serving every taxpayer in New Zealand is falling over with the whole country watching. So we rebooted while the country slept, on a schedule built from our own data, and nobody outside the team ever knew it happened.

This was Inland Revenue between 2013 and 2015, and this piece is about how a team ran the machinery of the New Zealand tax system on technology officially past its life, with methods you could politely call arcane, because we had no other choice.

The Mandate

My boss that ran Business Platform Services was a tough fellow who made sure you knew what you signed up for. Hands off otherwise, which suited me fine. He had one famous line, delivered whenever shit hit the fan: “We are here to protect the integrity of the tax system. This is what you signed up for and what the Act says.”

The Act is the Tax Administration Act 1994, and he wasn’t quoting it for effect. The integrity of the tax system is a statutory obligation, and whenever a P1 landed, or the tax agents flooded us with complaints about the portal, he’d deliver the reminder again: you are responsible and accountable for the integrity of the tax system.

So the obligation was fixed. Absolute, written into law, indifferent to our circumstances.

The resources were fixed too, in the other direction. Inland Revenue was waiting for business transformation, and everything in the estate was being sweated to survive until the new world arrived. We were two versions behind. End of WebSphere Application Server 6.1 support from IBM due on September 2013. Aging hardware, aging tools, and no appetite to invest in a platform that transformation was going to retire anyway. My team, Business Platform Services Channels, owned the digital services end to end: online, B2B, voice, and output services. That was about 40 percent of Inland Revenue’s ICT applications, but 68 to 70 percent of all application interfaces flowed through Channel, because we were the layer between the customer and everything behind us in the service line: database, then middleware and application, then the mainframe at the back.

And the load had teeth. Tax peak ran from the end of March to August. Tax agents ran refund services that scraped our portal relentlessly, their bots hammering the front end through the night on behalf of clients chasing refunds. Every one of those requests travelled through infrastructure that its own vendor no longer supported.

When the obligation is absolute, the resources are frozen, and the load keeps rising, the only variable left is method.

Year One: Reactive

Our first year was fraught. When a WebSphere server hit its limit, the fastest recovery was a reboot, so we rebooted, sometimes in the middle of the business day. And sometimes the second server failed while the first was coming back, because it took the full load alone and buckled under it, which turned a recovery into a bigger incident than the one we started with. We were learning the hard way that on old infrastructure, the cure can cascade.

Late night calls were routine. Teru, my Senior IT Consultant, would get the pager, an impromptu meeting would form on the phone with the Incident Resolution Lead and whichever technical specialists the failing system had dragged out of bed, and the discussion usually ended the same way: a call to reboot. I made that call, every time. The responsibility and onus sat with me as the end-to-end service owner for digital services, and that’s what owning a service means at two in the morning.

We logged 33 priority one incidents that first year. I told the story of what the whiteboards and the visible metrics did about that number in Two Kinds of Visibility. This piece is the other half of how the numbers came down.

The Turn

We learned the behaviour of our servers the way you learn the behaviour of an old car: by paying attention to it for long enough. We collected data. We correlated system variables, CPU, cache, database log files, with the tax calendar and with application releases, and slowly the patterns showed themselves. It wasn’t rocket science. A server that buckles every time a certain filing date meets a certain cache level isn’t mysterious; it’s predictable, and predictable is workable.

We built benchmark numbers for when each server would need a reboot to clear its cache. And once we could forecast the failure, we stopped waiting for it. We scheduled the restarts for the hours when New Zealand was inactive and asleep. Dawn, just before people came to work. Or 2:00 am, just after the refund agents’ bots finished their scraping runs. The same reboot that had been a midday emergency in Year One became an invisible maintenance routine in Year Two, and that shift is a large part of why Year Two improved so dramatically on Year One.

The War Room

Through tax peak we ran a war room. Every morning, representatives from Channel, Database, Middleware and Application, Mainframe, and Service Management sat in one room, the entire service line from customer to mainframe, and I chaired the briefing.

The rhythm was fixed. Yesterday’s summary first: what we saw, what happened, what action was taken. Then each area updated the health of its systems and applications in turn. Then the tax calendar, so our activities synchronised with what the country was about to do to us. Then the threshold forecasts, which servers were approaching a reboot and when we’d take them. Then pending releases from Development and any fixes we had scheduled ourselves. Service Management kept us compliant with the ITIL process flows and coordinated the preparation whenever a service outage had to be planned for. And when something critical needed a decision on the spot, the BPS managers joined the room and the decision got made there and then, with everyone who’d have to live with it present.

Written down like that, it sounds almost formal. In practice it was a group of people running a national system on obsolete kit with a whiteboard, a pager, and a forecast, and everyone in the room knew exactly why. The Act didn’t care what version of WebSphere IBM still supported.

What Constraint Actually Does

Constraint tends to get either romanticised or pitied, and I think both readings miss what happens inside it. Nobody in that war room would have designed the operation this way. Given the choice, we’d have taken supported servers, modern tooling, proper capacity headroom, and full nights of sleep. We were never given the choice, and the obligation didn’t shrink to fit the resources, so ingenuity had to close the gap. The data collection, the correlation with the tax calendar, the benchmark thresholds, the dawn reboots: every one of those practices exists because a gap existed that money wasn’t going to close.

That’s usually what arcane methods are, when you stumble across them in an organisation and wonder why on earth things are done that way. They’re the archaeology of some past gap between what had to be done and what was available to do it with. Before you rip one out, it’s worth finding the gap it was built to span, because the gap may still be there.

Business transformation did eventually arrive, years after I left in October 2015, and it retired the estate we’d been sweating. I doubt anyone still remembers the 2:00 am reboots, and that was the whole idea; the best nights were the ones nobody noticed.

It was what we signed up for.