Author: Wayne Loke

At 2:00 am, just after the bots from the tax refund agents finished scraping the portal, we rebooted the servers.

The servers were healthy. That was the point. By the time an out-of-support WebSphere Application Server tells you it’s unhealthy, you’re already inside a priority one incident, and a system serving every taxpayer in New Zealand is falling over with the whole country watching. So we rebooted while the country slept, on a schedule built from our own data, and nobody outside the team ever knew it happened.

This was Inland Revenue between 2013 and 2015, and this piece is about how a team ran the machinery of the New Zealand tax system on technology officially past its life, with methods you could politely call arcane, because we had no other choice.

The Mandate

My boss that ran Business Platform Services was a tough fellow who made sure you knew what you signed up for. Hands off otherwise, which suited me fine. He had one famous line, delivered whenever shit hit the fan: “We are here to protect the integrity of the tax system. This is what you signed up for and what the Act says.”

The Act is the Tax Administration Act 1994, and he wasn’t quoting it for effect. The integrity of the tax system is a statutory obligation, and whenever a P1 landed, or the tax agents flooded us with complaints about the portal, he’d deliver the reminder again: you are responsible and accountable for the integrity of the tax system.

So the obligation was fixed. Absolute, written into law, indifferent to our circumstances.

The resources were fixed too, in the other direction. Inland Revenue was waiting for business transformation, and everything in the estate was being sweated to survive until the new world arrived. We were two versions behind. End of WebSphere Application Server 6.1 support from IBM due on September 2013. Aging hardware, aging tools, and no appetite to invest in a platform that transformation was going to retire anyway. My team, Business Platform Services Channels, owned the digital services end to end: online, B2B, voice, and output services. That was about 40 percent of Inland Revenue’s ICT applications, but 68 to 70 percent of all application interfaces flowed through Channel, because we were the layer between the customer and everything behind us in the service line: database, then middleware and application, then the mainframe at the back.

And the load had teeth. Tax peak ran from the end of March to August. Tax agents ran refund services that scraped our portal relentlessly, their bots hammering the front end through the night on behalf of clients chasing refunds. Every one of those requests travelled through infrastructure that its own vendor no longer supported.

When the obligation is absolute, the resources are frozen, and the load keeps rising, the only variable left is method.

Year One: Reactive

Our first year was fraught. When a WebSphere server hit its limit, the fastest recovery was a reboot, so we rebooted, sometimes in the middle of the business day. And sometimes the second server failed while the first was coming back, because it took the full load alone and buckled under it, which turned a recovery into a bigger incident than the one we started with. We were learning the hard way that on old infrastructure, the cure can cascade.

Late night calls were routine. Teru, my Senior IT Consultant, would get the pager, an impromptu meeting would form on the phone with the Incident Resolution Lead and whichever technical specialists the failing system had dragged out of bed, and the discussion usually ended the same way: a call to reboot. I made that call, every time. The responsibility and onus sat with me as the end-to-end service owner for digital services, and that’s what owning a service means at two in the morning.

We logged 33 priority one incidents that first year. I told the story of what the whiteboards and the visible metrics did about that number in Two Kinds of Visibility. This piece is the other half of how the numbers came down.

The Turn

We learned the behaviour of our servers the way you learn the behaviour of an old car: by paying attention to it for long enough. We collected data. We correlated system variables, CPU, cache, database log files, with the tax calendar and with application releases, and slowly the patterns showed themselves. It wasn’t rocket science. A server that buckles every time a certain filing date meets a certain cache level isn’t mysterious; it’s predictable, and predictable is workable.

We built benchmark numbers for when each server would need a reboot to clear its cache. And once we could forecast the failure, we stopped waiting for it. We scheduled the restarts for the hours when New Zealand was inactive and asleep. Dawn, just before people came to work. Or 2:00 am, just after the refund agents’ bots finished their scraping runs. The same reboot that had been a midday emergency in Year One became an invisible maintenance routine in Year Two, and that shift is a large part of why Year Two improved so dramatically on Year One.

The War Room

Through tax peak we ran a war room. Every morning, representatives from Channel, Database, Middleware and Application, Mainframe, and Service Management sat in one room, the entire service line from customer to mainframe, and I chaired the briefing.

The rhythm was fixed. Yesterday’s summary first: what we saw, what happened, what action was taken. Then each area updated the health of its systems and applications in turn. Then the tax calendar, so our activities synchronised with what the country was about to do to us. Then the threshold forecasts, which servers were approaching a reboot and when we’d take them. Then pending releases from Development and any fixes we had scheduled ourselves. Service Management kept us compliant with the ITIL process flows and coordinated the preparation whenever a service outage had to be planned for. And when something critical needed a decision on the spot, the BPS managers joined the room and the decision got made there and then, with everyone who’d have to live with it present.

Written down like that, it sounds almost formal. In practice it was a group of people running a national system on obsolete kit with a whiteboard, a pager, and a forecast, and everyone in the room knew exactly why. The Act didn’t care what version of WebSphere IBM still supported.

What Constraint Actually Does

Constraint tends to get either romanticised or pitied, and I think both readings miss what happens inside it. Nobody in that war room would have designed the operation this way. Given the choice, we’d have taken supported servers, modern tooling, proper capacity headroom, and full nights of sleep. We were never given the choice, and the obligation didn’t shrink to fit the resources, so ingenuity became the bridge between them. The data collection, the correlation with the tax calendar, the benchmark thresholds, the dawn reboots: every one of those practices exists because a gap existed that money wasn’t going to close.

That’s usually what arcane methods are, when you stumble across them in an organisation and wonder why on earth things are done that way. They’re the archaeology of some past gap between what had to be done and what was available to do it with. Before you rip one out, it’s worth finding the gap it was built to span, because the gap may still be there.

Business transformation did eventually arrive, years after I left in October 2015, and it retired the estate we’d been sweating. I doubt anyone still remembers the 2:00 am reboots, and honestly, that was the whole idea; the best nights were the ones nobody noticed.

It was what we signed up for.

AI speeds up everything except the constraint. And the constraint is rarely the technology.

Every executive board I’ve sat in front of, whether at Te Wānanga o Aotearoa, Te Pūkenga, or earlier in my career, had the same shape underneath it, regardless of what the organisation actually did.

Papers had to go in weeks ahead of the meeting. There was a format, usually one set by governance requirements flowing down from a central government agency, and the format asked the same kind of questions every time. What were the procurement options. What competing quotes did you receive. How did you arrive at this recommendation. What’s the financial case. As a senior leader, you’d spend real time, real effort, getting that paper into shape, because the format mattered as much as the content, sometimes more.

And here’s the thing I noticed, the same thing in more than one organisation. Below a certain financial threshold, things moved. A CEO’s financial delegation might cover spending up to a certain amount, and inside that boundary, decisions happened at a reasonable pace, weeks, not months. Above that threshold, technology spending had to go to the board, or to the chairperson, or to a steering committee specifically constituted because the programme had crossed, say, a million dollars. And the moment something crossed that line, the pace changed completely. Not because the work got harder. Because the decision now belonged to a different cadence, board meetings, steering committee meetings, a calendar that ran on its own schedule regardless of how ready the work was.

Once you got past that approval point, interestingly, things sped up again, often quite quickly. Which tells you something important. The slowdown wasn’t about complexity, or capability, or even risk, not really. It was about which side of a financial line the work happened to sit on.

That’s the constraint. Not the technology, not the team, not even, most of the time, the people sitting on the board. The constraint is the threshold itself, and the cadence that governs anything above it.

Now think about what AI actually speeds up in this picture.

AI is very good at producing exactly the kind of paper that format demands. Procurement options, comparative analysis, draft business cases, the research that goes into “how did you arrive at this recommendation.” All of that gets faster. A paper that used to take a team two or three weeks to assemble properly could, with AI doing a lot of the drafting and research legwork, come together in days.

So picture what happens. More papers, better researched, arriving faster, all queuing for the same board cycle, the same steering committee cadence, the same financial delegation threshold that hasn’t moved an inch. The constraint doesn’t get any faster just because the things arriving in front of it got faster to produce. If anything, the queue gets longer, because now it’s cheaper and quicker to generate a business case, so more of them get generated, and they all still have to wait for the same monthly or quarterly slot.

This is the part that I think gets missed in most conversations about AI and productivity. Speeding up the non-constraint doesn’t speed up the system. It just means the constraint now has more sitting in front of it, dressed more convincingly, on the same clock as always.

At Te Pūkenga, this played out in a very specific way. There was a real appetite, especially in technology, to move fast, to make decisions and adjust later if needed. Seek forgiveness rather than permission, the old phrase. But that approach, taken too far, breaks the agreement between the organisation and central government, the governance arrangement that the financial delegation and steering committee structure exists to protect. So the choice sat with whoever held the role, often the Chief Digital Officer, and it wasn’t really a technology choice. It was a political one. Move fast and risk the relationship with the centre, and your own position along with it, or stay inside the governance lines and accept that the pace is set by someone else’s calendar.

Most people, reasonably, chose the second option. And the work that had been done, the research, the supplier engagement, the commitment built with development teams, much of it sat there, not wasted exactly, but not moving either, while the business division absorbed the cost of effectively standing still. Spinning the wheels, consuming budget, with nothing to show for it on the outside, because the thing that needed to move was waiting on a decision that hadn’t been made.

And here’s where it gets sharper still, because indecision isn’t neutral. Indecision is itself a decision, usually the decision to keep the status quo running for as long as possible.

At Inland Revenue, I started in January 2013. My first day on the job, we were hit with a Priority 1 incident, the kind that takes taxpayer-facing systems down entirely. My senior manager introduced me to the floor and told everyone I was the one who’d sort it out. That was day one. By the end of that year we’d had more than thirty Priority 1 incidents, on top of everything lower in severity that never made it into anyone’s headline numbers.

The infrastructure behind it told you why. The WebSphere Application Servers running our channel systems were three versions behind current, and they buckled under unusual load, especially during the tax peak season running April through August, exactly when the system needed to hold up the most. The Business Platform Services Channels team I was part of handled roughly 70% of IRD’s systems through its public-facing portal. Alongside that, the document management platform, an IBM Enterprise Content Management system that every tax form and every piece of taxpayer correspondence ran through, was already out of vendor support. We were paying IBM a premium just to keep getting support on a platform it no longer wanted to support, because the realistic fix, an upgrade to the lowest version still covered, never quite made it to the front of the queue.

Every outage got noticed. Leadership attention, public criticism, press coverage, a tax agency whose own systems taxpayers couldn’t access to file their returns, accountants scrambling to lodge clients’ returns through whatever workaround they could find. None of that is a small thing for an organisation whose entire mandate depends on people trusting the system enough to use it.

Keeping that infrastructure at 99.9% availability through tax season took real ingenuity, because the investment to actually fix it properly wasn’t coming. Leadership knew it. The plan, such as it was, was to sweat the assets for as long as they could be sweated, until the broader Business Transformation programme eventually arrived to replace everything properly. In the meantime, the onus sat with the people in Business Platform Services, keeping a system three versions behind current running at a reliability target that newer infrastructure would have struggled to hit without strain.

That requirement didn’t soften just because the underlying technology was ageing and the replacement programme was stuck in a queue somewhere above someone’s financial delegation. The risk of that gap, between what the system was being asked to do and what it was realistically capable of doing reliably, didn’t sit with whoever was holding up the approval. It sat with the people keeping the lights on every day, the ones whose job it was to make 99.9% happen on infrastructure that leadership already knew was past its best.

This is what I mean when I say AI exposes the constraint rather than removes it. If AI is used to speed up the analysis, the business cases, the options papers, the reporting, exactly the things that feed into a board cycle or a steering committee, none of that touches the actual bottleneck, which is a financial threshold and a meeting cadence set by governance arrangements that have nothing to do with how fast anyone can write a paper. What changes is the size and polish of the queue sitting in front of that bottleneck, and the distance between what’s technically possible and what’s actually approved gets more visible, faster, because the technical side is now moving at AI speed and the governance side is moving at the same speed it always has.

The gap was always there. Sweating the IRD assets, the Te Pūkenga standstill, these existed long before anyone in those organisations had touched an AI tool. What AI does is make the two speeds, the speed of producing the case for change and the speed of actually deciding on it, diverge so sharply that the gap stops being something you can quietly manage and starts being something everyone downstream can see.

I think that’s worth sitting with, if you’re the one several layers down, keeping something running at 99.9% while the decision about its future sits in someone else’s inbox. AI won’t move that decision any faster. It’ll just make it more obvious, to more people, sooner, exactly how long that decision has been sitting there.

Leadership, in my experience, genuinely doesn’t see this whole chain. They see their piece of it. The approval is pending, that’s a known item on a register somewhere, fine. The systems are running, availability targets are being met, also fine. Nobody in the room is looking at the line connecting those two facts, that the second fact is only true because of effort and risk being absorbed by people several layers down, for as long as the first fact remains unresolved. As long as it isn’t front-page news, the risk reads as managed.

I’ve been on both sides of that gap. I know which side feels the cost first.

One morning in 2013, a senior manager stopped at our whiteboard at Inland Revenue while my team was running through the day’s incidents, looked at the columns of cards for a while, and said, “Wayne, we don’t do agile here.”

I said, “No, we’re doing TPS. Toyota Production System.”

She wasn’t satisfied with that. “Just remember, Inland Revenue doesn’t do agile.” Then she walked off.

She ran ICT Solutions. She had sat on the panel that hired me. And she had carried Mary and Tom Poppendieck’s Lean Software Development around during her MBA sometime later, which made the line even better.

That was the entire exchange. Nothing came after it, no memo, no meeting invite to discuss my team’s unauthorised working methods. And looking back, the reason nothing came after it is the whole point of this piece. We never asked permission for what we were doing, and we never argued for it in a meeting room either, because we would have lost that argument. We did the work where people could see it, and let the results carry the case.

Some context first. I ran Business Platform Services Channels at Inland Revenue from January 2013 to October 2015, a team of 11 or 12 looking after the systems sitting between New Zealand taxpayers and everything behind the portal. Channel owned about 40 percent of Inland Revenue’s ICT applications in its own right, but as the front end of the service chain, up to 70 percent of all application interfaces flowed through us on their way to the middleware, the databases, and the mainframe at the back. So any incident touching the portal got assigned to us to triage first, whether or not the fault actually lived behind us. We wore every complaint, and that’s how busy we were: sifting through the shit to work out whose it was. When the portal goes down, nobody rings the database team.

The technology was waiting for business transformation to arrive. WebSphere servers out of support from IBM, infrastructure being sweated to survive until the new world got funded and built. Least resources, oldest kit, first in line for blame. And in my first year, 2013, we logged 33 priority one incidents. Thirty-three times in a single year, a system serving every taxpayer in the country fell over badly enough to be declared a P1.

What We Actually Did

Every morning we stood at a whiteboard and went through the work. To do, in progress, done. Whatever needed attention across the entire service line got a card. This wasn’t proper agile; there was no Scrum Master and no sprint cadence, because production support doesn’t run in sprints. Incidents don’t wait for planning day. We were level 2 and level 3 in the support chain, taking what the Service Desk at level 1 passed up, triaging it the way an emergency department triages patients, firming up the diagnosis, fixing what we could ourselves, and passing the rest to the development teams at level 4. Inland Revenue had run Scrum on some projects before, and several of my people knew the ceremonies well, so the muscle memory was there. We simply pointed it at operations instead.

I called it Toyota Production System because that’s what it was: make the work visible, limit what’s in progress, let the team pull the next priority, and treat every incident as a signal about the system rather than a fire to forget once it’s out.

Teru Yanagihasi, my Senior IT Consultant, took the visibility further than I would have on my own. He put a metrics dashboard straight onto the whiteboard, openly displaying where the issues actually lived across the service line, which trends were building, what needed watching next. He also introduced a Niko Niko board, a Japanese practice where each person marks their mood for the day, which sounds soft until you realise it tells you about the condition of the team before the incident queue does. A tired team misses thresholds. Many of the ideas that worked in that team started as Teru’s; his dedication and commitment were second to none.

None of this was electronic. There was no tooling budget. Whiteboard, markers, and a team that showed up every morning.

My peers and my leadership tolerated all of it, I guess because the numbers kept moving in the right direction and it cost them nothing to look the other way. The P1 count kept falling, and they left us alone. Results bought us an autonomy that no argument would have.

The Viral Part

Then something happened that I didn’t plan. Colleagues from the other Business Platform Services teams walked past our corner, saw the board, watched the morning ritual for a while, and whiteboards started appearing in their spaces too. Nobody mandated visual management. There was no workshop, no change programme, no lunchtime session on lean methods. The practice spread because it sat in the line of sight of anyone walking the floor, and because the thing it was attached to, our incident numbers, was visibly improving month after month.

Eventually my Director for ICT Operations bought us a 45 inch TV to display our monitoring dashboards, an envy to the rest of BPS. I suspect that TV did more for adoption across the floor than any presentation I could have given.

The numbers are the part I can still recite. 33 priority one incidents in 2013. Seven in 2014. Three in 2015, my final year.

Behind those numbers was a team watching thresholds against upper control limits, CPU, cache, database log files, queue depths, and taking preventive action in quiet hours before any limit was breached. The whiteboard made the priorities visible. The dashboards made the thresholds visible. The discipline was simply that somebody was always watching and always acting early, and that discipline held availability at 99.9 percent through tax peak seasons, on assets that were officially past their support life. How we sweated those assets through peak season deserves its own piece, so I’ll leave that story for another day.

The Newsletter

Three P1s in a year got attention at the top. The CIO came down and asked what we had done to make that possible, and then he wanted a piece in the internal newsletter about the improvement in operations.

Here’s where I learned something about visibility that the whiteboard hadn’t taught me. I told him the truth: we relied on more eyeballs watching the thresholds of the upper control limits for failure. He looked puzzled. I explained the eyeballs again, and he looked more puzzled. You can’t write “we watched harder and acted earlier” in a newsletter. There’s no chart for it, no milestone, no budget line, nothing that reads like an achievement.

In the end, I suggested he attribute the improvement to the mainframe, since we had a mainframe project running at the time. A project is quantifiable. It has a name, a start date, an end date, a cost. It fits the shape a newsletter expects. So the mainframe got the credit, and I’m the one who offered it, because I understood the constraint he was working under. The improvement he actually came down to ask about was the ingenuity of an entire team monitoring thresholds and sweating aging assets, and that never gets published, because it sounds simple and boring.

Two Kinds of Visibility

I’ve thought about this a lot since. There were two kinds of visibility in that story, and they behave completely differently.

Sideways visibility spreads practice. Peers copied the whiteboard because they could see it working with their own eyes, on their own floor, attached to numbers that kept falling; no persuasion required, no business case, no mandate. When results are physically visible, adoption becomes voluntary, and voluntary adoption sticks in a way that mandated change rarely does. That’s the persistence half of the story. We kept doing the same simple things, quietly, for three years, and the copying followed the results.

Upward visibility grants credit, and it only registers what is countable and project-shaped. The institution could see the mainframe project because it had the right form. It couldn’t see the eyeballs, even when the person responsible stood in front of the CIO and explained them twice. So the official record of that improvement, if anyone ever digs out that newsletter, says the mainframe did it.

Both kinds are real, and both matter. But if you run operations anywhere, it’s worth knowing that the work that spreads among practitioners and the work that gets credited by the institution can be entirely different work, and the gap between them is where a lot of quiet operational excellence lives and dies unrecorded.

Teru returned to Japan after 2021. The senior manager was right in the end; Inland Revenue never did do agile, at least while I was there. It did TPS for three years in one corner of the floor, and the whiteboards told everyone who cared to look. Even she was converted before I left.

I still think the eyeballs deserved the newsletter.