At 2:00 am, just after the bots from the tax refund agents finished scraping the portal, we rebooted the servers.
The servers were healthy. That was the point. By the time an out-of-support WebSphere Application Server tells you it’s unhealthy, you’re already inside a priority one incident, and a system serving every taxpayer in New Zealand is falling over with the whole country watching. So we rebooted while the country slept, on a schedule built from our own data, and nobody outside the team ever knew it happened.
This was Inland Revenue between 2013 and 2015, and this piece is about how a team ran the machinery of the New Zealand tax system on technology officially past its life, with methods you could politely call arcane, because we had no other choice.
The Mandate
My boss that ran Business Platform Services was a tough fellow who made sure you knew what you signed up for. Hands off otherwise, which suited me fine. He had one famous line, delivered whenever shit hit the fan: “We are here to protect the integrity of the tax system. This is what you signed up for and what the Act says.”
The Act is the Tax Administration Act 1994, and he wasn’t quoting it for effect. The integrity of the tax system is a statutory obligation, and whenever a P1 landed, or the tax agents flooded us with complaints about the portal, he’d deliver the reminder again: you are responsible and accountable for the integrity of the tax system.
So the obligation was fixed. Absolute, written into law, indifferent to our circumstances.
The resources were fixed too, in the other direction. Inland Revenue was waiting for business transformation, and everything in the estate was being sweated to survive until the new world arrived. We were two versions behind. End of WebSphere Application Server 6.1 support from IBM due on September 2013. Aging hardware, aging tools, and no appetite to invest in a platform that transformation was going to retire anyway. My team, Business Platform Services Channels, owned the digital services end to end: online, B2B, voice, and output services. That was about 40 percent of Inland Revenue’s ICT applications, but 68 to 70 percent of all application interfaces flowed through Channel, because we were the layer between the customer and everything behind us in the service line: database, then middleware and application, then the mainframe at the back.
And the load had teeth. Tax peak ran from the end of March to August. Tax agents ran refund services that scraped our portal relentlessly, their bots hammering the front end through the night on behalf of clients chasing refunds. Every one of those requests travelled through infrastructure that its own vendor no longer supported.
When the obligation is absolute, the resources are frozen, and the load keeps rising, the only variable left is method.
Year One: Reactive
Our first year was fraught. When a WebSphere server hit its limit, the fastest recovery was a reboot, so we rebooted, sometimes in the middle of the business day. And sometimes the second server failed while the first was coming back, because it took the full load alone and buckled under it, which turned a recovery into a bigger incident than the one we started with. We were learning the hard way that on old infrastructure, the cure can cascade.
Late night calls were routine. Teru, my Senior IT Consultant, would get the pager, an impromptu meeting would form on the phone with the Incident Resolution Lead and whichever technical specialists the failing system had dragged out of bed, and the discussion usually ended the same way: a call to reboot. I made that call, every time. The responsibility and onus sat with me as the end-to-end service owner for digital services, and that’s what owning a service means at two in the morning.
We logged 33 priority one incidents that first year. I told the story of what the whiteboards and the visible metrics did about that number in Two Kinds of Visibility. This piece is the other half of how the numbers came down.
The Turn
We learned the behaviour of our servers the way you learn the behaviour of an old car: by paying attention to it for long enough. We collected data. We correlated system variables, CPU, cache, database log files, with the tax calendar and with application releases, and slowly the patterns showed themselves. It wasn’t rocket science. A server that buckles every time a certain filing date meets a certain cache level isn’t mysterious; it’s predictable, and predictable is workable.
We built benchmark numbers for when each server would need a reboot to clear its cache. And once we could forecast the failure, we stopped waiting for it. We scheduled the restarts for the hours when New Zealand was inactive and asleep. Dawn, just before people came to work. Or 2:00 am, just after the refund agents’ bots finished their scraping runs. The same reboot that had been a midday emergency in Year One became an invisible maintenance routine in Year Two, and that shift is a large part of why Year Two improved so dramatically on Year One.
The War Room
Through tax peak we ran a war room. Every morning, representatives from Channel, Database, Middleware and Application, Mainframe, and Service Management sat in one room, the entire service line from customer to mainframe, and I chaired the briefing.
The rhythm was fixed. Yesterday’s summary first: what we saw, what happened, what action was taken. Then each area updated the health of its systems and applications in turn. Then the tax calendar, so our activities synchronised with what the country was about to do to us. Then the threshold forecasts, which servers were approaching a reboot and when we’d take them. Then pending releases from Development and any fixes we had scheduled ourselves. Service Management kept us compliant with the ITIL process flows and coordinated the preparation whenever a service outage had to be planned for. And when something critical needed a decision on the spot, the BPS managers joined the room and the decision got made there and then, with everyone who’d have to live with it present.
Written down like that, it sounds almost formal. In practice it was a group of people running a national system on obsolete kit with a whiteboard, a pager, and a forecast, and everyone in the room knew exactly why. The Act didn’t care what version of WebSphere IBM still supported.
What Constraint Actually Does
Constraint tends to get either romanticised or pitied, and I think both readings miss what happens inside it. Nobody in that war room would have designed the operation this way. Given the choice, we’d have taken supported servers, modern tooling, proper capacity headroom, and full nights of sleep. We were never given the choice, and the obligation didn’t shrink to fit the resources, so ingenuity became the bridge between them. The data collection, the correlation with the tax calendar, the benchmark thresholds, the dawn reboots: every one of those practices exists because a gap existed that money wasn’t going to close.
That’s usually what arcane methods are, when you stumble across them in an organisation and wonder why on earth things are done that way. They’re the archaeology of some past gap between what had to be done and what was available to do it with. Before you rip one out, it’s worth finding the gap it was built to span, because the gap may still be there.
Business transformation did eventually arrive, years after I left in October 2015, and it retired the estate we’d been sweating. I doubt anyone still remembers the 2:00 am reboots, and honestly, that was the whole idea; the best nights were the ones nobody noticed.
It was what we signed up for.