The audit of the Commonwealth Office of Technology was released today. Sometimes, when you read an audit of an IT shop and see a finding, you think “well, that’s just being picky; it doesn’t matter that much.”
But sometimes, you read a finding and you think, “SMH – really?” And then sometimes, you read something and you go “WHAT?!?”
Today, when I read the COT audit report, there were three details in the findings that jumped out at me. On the SMH Scale (Shake My Head), they range from a 7 (“Really?”) to a 9 (“WHAT?!?”)
Look, I worked in IT for decades. There’s a reason there’s a book calling it “the toughest job in the world.” Even done well, it’s hard.
But done poorly, it’s both hard and dangerous. And these three details look dangerous to me.
#1 – “COT does not have a complete list of machines, files, or data currently being backed up.”
What? The Commonwealth Office of Technology (COT) does backups for agencies of the state, but doesn’t know what’s being backed up? They don’t know which machines are being backed up? And where it’s being backed up? Then how do you find a file to restore it, if you don’t know where it is?
The audit notes that business continuity (BC) plans, which would include planning for backups, are still the responsibility of the individual agencies. But, the COT provides the backup service for each agency. And no one at the COT knows what the agencies are backing up. Apparently, the COT and the agencies don’t talk about it.
That’s just nuts.
*** SMH Scale: 7 ***
#2 – “Approximately 2,000 servers do not have current backups.”
I’m sorry, but this is just inexcusable — for both the COT and for the agencies. According to the audit, the original number in 2017 was 2,500. After working for about a year, the COT has that number “down to 2000.”
I mean, WTH? Have these people never heard of power surges, or failing hard drives, or bit rot? Do they think computer equipment lasts forever?
The audit notes that in the “IT Infrastructure Initiative,” the COT was given management authority over all servers, infrastructure, and network directory services. So, supposedly, they can tell the agencies “look, either you get a backup in place, or we’re going to do it and charge you for it.” This isn’t about whether your screen is blue or green; it’s about data—data that makes the state government able to function—being protected and available when needed. Two thousand servers not backed up? That’s a firing offense in most businesses.
*** SMH Scale: 8 ***
#3 – “COT does not have a complete list of all servers they manage and maintain for the Commonwealth.”
This sentence is in the middle of a paragraph talking about something else (data security), but when I read it I just stopped. I read it again, to make sure I had read it correctly.
You do it too: just read it out loud, slowly. Then think about it: the part of state government that is responsible for managing all the servers across state government does not even know what servers are out there.
Look, I can understand if the COT had just been assembled from disparate organizations that had never worked together before. I can understand if the COT was a startup, or some family business that had not made it into the 21st century.
But this is the technology office for a $30 billion business. Not having a reliable list of servers is like McDonald’s not having a list of its stores.
Note: The COT was told about this in last year’s audit as well. And it still hasn’t been fixed.
*** SMH Scale: 9 ***
The bottom line
So here’s the bottom line, for me. I actually think state government does a number of good things in its use of technology. Most of the web sites are both usable and attractive. The transparency continues to get better, and more and more of our interactions with state government can be done online.
But these three sentences in this year’s audit? They strike at the root, the foundation, of the IT work done by the state. Any one of them could be a disaster, waiting to happen. There is just no excuse for these issues not being addressed since the last audit. These need to be fixed, ASAP if not sooner.
And if they show up on next year’s audit as well, someone – or many someones – should be fired.