There are days when your only consolation is that someday, somewhere, your mistake might help prevent someone else’s. The Federal government’s Office of Personnel Management (OPM) has been having a bunch of days like that recently, which gives us all an opportunity to learn what to do differently.

It may have started as long as a year ago when hackers first broke into OPM systems, using credentials from a contractor, and installed malware that would enable them to steal data in the future.

“Hackers obtained a credential used by KeyPoint Government Solutions, a Colorado-based contractor that OPM uses to conduct background investigations of applicants for federal jobs that require a security clearance,” explains Erin Kelly in USA Today. KeyPoint doesn’t know how the employee’s credentials were compromised, she continues. And multifactor authentication wasn’t implemented.

The upshot is that personally identifiable information from anywhere from 4.2 million to 21 million Federal employees, job applicants, and contractors was stolen. That includes names, Social Security numbers, addresses, fingerprints, and so on. Apparently the data wasn’t encrypted, in part because the hardware on which it was stored was so old—on the order of three decades—that encryption software for it didn’t exist.

It turns out encryption wouldn’t have helped much anyway, because the administrators responsible for managing the records had root access to the system, writes Sean Gallagher in Ars Technica.

And we only assume that data on employees was taken. “What if records were not only taken, but some were added as well?” writes Steve Ragan in CSO Online. “Would the OPM be able to tell? The attackers had at least a year of unchecked access on the network—plenty of time for someone to do whatever they wanted.”

To make matters worse, the hackers apparently also stole data from Standard Form 86 and similar data, called “adjudication information.” What is Standard Form 86? It’s what you fill out when you have a security clearance, so they ask you all sorts of personal information looking for things that could potentially be used to blackmail you in the future. So the OPM databases included information on employees’ friends and family as well.

“This did not have to happen,” says Dr. Stan Stahl, president of the Los Angeles Chapter of the Information Systems Security Association (ISSA-LA) and the founder and president of Citadel Information Group, Inc., an information security management firm. “This was mismanagement. For that to have been allowed to happen, there’s no excuse.”

If the data actually got stolen as much as a year ago, apparently nobody’s tried using it yet. Or if they are, nobody’s talking about it.

It’s even possible that data on classified employees was also stolen, because there was an effort after 9/11 to merge databases to make them easier to search—even though the classified agencies didn’t want to do this because they were concerned about security. If that’s the case, nobody’s talking about that, either.

Altogether, 65 percent of OPM’s data was stored on 11 major systems that had not been properly certified as secure and were run by OPM’s own IT department, out of a total of 47, Gallagher writes. (Some systems were run by contractors.)

This has been a developing story for the past several weeks, which is why the number of people affected keeps rising. In fact, there are those who contend that 32 million people were actually affected—basically, anyone who’s applied for a Federal job.

That’s essentially 10 percent of the US population, writes Steven J. Vaughn-Nichols in Computerworld, going so far as to suggest that compromised people should all receive new Social Security numbers. “Any way you cut it, fixing this is going to take a minimum of tens of billions of dollars,” he writes.

Another part of how the story has developed is how the break-in was discovered in the first place. The first story is that it was through EINSTEIN, a system run by the Department of Homeland Security to record, detect, and block cyber threats. Then a security vendor, CyTech Services, reportedly discovered the malware while it was doing a demo for OPM. Then OPM said, they’d discovered it themselves a few days before that, using a different vendor’s product.

It does seem like, in one way or another, changes are likely at OPM. There have already been a series of Congressional hearings on the issue, and director Katherine Archuleta has resigned. Though finding out whom to blame isn’t necessarily the most important part of this incident.

“Instead of hearings in D.C. that are focused on blame and attribution, perhaps there should be hearings to address budget cuts and the lack of proper security staffing in critical areas of the government,” Ragan writes.

“Clearly, OPM long knew they had a major problem on their hands due to their reliance on out-of-date equipment and software,” agrees Vaughn-Nichols in a piece for ZDNet. “They knew their obsolete IT infrastructure made them more vulnerable to hackers. And, they knew what the answer was. It’s just too bad they couldn’t get Congress to pay for it.”

So now we get to the part where we talk about what we’ve learned to do differently.

• If you’ve got a lot of personally identifiable information about people, encrypt it.
• If it’s on hardware that is too old to support encryption, put it on different hardware.
• If you can’t afford different hardware yet, at least don’t hook the hardware up to the Internet.
• Similarly, ensure that your records management software is up-to-date to support modern security techniques—and if it doesn’t support them, find new software.
• Limit the people who have access to passwords into the system.
• Especially limit the people who have root access.
• Log the times when that access is used, so you can see if they come at an odd time or from an odd place.
• Implement multifactor authentication, so someone can’t break in by just having a password.
• Periodically run scanners looking for malware on your system.
• If you need a cybersecurity expert, hire one right away.

To do this, CIOs—working with CISOs if they are separate positions—need
to talk to boards about security, Stahl says. “The most important thing is
that CIOs have got to find their way up to the executive office, the
boardroom,” he says. “They’ve got to be able to explain security in ways
that management can understand. Boards need insight.”

Look at the bright side: This is probably a great time to get that security budget line item funded.