A recent Washington Post article on the largely paper-based civil service retirement system, which takes an average of 61 days—the same as in 1977—to calculate the pension checks of government retirees, is serving as an excellent microcosm of the problems in developing large IT systems, particularly government ones.

The Post uses “bureaucratic sinkhole” as a metaphor for the fact that the work is performed in an underground storage facility in a remote part of Pennsylvania because it’s the only place that can hold the required 28,000 file cabinets of paper records. There have been multiple costly attempts to automate the system, but all have failed. “During the past 30 years, administrations have spent more than $100 million trying to automate the old-fashioned process in the mine and make it run at the speed of computers,” writes reporter David A. Fahrenthold. “They couldn’t.”

In just the first few days since the article was originally published, it has received more than 700 comments,with respondents variously blaming the problem on President Obama (though the situation dates back to at least the 1970s), socialism, the federal government, private contractors, and Congress.

However, this problem isn’t new, nor is it simply solved. Government IT systems have a reputation—deserved or not—for being inefficient, expensive and, worst of all, not actually solving the problem. Numerous other agencies, such as the Veterans Administration in its attempt to reduce the backlog in veterans’ medical care, have also dealt with similar situations.

Ironically, while some of the comments to the article were from retired government workers telling horror stories of having to wait a year to get correct retirement checks, a number were other retired government workers praising the Office of Personnel Management (OPM) staff for how accurately and swiftly they worked with the paper-based system, and suggesting that it not be changed—a conflict that is likely familiar to any number of IT people who’ve tried to change an existing, if outdated, process.

Why is automating systems so hard?  

Breaking the problem down into its constituent parts can be instructive, not just for this case but for automating business processes in general.

1. The data is complex: The retirees in question have sometimes worked for the government for 40 years or more, in a variety of agencies, in a variety of roles. Moreover, over the years, Congress has implemented a number of different systems to provide varying levels of benefits to various types of employees. “If you start out with the entry of an employee’s record and there are 73 possibilities of exactly which category that employee might fall into, and there are specific requirements to validate all of the classifications, and you are trying to bring in records from many dozens of separately designed personnel systems, the complexity can be astonishing,” reported one commenter on the article.

One of the automated systems that was tried—and failed—in 2008 needed to be able to make 150 calculations, reported the Washington Post at the time. It was scrapped when it was found that while it could correctly calculate 15 of the most common cases, it could correctly calculate only 5 of the next 61.

2. Some of the data is paper-based to begin with: While some government agencies are now using electronic data to keep track of employees, agencies weren’t necessarily doing that 40 years ago. And because some of the data is on paper, that means even when some data is electronic, it may have to be printed out, coordinated with the paper-based data, and then manually re-entered—all processes that are time-consuming and prone to errors.

3. The paper-based data isn’t necessarily “clean”: In the 2008 automation effort, the agency found that the records required significant “cleaning up”—“restoring missing information, translating writing in margins, and verifying dates and other data,” the Post reported.

4. It may not be worth it to convert records to digital: Even if converting the paper records to digital could be done, the majority of the records don’t need to be converted, so converting them would be costly and useless. “The number of WWII Veterans and Korean War Veterans still having any new claims processed is infinitesimally small relative to their numbers at the end of those respective wars,” a commenter writes. “It would make no financial sense, none, to try to digitize those sorts of records now and even less (if that’s possible) in the future.”

5. The processes themselves are complex and not well-documented: Numerous commenters noted that the paper-based system was so complex that it would be difficult to automate. Yet, at the same time, any efforts to simplify the system would be fought by people who perceived that such a simplification would mean they—or their constituents—would lose benefits. “As long as the retirement system is bound up in 100 years of byzantine laws that control the actions and outcomes of dozens and dozens of federal agencies, this problem will continue to defy automation,” reported one.

6. It can be hard to incorporate institutional knowledge into business processes: The current people-based system appears to be successful, to the extent that it is, primarily because the staff themselves are experienced with it—an institutional knowledge that isn’t easy to convert to automated business processes. Indeed, a 1981 report cited by the Post on delays in the system—back when it took just 56 days to process a retirement claim—blamed that backlog on a loss of experienced employees.

Part of the problem with developing an automated system, noted one commenter, is that the people writing the specifications were typically administrators rather than the front-line workers who’d been doing the job for many years. “People, and people with deeply ingrained ‘unwritten’ knowledge of ‘how things work’ are critical for your day-to-day success with the paper process and for getting a full picture of what needs to be done,” he writes.

How will this “bureaucratic sinkhole” be addressed?

All that said, how will this specific problem be solved? First, government retirement is gradually being converted from a defined-benefit system (a pension) to a defined-contribution system (like a 401K), which will make it less complicated to calculate.

Second, as time goes on, more and more retiring employees will have been working in agencies with electronic employment records that, even if they aren’t equivalent across agencies, will at least be easier to work with than paper.

In the meantime, particularly since an increasing number of federal employees are reaching retirement age, the federal government is dealing with the problem by throwing more bodies at it—200 new employees in the past five years. Just to stay in the same place as in 1977.