BAM’ll Fix It

Auditor General Thomas McTavish’s very critical assessment of the BAM project is available in PDF format from the State of Michigan website.

Since Michigan Secretary of State Ruth Johnson mentioned–and bad-mouthed–the Business Applications Modernization project in her speech the other day, I thought I’d follow up on an earlier posting. Please understand that I’m not defending the BAM project. I may (or may not) discuss what might have been done differently in some future post.

Fair warning: This post discusses a system which hasn’t been implemented. Prior to the project’s web phase I was working on other projects, and only rarely an active BAM participant, so this post summarizes discussions I heard only second- or third-hand. I’ve also simplified things enormously to fit this treatment into a still too long essay, with the unfortunate consequence that I’ve ignored entire initiatives, made some difficult decisions look painless, and rendered some heated disagreements as settled questions. Finally: At no point was I a BAM project insider. Where I say “we” in this essay I mean The Department of State.

Introductory

The State of Michigan has been working with a vendor to replace the Department of State’s mainframe system, a key component of the State’s DMV. The vendor is HP, which you can find from their website (PDF), though the contract was originally with EDS. After years of missed targets, it’s fair to say we’ve proven that a system which took decades to build cannot be easily replaced. I’m reasonably comfortable with that, but my opinion’s not universally shared. Moreover, recognizing that the problem is difficult is very different from accepting an inadequate solution, which remains a real danger.

Yes, we might have done things differently. It’s likely we should have. This document is a description, not an argument.

Proto BAM

I mentioned a few weeks back that the Department of State’s management had concluded by the late 1990s that assuming a long-term future for our mainframe was untenable. It took a few years, and a new Secretary of State, for a project to completely replace that system to gain traction.

Terri Land was elected Secretary of State in 2002, and took office early in 2003. One of management’s recommendations to Secretary Land was replacing the Department’s mainframe with a more modern and maintainable system. After the new Secretary’s staff had digested that recommendation, knowledgable senior staffers were invited to brainstorming sessions, where we were invited to propose technological improvements for every existing workflow and to help define how those new systems should work. Those meetings’ notes, after considerable further digestion, became a formal proposal to consolidate nearly every Department of State technology initiative on a single platform. Thus was the Business Application Modernization (BAM) project born.

First BAM

BAM began with high hopes. Management believed the new system would address most existing workflow problems, and began to make staffing decisions on the assumption that the project would become available in 2006. 2004 saw the first mentions of BAM interfaces in purchasing contracts. Meetings with EDS staff became a regular feature of our work life. “BAM’ll fix it” became a mantra, particularly since the programming staff’s necessary involvement in the endless meeting made it still more difficult to implement even vital mainframe programming changes.

Unfortunately, the unending meeting soon made it clear to our participants that the vendor didn’t understand the basic problem. That the discussions opened at a high level was obviously necessary, but an oft-voiced concern was that their staff clearly wasn’t much interested in the details and complications which we understand as our job. “BAM’ll fix it” devolved into a running joke, and the initial project collapsed after two years of requirements-gathering and a clearly inadequate, unsatisfactory prototype. The contract was renegotiated, and EDS acquired Saber Government Solutions, a company with experience with another state’s DMV programming, to revitalize our effort. (This was a few months before HP bought EDS.) The project was relaunched as 2008 began.

Second BAM

Since the Saber team had previously worked on a similar conversion, they recognized that the details mattered. This worked far better, though Michigan’s differences from the other state sometimes provoked discussions which were, well, interesting. This project was plagued by obviously-unrealistic milestones and “deadlines,” but at least the still-perpetual meeting revolved around issues our staff actually considered significant. There was an inevitable retreat on project scope. Where the original objective was to consolidate nearly every departmental workflow and database in a single system, in the end the project became mostly an effort to replace the mainframe with a perhaps-better system having roughly the same capabilities, plus some few obviously-valuable improvements. Operationally, we’d be replacing one set of system quirks with another. This platform could be the basis for further enhancements, someday, but those would be some other team’s project.

One of the compromises was to initially implement BAM as a hybrid system, with much of the decision-making code, including Computer Decide, remaining on the mainframe. In trade we made a commitment to a more logical and efficient data structure. A new user interface, which everyone recognized would involve gains and losses, was agreed upon. This compromise was defensible as a tactical budget and resources decision, but it seriously undermined the project’s strategic rationale.

The Saber team began writing code, and members of our staff began testing the system. As is often the case with a major new system, the initial results were daunting. Almost nothing worked. There were simple coding errors, there were serious flaws in the design, and there were significant misunderstandings about our processes. Outputs were reliably poor in many dimensions. For the few routines and screens which did work, system performance was unacceptable. It was weeks before the testers were first able to run a transaction to completion, and months before they began to see acceptable results. Failures were reported by the main testing team, which was primarily assessing the branch office programming, and by folks developing interfaces between BAM and other systems. Thousands of defects were logged. Fixes tended to reveal new bugs, so more defects were filed. After a few weeks, even the most battle-hardened testing veterans were discouraged. The programmers plugged away at the problems, and in time the bug list became more or less manageable. But by that time many testers had grown weary and cynical. After months of logging defects, any testing team tends to be much more aware of unresolved issues than whatever progress can be shown. A life’s lesson, that.

Web BAM

Late in 2009 the most important of the agreed enhancements–a web interface which we expect will move many transactions out of the branch offices–was launched as a semi-independent project. The original plan was that BAM Web would go live after the main BAM system was introduced in the branch offices, but it became clear last summer that the web application would be stable before the branch system was available. This forced re-prioritizing the main project, as the web implementation required devising backend support processes to replace tasks which had heretofore been handled only over the counter. Sorting this out temporarily stopped coding for both projects, and disrupted the main project’s development plan.

Third BAM?

This paragraph is speculation. The clearest statements about BAM in Secretary Johnson’s speech are that the effort is still alive, at least in spirit, and that the web interface is important. I take this to mean they plan to build the web interface approximately as designed. Probably its back end will interact more or less directly with the mainframe, which will require some reengineering on both systems. What will happen with the rest of the project is likely still under discussion. We’ll have to see how things develop.

Edit 5/12/11: I’m told the web effort will be using the BAM hybrid back end, described above, which implies that the rest of the project will eventually be implemented as the hybrid is not a viable long-term solution.

Secretary Johnson’s renegotiate-the-contract effort is significant, but not essential to the project itself. She’s wrong, though, about what would be tolerated in the private sector. I’m pretty sure she’s letting ideology blind her to known facts. See this.

Future BAM

Everything an elected official’s staff does has potential political impacts, but this project was unusually political. Secretary Land had proposed to remake the way we do business, and fully intended to see the changes implemented on her watch. One result was an ever-present tension between the desires of the Secretary’s personal staff and the honest expectations of those working on the project. An example: When I retired on November 1 the officially-unofficial planned website launch was known to be mid-December, with the branch changes to follow over the following weeks. My fairly-informed opinion was that a more realistic expectation was an April kick-off for the web, and the branch changes perhaps by autumn.

BAM Web will likely launch later this year. I fully expect the rest of the mainframe-replacement implementation to eventually occur, because the alternatives are ugly, but it’s unlikely that this project will be Secretary Land’s BAM. The simple truth is that maintaining the old system carries similar risks to pressing forward with its successor, at the sacrifice of BAM’s potential benefits. I’m less sanguine about the eventual implementation of the original BAM plan’s ambitious vision of a comprehensive platform. Better, perhaps, to design ways to easily integrate external systems.

One hazard haunting any years-late technology project is the prospect of failure. That may yet occur with BAM, particularly since Governor Snyder and Secretary Johnson both seem skeptical about the project. A greater risk, though, is that after investing years of staff time and a fortune in dollars to develop this system, management will elect to accept a mediocre solution, with a vendor pledge to eventually repair the faults. Truth is we’ve a poor acceptance test track record, and a decades-long tradition of making do. We can’t afford that with mission-critical platforms.

Ms. Land no longer holds any public office. Here’s hoping Secretary Johnson introduces her at the press conference when the web app goes live.