Every software project will, at some point, ship a major bug. It happens. No amount of testing, code review, or process can entirely prevent it. What you do when that bug hits your users – how you respond, communicate, and recover – matters just as much as the bug itself. A poorly handled bug can erode user trust, create security risks, and cause far more damage than the original issue.

In this article, I'll talk about what goes wrong when a fix exists but fails to reach users, why those decisions cause additional problems, and outline a workflow you can follow to handle critical issues properly in your own projects.

A recent example

A recent release of a well-known piece of software introduced a major bug. The specifics of the project or the bug are less important than how it was handled. What matters is that it was significant enough to completely break some users’ installations, it was reported almost immediately, a fix was proposed and merged within days. Still, a few weeks later, thousands of users remained affected by a known issue with a known fix.

The maintainers deemed the issue important enough to add a note to their blog post announcing the affected release, including a link to more information and a “hotfix” patch. However, the broken version remained live and reported as the latest available release, still proposed for download and updates without a warning when doing either. The fix existed in the codebase but was not released as a new version. Users who updated during this window received the broken version automatically, with no warning that anything was wrong.

The problems with the way it was handled

The issues with this response can be framed as a trifecta of issues which do not each individually constitute a major failure, but when put together create the perfect storm: no official fixed release was published, information was inadequately surfaced, and there were unintended security implications.

No new release

Two weeks after the bug was discovered and confirmed, there was still no fixed version available. The affected versions remained live as the latest releases. Anyone installing the software fresh, or whose site auto-updated during this period, would receive the broken version.

Given that modern software like the one I am looking at often includes automatic updaters, it was entirely possible for users to update to the broken version without their operators knowing. Since the software itself provided no indication that an automatic update had occurred – let alone that it updated to a known broken version – users were left confused and frustrated when their sites suddenly broke. Non-developer users, the majority of the user base, would waste time and money trying to diagnose and fix something that was not their fault. This breaks user trust and creates a compelling reason to avoid the software in future.

Information not surfaced properly

The information about the known issue was added to the affected version's release announcement. It was not surfaced on the download page, or as a special call-out in the update page – even though the capability for that does exist in that software.

To make matters worse, many years of anodyne, cookie-cutter release announcements had taught users to expect the same boilerplate text, maybe a thank-you to contributors, and a changelog. Users were conditioned to ignore release announcements. Putting this information only in the one place people have learned to ignore by default is about as useful as not publishing it at all.

While not being the maintainers’ intention, this felt like someone checked an item off their list, or fabricated an excuse to deflect user complaints about lack of communication. In some ways, it is worse than not communicating at all.

The hotfix created security concerns

The hotfix was not distributed as a full release, not even a semiofficial release, or even a number of files to be installed manually.

The software in question has a mechanism to install third party code to extend or modify its functionality. The hotfix was delivered as such a code package, with a dangerous twist: it would patch (modify, not replace) the core product's files, then uninstall itself without any user interaction. 

There would be no indication that the hotfix was applied, making it harder for installations managed by more than one person to know this is the case. There is no way to remove the hotfix in case something goes wrong; it would require at least a partial reinstallation of the product to return it to “stock” condition.

This is a significant security concern. It is functionally identical to a malicious code package (file-dropper). The project was conditioning users to follow an unsafe practice unquestioningly, going so far as to label it ‘safe’ in official documentation.

Replacing core product files runs counter to fundamental security advice. Those of us dealing with software security have long preached the sanctity of the immutable core product throughout a patch release. Moreover, this particular project has strict controls penalizing third parties who modify the core product instead of extending it cleanly. Having the maintainers of the product violate their own rules sets a dangerous precedent. At best, they are teaching users to ignore the problem of the core product having been modified, the loudest warning they would have in case their installation is compromised.

How it could have been handled

A major bug slips through and gets reported. Unfortunate, but it happens. What matters is what happens next.

Once confirmed, the first step is carrying out an impact assessment. This bug broke perfectly functioning installations with no practical workaround. This is an impactful bug warranting an immediate response.

When a critical bug is discovered and confirmed, the first action should be to pull the downloads and updates. As a maintainer, you want to minimise the blast radius of the critical issue. Do not let the affected user base increase while you work on a fix. This was not done.

Next, triaging: determine why the bug occurs and where in the code it originates. This was done.

After triaging, maintainers need to develop and verify a patch. This was also done.

Once a working fix exists, maintainers need to apply it in two places. First, the planned next version of the software. This was done. Second, a branch should be created off the released tag and the patch applied there as well for imminent release. This was not done.

At this point, maintainers should make an emergency release, bumping the patch version (x.y.Z) as per Semantic Versioning guidelines. This immediately rectifies the issue on affected sites and ensures that people downloading the software afresh or updating now will not be affected. This was also not done.

Finally, a post-mortem should be written and published. This has also not been done at the time of this writing.

How you should handle major issues in your software

The example above and how it should have been handled gives us a framework for responding to critical bugs. It's not a matter of if this will happen to your project, it's only a matter of when.

Here's the recommended, tried-and-tested workflow:

• Confirmation. Upon receiving a bug report, confirm it stands. You may need to ask for clarifying information to ensure it is reproducible.
• Impact assessment. Once verified, assess the impact. The rest of this list applies to impactful and security issues.
• Pull downloads and updates. Minimise the blast radius by rescinding downloads and updates to the affected version. Do not let the number of affected users increase while you work on the fix.
• Triage. Figure out why the bug occurs. Isolate the affected code.
• Fix and fix verification rounds. Fix the affected code. Test your fix. Ask others to test it. If you did not get it right the first time, try again. Affected users who filed bug reports may be very keen to help verify the fix.
• Apply fix to next stable. The fix should be merged into your main branch so it lands in the next stable version. This is your future version, not your immediate release.
• Apply fix to current stable. Fork off the tag of the last stable release and apply the patch there too. Triple-check nothing is broken as a result of your actions.
• Release new version. This is based on the fixed current stable, bumping the patch release number.
• Post-mortem. This is optional but highly encouraged for critical issues. Explain what happened and the steps you took to address it. A small minority of users may complain about frequent updates. The silent majority will appreciate your immediate attention and transparency.

Here is what not to do with an impactful bug:

• Shrug it away. If it is a major bug affecting users and you are perceived as doing nothing, you are breaking trust. When users stop trusting you, they stop using your software. A piece of software without users is called dead.
• Leave the affected version up. This will increase the number of affected users beyond the first wave. This is also perceived as inaction and breaks user trust.
• Release under the same version number. While it is annoying to have two or three releases in quick succession, it is better than releasing a fix under the same version number. Doing that would leave some affected users without a clear update path to a fixed version. Again, this breaks user trust.
• Offer a hotfix or semiofficial release. This is only reasonable when you are at the fix verification stage, and only for a small number of users you need to confirm the fix works. If you teach your users to mix and match files, they will eventually apply the wrong fix to the wrong version, or at the very least learn that they cannot reasonably compare the published release with what they have on their sites.
• Don't stick a note where nobody will see it as your sole mitigation. This is a cop-out. Sure, there is a public announcement where nobody will look for it. The perception – and quite rightfully so – is that you just shrugged it away.

We learn from past mistakes

I chose not to name the software or provide information about the bug because it’s not my intention to tear apart hard-working, unpaid maintainers for a legitimate mistake. I have seen the same happening, time and again, with software published by trillion-dollar multinationals with thousands of engineers. Those mistakes do happen, by everyone, all the time – and I will be the first to admit that I’ve dropped the ball in the past. Live and learn. That’s the point of this article.

As a software developer, I hope you now have some better context of how not to handle an impactful bug or security issue, and how to handle it properly. The world will be marginally better if we all learn to handle our software releases responsibly, despite our first instinct being to put out the proverbial fire. When all is said and done, it is better to have a small number of users with loud voices and no common sense begrudging you about "too many updates" than to mishandle severe bugs to the point where it breaks user trust and raises legitimate security concerns.