Debugging like a Detective

Last week we received an email from a DoneDone customer who was experiencing a strange file upload error. We attempted to reproduce the error with no luck, so we asked the user to notify us if it happened again. They replied that the error had disappeared. Case closed.

A few days later, a second customer reported the same error. "Uh oh," we thought, "there's a killer on the loose." It was time to open an investigation.

Debugging software issues, especially those reported by your users, is much like street-level detective work. You need to interview witnesses, gather evidence, establish the scene, re-enact the event, and ultimately solve the mystery of what happened and whodunnit. Here's a walkthrough of a typical debugging case, and how each step parallels traditional detective work.

Interview your Witnesses

Always start with the people who witnessed the crime - in this case, the user or users who reported the bug. Ask them to describe what happened in as much detail as possible: Where in the application did the error appear? What steps did you take before the bug occurred? What operating system and web browser were you using? Can you provide a screenshot of what you saw?

Canvas the Block

Sometimes your witness may be unreliable, so you'll need to corroborate their story with other sources. It's time to look for physical evidence in your logs. Do you notice anything out of the ordinary that helps explain the user's issue? Does the same log entry show up multiple times?

Examine the Crime Scene

Now that you have your witness statements and forensic evidence, you can start reproducing the crime scene. Hopefully, the user experienced the bug using the same OS and/or web browser that you have installed, which will allow you to quickly attempt to reproduce the issue. If not, you may need to find a coworker with a similar setup, or get a VM that matches the user's environment.

File your Paperwork

Once you can consistently reproduce the issue, it's time to get everything in writing. Open a new ticket in your issue tracker, or, if a ticket already exists, add the steps for reproducing the bug. Include any important information you received from the user and the logs. Determine who should take the case, and assign the bug to them. Make sure to include any specialists who may have seen this behavior before - after all, they might notice something you missed.

Trust your Hunch

If you're assigned to the issue, you're probably already familiar with the neighborhood. You've probably worked in the section of code where the bug exists - you may have even created this monster yourself.

In any case, you might have a hunch - an educated guess as to what is causing the bug, and how to apply a fix. This is always the best place to start; even if your hunch turns out to be wrong, it will usually quickly lead you down the right path.

Build your Case

After a bit of trial and error (or a lot of it), you'll eventually find the offending piece of code that caused the bug. Now it's time to build your case and determine how best to apply a fix. If your update might be confusing to other developers, be sure to add a brief comment explaining your new code's purpose. When you check in your updates, try to be as descriptive as possible, and be sure to refer to your tracked issue.

Call in the Feds

Sometimes you'll get too close to a case - you might find that you don't fully understand how to correct the bug, or that the fix should actually be made by a different developer in another section of the codebase. Feel free to call a teammate for backup, or reassign the issue to the appropriate dev.

Get a Warrant from the Judge

Once you've arrived at a fix and have checked in your code, it's helpful to verify your work with another developer. Ask a coworker or manager to review your code before it gets merged into your primary development branch.

Bust the Perp

Merge your fix into the project's codebase and deploy it to production (or let the appropriate team member handle this). One more bug off the streets!

Time for a Lineup

Once your fix has been made available, follow up with the user who originally reported the issue. Verify that they no longer experience the bug, and once confirmed, mark the appropriate issue as fixed in your issue tracking system. Case closed! Time to head back downtown and wait for the next one to roll in.