Are you and your company suffering from the side effects of spending too much time on debugging tasks?
We asked some of the engineering teams that we work with how much time their developers spend on debugging processes. The answer? On average, 25% of developers’ time is spent on debugging alone. That means that more than a full day of the work week is dedicated to troubleshooting and problem solving.
In a previous post, we looked at 5 of the top areas where this time is spent (think searching through log files, reproducing errors, “war room”-type situations, etc.). Now, we need to talk about how all of this time spent debugging impacts your company.
Let’s dive in.
— OverOps (@overopshq) November 14, 2017
Relying on User Reports and Complaints
No company wants to rely on user complaints or, worse, reach out to the users and ask them for more information. Without visibility into the user’s actions in production, it’s almost impossible to reproduce and solve issues before they impact other users.
Comcast’s team in charge of the XFINITY X1 Platform now uses Automated Root Cause detection to reduce damage to user experience caused by hard-to-solve errors in production. John McCann, their Executive Director of Product Engineering, said before that, “[debugging] was a tedious process, that involved manual effort from our team. Since there are millions of devices that run our application, pinpointing a single error or trying to reproduce it takes up too much time and resources.
When issues hit production, it would impact our customers and it was up to us to try and figure out what went wrong and how to quickly fix it.”
One challenge that many companies face, especially those struggling with error resolution time, is that users don’t report issues that they experience at all. Instead, they just stop using the service or application. The fact is, with traditional or manual debugging processes, by the time developers get the information they need to resolve the issue, the customer has already been lost.
Delayed Product Roadmap
For companies with hundreds of developers working on releasing new code every day, new errors are inadvertently introduced to their application as well. This explains how they end up spending so much time on debugging, but not why.
Clearly, the more exceptions there are, the more time needs to be dedicated to handling them. Right? While in some sense that IS true, traditional debugging processes force engineering teams to focus way more time and energy on error resolution than they should.
An unfortunate side effect of spending all that time on debugging (remember, 25% of your time means more than a full day out of the work week), is that teams have less time to dedicate to advancing the product roadmap.
A full day of the work week, a day that should be spent clearing the backlog and creating new features, is wasted on manual tasks and debugging. There are endless possibilities of what can be done with an extra day of work. Imagine the goals that you and your team could accomplish, if it weren’t for the delays caused by manual debugging.
One thing that we hear from almost all of our customers is that they simply spend too much of their time looking for the source of a problem before they can start trying to fix it.
Until recently, there was never a tool that could automate root cause detection and analysis (Now, you can try OverOps). Digging through log files was the only way to find the source of the issue. Without an alternative of any kind, spending anywhere from 5-50% of your time debugging is easily justifiable. What else are you supposed to do?
When something seems normal, it often passes by us without a second thought. Just like the obscene amount of time and money that’s wasted with manual debugging and troubleshooting. If you crunch the numbers, you may find that you are potentially wasting millions of dollars on your debugging process every year.
In fact, we heard from one company that between 100 developers spending 20% of their time and 21 support engineers spending 50% of their time on debugging tasks, they’re paying the equivalent of 30 developers at $200K a year for the debugging process. That adds up to $6 million per year in potential savings!
It’s not hard to do the math on this. With 100 developers spending 25% of their time on debugging tasks, that’s the same as paying 25 developers a full-time salary to do manual labor and hopefully solve the problems that come up. That means millions of dollars a year spent on debugging labor alone.
25% is a lot of time and resources masquerading as a minority percentage. Twenty-five percent is more than a full day out of your work week. Twenty-five percent means adding another full-time employee for every 3 developers. Twenty-five percent means potentially millions of dollars wasted every year.
Employee Happiness and Productivity
Considering the office atmosphere and contentment of employees is, of course, another important factor to take into consideration. We recently reviewed a study into the causes for unhappiness in developers. And the most significant factor? Being stuck on a problem.
Nobody likes to feel stuck or like they can’t progress from where they are. Unfortunately, this is one of the most common states for developers to be in at any point during the day. It’s basically part of the job description to come up with solutions to some of the toughest problems that have been thought up.
There’s still something to be done, though. Debugging is one of the most time consuming tasks that developers face, so if we can decrease that time we’ll be happier already! Well, that’s actually why we created OverOps in the first place. We saw how much teams struggle with the debugging process, and we want to give them what they need to cut down the time it takes.
When it comes to debugging in production, most of the time that’s wasted is spent on digging through log files and trying to reproduce the variable state from the moment of error. To reduce the time that it takes to solve the error, it helps to automate the process. OverOps gives developer teams Automated Root Cause identification including the exact variable state behind any exception, logged error or warning regardless of whether or not it was printed to the log file in production.
Cut Down Your Debugging Time
The bottom line is that there are many negative implications of dedicating too much time to manual tasks like sifting through log files. Tasks like these are just slowing us down.
Rather than needing to waste hours digging through log files or trying to reproduce an error, developers should be empowered by automated root cause detection. Can you imagine how much money you could save and how much happier you and your team would be minus those sleepless nights and endless days just trying to figure out what went wrong?
We can’t get rid of debugging and exception handling entirely. After all, handling exceptions isn’t always just about fixing a problem, it’s also about understanding your application in production. With OverOps, application visibility increases and time spent identifying and solving production errors within Java applications is cut down from days to minutes.
Debugging is definitely necessary, but letting it take up so much time ISN’T! That 25% adds up quickly. It adds up to a full-time salaried developer for every 3 working on new features and projects. That’s crazy!
It’s time to start automating the debugging process so that we can get out of the war room and back to your backlog. Interested in automating your own debugging process? Download our new eBook, The Complete Guide to Automated Root Cause Analysis, to learn how to get started!