Java Staging

How come there are companies who skip staging and push new code straight to production? 

The classic model of developing software applications has several environments: development, integration, testing, QA, staging, and finally production. In practice, particularly today, several of these environments are streamlined or done away with altogether. As the software world has evolved, it’s worth revisiting some central assumptions about the development models we use. Today, we’re looking at one environment in particular: staging. Many developers are forgoing it anyway and taking on the risks of testing directly in production, so the question is: do we really need staging environments anymore?

To help ruminate on this question, we spoke with Avishai Ish-Shalom from Fewbytes (who also shared his insights with us on my earlier Docker post). What follows is drawn from that conversation.

With the disclaimer that there is no one-size-fits-all development model, here are five reasons to reconsider your staging environment:

1. Staging environments were designed to solve a need that often no longer exists

Before we really dive into the question itself, let’s look at why staging environments exist in the first place. When software applications was shipped in boxes, there wasn’t a controllable production environment to test or gather data from. To get around this, staging environments were created. Staging environments were supposed to exactly mimic production environments, in order for you to deploy your app and test it in “real life” conditions. Ultimately, the whole thing is a simulation, but a simulation can still tell you something valuable.

BEST.SIMULATOR.EVER

With the advent of the Internet (and error reports), developers were first able to start getting actual data from production. It wasn’t real time and it was passive rather than active, but it still began to chip away at the value of staging, since it was real and true data. As the world has moved to a SaaS model and always-on connections for software, there is now a rich and full production environment that is often quite available to gather data from. Simply put, the original driver for staging environments no longer exists. Especially as modern developer teams have moved away from the waterfall model where they didn’t even have access to their own production environment.

2. Staging environments require significant upkeep

Today’s deployment environment is fast-paced and continuous. Your fixes may come out monthly, bi-weekly, or even more frequently. There’s often no time to set aside for testing a fix to a production bug in staging, and deploying straight to production is a reality for many developers. However, if you’re not careful, your staging environment can start to slip. That tiny fix you deployed straight to production and forget to later deploy to staging can break something in a bigger version you deploy a month later.

Beyond just version upkeep, staging environments require a careful eye, monitoring, and testing of their own. As they’re supposed to be mirror images of your production environment, staging environments are often deployed on the same servers or some locations as production environments. That creates a risk of leakage, which requires frequent testing to monitor.

Find the Crap in Your Java App

Show me how >>

Fred

3. The data in staging is simply not as good

The main limitation of staging environments, no matter how exactly they mimic your production environment, is that they aren’t handling real, live interactions. When simulating interactions, it can be nearly impossible to accurately simulate what your production environment will go through. High loads, corner cases, weird user actions, and many other scenarios that pop up in production are difficult to generate in staging. And if you’re fortunate enough to be working on a JVM based language, you can actually get all the debugging data you need in production as well.

Depending on your business, you might not be able to deploy 100% of features in your staging environment anyway. For example, in a financial business, your user interaction is going to be credit card transactions. That’s not something you can simulate, unless you’re willing to set aside some budget. And that brings us to #4.

4. Staging environments are expensive

If you have a staging environment, it can nearly double your costs. To get anything out of staging, you have to make sure your staging environment mirrors your production environment as closely as possible. That can mean doubling the number of servers you have, doubling the bandwidth, and doubling engineering time. This is a challenge that holds true for both startups, where money can be tight, and for large companies, where issues of high scale come in.

5. Staging environments are hard to scale

One of the biggest challenges in development is building environments that can handle high loads and scale that can come in production (if you’re lucky). Staging environments are no exception. Systems at high scale behave differently than those at small scale, even when everything else is the same. There are limits to what you can achieve with a staging environment solely because of scaling challenges. Simulating the scale of your production environment is often unfeasible or simply impossible. There are obstacles around physical infrastructure, traffic loads, and types of interactions, none of which are easy to overcome.

Conclusion

So to recap, staging environments are expensive, difficult to get accurate data from, and based on an outdated need. Does that mean you should do away with them altogether? To answer that, you have to answer two questions first. One, are you able to accept some loss? Not every app can afford to take the level of risk or losses that can come from testing completely in production. For example, if you’re building an app for an operating room for patients, you can’t exactly test in production, and no loss is acceptable. In such a case, you absolutely will need a staging environment.

Operation Game

The second question is, are you able and willing to test completely in production? Full production testing is possible for many apps, but it requires planning, good practices, and a set of dedicated tools for the job.

Keep an eye out for my followup post, where I’ll be exploring some methods and considerations for testing in production. Want a sneak preview? Then check out one tool that’s extremely valuable for testing in production: Takipi.

fredjava

Level up your production environment with 15 tools you can use starting today – View tool list

email
Josh does product marketing for OverOps. He's a big baseball fan and a small beer nerd.
  • Heath Malmstrom

    As someone who has been doing devops before it was a buzzword, makes extensive use of cloud computing for a SAAS company, and develops SAAS products on a daily basis, I can’t disagree more with the direction of this article. With a small amount of effort staging environments are a cheap, low maintenance way to insure problems like programmatic configuration and data changing bugs don’t impact production. In addition this is a great chance to test artificial barriers for scaling up web applications without the continuous work churn occurring in your Dev environment. If you’re using tools like chef, puppet, saltstack, or ansible your environments should be totally reproducible and easily configurable (see AWS Opsworks or other similar services).
    With cloud server commoditization this is really cheap and you can spin up spot instances just when needed if your smaller. All of your points I would disagree with in a cloud based environment, and if this is how you write your software it would give me pause before being one of your customers.

    • http://www.allthingsdork.com Jeff Smith

      Agreed. Configuration management makes up keeping staging as simple as production. The same tools I manage production with manage staging. Not to mention it’s where I as the OPS representative test my system changed.

    • Josh Dreyfuss

      Thanks for your thoughts. This was more of a food-for-thought kind of post, and definitely doesn’t hold true for every environment. In the AWS cloud-based environment you’re talking about, you don’t have to think about replicating the physical environment, so a bigger concern would be equivalence of data than issues like hardware costs, for example.

    • http://www.fewbytes.com/ Avishai Ish-Shalom

      let me take your point about cloud further: since it’s so easy to provision new environments, you can (and probably do already) provision a test environment ad-hoc for testing configurations, versions, features, etc. so which is your “staging” environment when you have a changing number of identical environments? why do you even need one when you can have “staging environments” (plural)?
      Another point is that as your service growth and scales further, emergent behavior is observed. Scale and complexity makes things that are supposedly “identical” behave in a radically different way. A 20 node hadoop cluster cannot be used as a staging environment for a 500 node cluster, it exhibits a very different behavior. At most, your 20 node cluster can be a used for smoke testing and development, assuming you can even get insights without complete datasets which are simply not economical to duplicate.

      I’m not saying you shouldn’t have a staging environment, far from it. I’m saying that like many other practices they only cover 80% of situations and you need understand if/when you’ve diverted into the other 20%.

      • Heath Malmstrom

        I mostly agree, however these are pointedly not duplicate environments. Development requires integration with a different set of (future) configuration and service integrations, staging is a reflection of production with your intended next available push to prod (multiple change sets that are intended to be a next candidate for deploy). Staging is just there to catch integration and breaking configuration or data changes due to some of these necessary differences between a development environment and production. Cloud isn’t necessary by any means, and “staging” as I see it could be a virtual environment on a build machine even if you have a trivially simple product (who does though).

        • http://www.fewbytes.com/ Avishai Ish-Shalom

          And the further we go into the world of CD, micro-services and SAAS based infra the more this becomes unrealistic. In a big micro-services infra it is very hard to do integration this way, you can have 30 teams all doing CD to a 200+ micro-services not including external services which are also doing the same. Your staging environment (and your production) are in constant flux and (re)integration. You could, and probably will, have a test/integration environment for your own little piece of the puzzle – the micro-service you are working on. But the concept of “integration” environment with “the next version” of all services begins to crumble very quickly when you start doing micro services and CD.

          • Heath Malmstrom

            This in my opinion is the most ideal situation to have a staging environment. Without it using microservices is going to be pretty error prone (the more services the more likely you need a staging environment), because you need to verify clean interaction, especially on events like deploy/provisioning of new and existing services, or erroring due to capacity and other environmental constraints. Versioning issues are one of many class of problems that are hard to test in a dev environment, but having staging makes this an easy checkmark before moving to prod. Marshalling data changes as well. Both common in microservices. These are the kinds of problems that are notoriously difficult to catch via programmatic testing, but crossing a version threshold on push to staging is a great litmus. In addition any one using a relational database will attest to the value of staging for the same kind of schema migration check.
            For managing 200+ micro services I would take a look at some of the great deployment tools mentioned above. We have only dozens of services, but these make them easy button to manage.

          • http://www.fewbytes.com/ Avishai Ish-Shalom

            I think we agree on principle, although I would prefer testing in prod (and yes, there are safe methods to test with production traffic) rather then synthetic tests on a disconnected environment. I also think different companies have different definitions of what “staging” is.

  • vsync

    and where should you run your unit tests? I would rather push to staging, and then staging itself will run all the tests, and if they pass, the code will be automatically deployed to production. I just want to be able to push push & push more without running the tests on my machine each time I save a file.