Through our 20 years of providing managed services, we are often called in for project rescues and major fixes. Our team recently began working with a client who was hoping to get more out of their managed services partner and their solution.
The environments landscape, the code, and the release methodology all required major improvements, and our team had to deal with more than a few obstacles in the process, including one hidden surprise.
A number of shortcuts had been taken to manage code and release management, the deployment process was manual, and the testing process was unclear and poorly documented. To add to that, production deployments lacked a basic procedural mechanism to allow bringing servers in and out of the pool on demand.
But we love a good challenge! The team worked more than a few late nights getting to know our way around the implementation while addressing these issues.
The end-result of our initial wave of effort is a first successful Production deployment of an improved code base, and a number of improved outcomes:
- Automated deployments.
- Documented and trusted test cases.
- Solid procedure for production rolling restart deployments.
Additionally, Business and IT have a better understanding of their environments and how to use them. This gives them more confidence in the release management process.
All that leveraging our trusted processes and tools to get the job done while increasing overall performance and stability. But the hidden challenges weren’t over.
After our first deployment we discovered some manual fixes that hadn’t been pushed to the codebase, and were reverted with the deployment. These crushed our server several hours after deployment, and were not something we could have identified in lower environments prior to the production deployment. While we fixed the issue within 20 minutes of realizing the problem, it means more work fixing the deeper root cause with the next deployment.
With their new deployment process solidified, we will now be making numerous enhancements to reduce the time needed to maintain the site for both our team and the client’s, to make life easier for both teams.
The result is a very happy client, a much more stable site, and ultimately a better experience for their customers!
If you’d like to discuss how SMITH’s Managed Services team can make your customer experience better, we’re just a click away.