Often when we first engage with organizations, we find they enter the conversation with a clear idea of what their problems are. Sometimes they get it right and other times - more often in my experience - they are focusing on their own belief of where the problem lies.
For example, if the problem is the deployment process, why does the automated script take 5 minutes to run. Having successfully worked with development teams to automate deployments of their major platforms, being told deployment is the issue seems like the wrong place to focus. If it still takes weeks to get code into production, the problem lies elsewhere. Perhaps our test verification takes five weeks?
Ok. Well, if deployment of code isn’t the issue and testing is, let’s focus there I hear the cry! Well, let’s see…
Unpacking
The first week is spent getting the right data in the right place to execute the initial test run. The first two deployments to these new environments fail due to patches applied to the underlying systems which updated the local firewall rules. Now ready to test, week two flies by through a series of manual and partial automated tests. The development team is now the hold up as they are onto their next sprint but being asked to resolve issues from the previous one. The test team needs their help to resolve why the application won’t deploy (of course they are using different scripts to the ones written for production deployment). And it goes on…
I was once speaking to a solution architect who claimed to have “nailed DevOps” for his application team. Walking through the testing process, it quickly became apparent that every time they ran performance testing, the operation failed at least twice. By then, the results they got were so late, nobody looked at them. The test cycles were, for the most part, not failing due to the test cases themselves, but due to the complexity of the environments.
So what’s an approach to identify the right problem faster?
There are several issues described in the examples above:
Preparing environments for test purposes is difficult and often, the environment is not ready for testing or contains errors.
Usage of different deployment mechanisms across environments introduces errors as there is insufficient testing of the deployment itself.
Lack of visibility into the process means that focus and effort are being put into the wrong place.
With so many issues in a delivery process, it can sometimes be hard to see where to focus first. Asking questions and conducting, even simplified, value stream mapping exercises can significantly help. As I was talking to last week, prioritizing fixing the delivery process itself is paramount to the delivery of features. Start asking why do we need those complex environments to validate our changes? How else might we reduce that risk?
Smaller changes are easier to validate, especially when I can quickly delineate the impact of the change from all other parts of the system. Learn that your only environment that looks like production, is production. Combining technologies like feature toggling with active load balancing can allow for this. It would help if you also built a deep understanding of the system dependencies to create abstractions where needed. To truly make that work requires building the right team.
On to the next!
It is essential to understand that this is an iterative process. This is one of the reasons a product mindset works better than a project mindset. A project ends, whereas continually improving how you get value to your customers does not. Even once you have made it through all of the issues described above, more problems will arise. This is the nature of the complex systems we work with today.
Summing up
Focusing on the right problem in your value chain is critical to continually improving. If your focus is off, it can be akin to moving deckchairs on the Titanic; it may look like to are achieving something, but you are ultimately liable to hit an iceberg.