The story
One of clients I have worked for in the past, initiated a multi-year project for migrating services from an existing platform to a newer platform, using AWS as the Cloud. The intention was to perform a "lift and shift" as much as possible, with some architectural improvements for some of the services and the goal of creating a better process and user experience than it has been delivered in the original platform.
For almost a year, more than 10 teams worked on creating the services and the corresponding unit test cases. As the functionality for some of the dependent services was delivered in several Program Increments, integrating the services to provide and test a full end-to-end functionality did not happened until around the 10th month of development. A user testing environment was created but users were not on-boarded yet as the services development and integration work was still in progress.
During the development process, the architecture team provided a way to test services and their immediate dependency using an in-house custom solution - that simulates the contract's testing ideas - as part of the development pipeline.
One month before the MVP release, the teams ran into difficulties testing the integration of their services as it became clear that the workflow describing the service interaction has not been documented and made it clear to the teams.
As the accent was placed on migrating and deploying the services on the new platform, the planning phase of the Agile process which discuss the backlog, create the requirements and the acceptance criteria for the new features, initiatives, epics and user stories, was left up in the air. Product owners were not allocated to the teams and the only input received regarding the changes needed has been provided by the system architect during weekly discussions.
One month before the MVP release, the need for creating UML diagrams to describe the requirements, the services interactions and the overall process workflow became obvious, therefore a few people were asked to create them.
Test-Driven Development
Test-Driven Development looks more like a nice theory that is not actually applied in most of the cases when developing software. The well-know idea is: create the test cases first and let them fail. Next, implement the functionality in the code and make the tests pass.
In theory, the system should be built top-down as in the pyramid from above. Step 1 it is done to put in practice your system's design based on the intended requirements. These test cases will not pass for long time but they allow to define the system workflow and overall functionality.
Once the design and the workflow is understood, the next step is to write integration test cases for services that have not been built yet.
The last step will be to implement the right logic that will make the unit test cases pass.
The process will happen as follows: move onto the unit/component tests and start the red-green refactoring in the unit test layer - step 3, leaving step 1 and 2 in the ‘red’ stage and completing the unit tests to the refactoring stage. Then move back to the integration tests -step 2 - make the test green, then refactor. Afterward, applies the same process to step 1.
Suggested approach
All the services are performing APIs call in the backend, therefore they can be tested using a tool as Postman. At the same time, since we are using AWS, we can make use of AWS Step functions and build a fully testable workflow out of the services APIs, workflow that describes the system requirements.
Instead of starting the project's work from step 3, as it is the case in our story, why not spend enough time at the beginning of the project and describe the system architecture and test cases required by step 1 and 2? A comment I have received about this approach is that it is not agile enough as the design directions are constantly changing during the project. As a side note, my opinion is that a lot of people confuse the agile process with a lack of planning for the various phases of the process. (And a more phased approach is possible as well by running the process from step 1 down to step 3 several times, each time involving a smaller number of related services.)
Step 2, defining the integration test cases can be done by employing a testing framework - as is the case with contract testing - to make sure the services details (contracts) are defined, adhered to and implemented by both the provider and the consumer services.
If the first 2 steps of this pyramid were well designed and known by the teams, executing step 3 and going back up the path to close all the integration test cases and validating each workflow would have allowed the process to progress better.
Applying these ideas in practice will have required a certain amount of work up-front, in the designing phase of the project, to define the requirements and the services' workflow and interactions. It is easy to dismiss this work as "not agile", but good results are achieved if you have a vision and implement it consistently making sure that you constantly adapt to changes.
Another idea worth exploring is integrating the circuit breaker design pattern with the workflow. The intent here is to take a service offline and later on bringing back online and have the AWS step function visually displaying both states of the workflow.
To see an example of using AWS Step functions, read the article from below:
These are the AWS X-Ray traces for the AWS Step functions' calls (more details in the article indicated above).