In this article I want to explain the differences between using a contract-testing framework and a custom solution that I have presented before.
In the previous articles from below:
I have described how to create test cases based on services generated dynamically by reading the Swagger file.
By using this approach, a team designing a consumer service can create test cases against a dependent service and test basic CRUD operations: get a list of objects, create or delete an object.
These test cases are useful when the definition of a service’s API has changed and these changes are breaking changes: either it was a mistake introduced in the provider and the error it is caught in the consumer service or the break change has been introduced by intent and the consumer's test cases will have to be modified to accommodate the change and use an incremented version of the service.
"Contract testing" allows 2 services that are in communication to be implemented by adhering to a contract.
The testing framework creates automatically:
the stub service used to test the consumer
the test cases used to test the provider
at a time when the services are not coded yet and only the contract is known.
The contract it is written into a groovy file as the one presented below and contains the knowledge about the business logic a service needs to implement and test it.
In this example, the contract can contain 2 separate requests that both are returning a status code of “200” and a result of “Granted” or “Denied” based on the user card type: “Gold” or “Plain”.
The intent designed in the contract can be tested as such:
by the consumer service, without the need of a provider service to be created.
by the provider, at the time when the provider’s team will write the functionality described in the contract.
The framework generates automatically the verification test cases to test the provider - see below:
What is the difference between the 2 approaches presented above: the contract-testing framework approach and the custom solution?
The custom solution presented in the previous articles reads the Swagger file that is agnostic of the business logic which needs to be tested by a service.
This solution implements a “schema-based” testing as opposed with a “contract-based” testing.
“Schema-based” testing allows an API to be called if it respecting the API definition / schema and makes sure that the breaking changes to this definition are caught.
The contract in this case is the API definition / schema.
“Contract testing” is more than just respecting an API definition, it describes the API’s business logic.
The contract in this case is the API’s functionality and test cases can be extracted from it.
Both services developed, the consumer and the provider of the functionality, are forced by the framework to respect the contract and the business logic defined during the contract definition.
One team cannot miss creating a test case that has been implemented in code by the dependent’s service team.
For the custom solution, there is no way to enforce the consumer service to implement a test case required when adding new functionality in the provider service - other than the teams doing the work diligently.
On the other hand, to allow the ability of the custom solution to produce various result sets, we have to work on defining scripts as it is the case with the Imposter's implementation: in time, this effort will grow and became a small framework on its own similar - in some respect - with Pact and Spring Cloud contract frameworks.
Your choice regarding what solution to use it is given by what type of tests you need to implement: structural (schema-based) or semantic(functionality-based) test cases, validate the OAS schema or the services communications scenarios.
I will add one more observation - relying on the work of other teams of people who invested time and effort to create a framework with a lot of functionality provided out-of-the box, can be a good thing to consider in most of the cases.
References:
More details about “schema” versus “contract” testing can be found here:
