Last year I worked on several Cloud integrations between various vendor applications used by an educational institution. These integrations make use of a disconnected architecture where a service bus is located in the middle and 2 or more Azure Functions read and process the data between the source and target systems.
The most versatile functions in Azure are Azure Durable Functions. These functions were designed to perform a few things:
store the execution's history within their storage and queues allowing them to be restarted and continue the process from where it left off in the eventuality of a Cloud virtual machine restart.
execute several activities in parallel based on a fan-out/fan-in design pattern: more work coming in makes the creation of more activities as part of a single orchestration process. However, each activity is treated as a separate function in the Cloud and can be spawned on a different virtual machine to improve performance.
Click on the link below to see my previous article describing the internal storage of Azure Durable functions.
As part of implementing these activity functions, it arises the need to perform and coordinate transactions against the target system. For each service bus message that has been processed the transactions state need to be consistent:
all transactions have been processed successfully
some transactions have been processed successfully, some failed - in this case, rollback all transactions successfully and re-try at a later time (presumably after the issue has been fixed by the vendor)
same case as the previous one but some rollback failed as well - in this case, keep track of these messages separately and send them to a separate queue for retry at a later time. At the same time record the overall transaction state at the message level in the queue.
The intention was to create a small framework that does the following:
enroll the activities performing transactions in a process that will be processed sequentially, in an orderly manner, or in parallel
coordinate the commit and rollback of all transactions without the intervention of the developer.
log the state for the overall transaction
re-try the failed queues at a later time, until all the transactions have been processed successfully.
A visual representation of this process is displayed below:
The logged message indicating the overall transaction's state is displayed below:
A few implementation details for my first version of this POC are presented below:
1) The project's structure is displayed below, each activity corresponds to a transaction:
2) The "SagaOrchestrator" function performs the following actions:
3) Each transaction is a class that implements "execute" and "cancel" activity functions that are called by the framework.
Note 1:
Each transaction class contains a similar code: the difference is in the name of the functions.
Each of these functions is failing on a particular message's count:
Transaction1 for the 3rd message
Transaction 2 for the 5th message
Transaction 3 for the 7th message
Note 2:
The generic nature of these functions is based on a naming convention for all the transaction's functions involved, similar to the way the Controllers were implemented in ASP.NET MVC:
Transaction1Orchestrator, Transaction1OrchestratorCancellation, ExecuteTransaction1, CancelTransaction1, etc.
4) There are more implementation details I will skip for now
Testing the solution:
I have modified the QueueSender utility I created previously when testing the functions' performance and sent 10 messages to the service bus, with a message from 1 to 10.
The messages numbered 3, 5, 6, 7, 9, and 10 will fail. For message 7 the rollback activities for transactions 1 and 2 will fail as well.
The results are displayed below:
Transaction Order
I have updated the code to allow the sequenced transactions to be enrolled and processed in a specific order. The transaction rollback is happening starting from the higher transaction's order number down.
The order in which the transactions are executed is visible in the DeadLetterErrorDescription:
If you are interested in finding out more details, please let me know.
Cheers all!
Comments