Introduction

As part of a project with one of our major customers, we had a requirement to import large amount of data from an excel files to our backend systems. The entire process involves multiple interconnected activities.

High level Scenario

The excel file will be uploaded by admin users to a designated SharePoint site folder. The excel file must be validated for errors and if no errors, the data records in the excel file will be processed and import it to the database. After importing the data, we must index all those records to elastic search for complex searching functionalities. After indexing records, the system must upload several different types of documents from SharePoint and associate those documents to respective records. At every stage of the process, information notifications, error notifications, success notifications of each activity to be sent to teams’ channel and via emails.

A high-level flow of the entire process has been depicted in the following flow diagram.

The Problem

Since this is usual scenario in traditional backend systems, at first look, you might feel that this could be done via workflow engines like Logic Apps (Azure), Step Functions (AWS), Cloud Composer (GCP) or other third-party workflow engines. But it is very hard to build and manage such massive workflow because of the following reasons

  1. Most of the activities may take longer time to finish. Especially step 3, which handles actual data processing and importing to database, may take several hours to complete. Calling such long running process from Logic Apps using HTTP is a bad idea. Without proper call back mechanisms, it is impossible to implement the flow. Even with callback, it would be very tedious to implement and manage the flow by polling the status in regular intervals for all these activities.
  2. Email sending, Teams message sending, Error handling are needed in many places. Email configurations such as Subject, Body, To address are configured for each activities. Teams Messages are also configured for each activity. If we use a single monolithic workflow, then it needs to repeat these activities in lots of places.

The Solution

Best solution is adopting “Event Driven” approach. Making the whole process event driven has the following advantages.

  • It helps achieve pure asynchronous handling of all these activities.
  • De-couple all the activity components based on its responsibility. Remember Single Responsibility Principle
  • Since it is decoupled, we could independently test and verify each component.
  • We no need to worry about timeouts usually happening in synchronous HTTP calls.

In this approach, all the activities such as Data Validation, Data Processing & Import, Elastic Indexing, Teams message handler, Error handler, Email handler etc. are independent components which is triggered by “Event Messages”. All event messages are sent to Azure Service Bus queues. Activity components such as Logic Apps and Azure Functions are triggered by event messages posted to its respective message queues.

Let me explain the de-coupling in more details by taking some of the activities as example here and show you how it works together independently in a pure asynchronous fashion.

Activity A invoking Activity B Example

In this scenario, Activity A, which is a long running process needs to invoke Activity B once it is finished. Instead of calling synchronously via HTTP endpoints Activity A post a message to the service bus queue with payload containing required information. Activity B triggered by the message asynchronously.

 

Error Handling Example

All activity needs to catch errors and send the error log via email. In this case, we have a common error handler module. The activities need to report errors, simply post a message to the service bus queue with a payload containing required information.

What error handler does is generic. Based on the information such as activity code and error code it fetches the error log from the database and create a csv file and store it in blob storage. The errors are already inserted in the database by the calling activity with a unique code. To email the errors, it posts a message in the service bus which is picked up by email handler.

Email Notification Example

Most of the activities needs to send emails. In this case, we have a common email handler logic apps which uses SendGrid API for sending emails. The activities need to send emails, post a message to service bus queue with a payload containing required information.

What email handler does is also very generic. Based on the activity code and template name it fetches the email configurations such as subject, body, to, cc etc. from the database and if has attachment set to true it attaches the filename also with the mail.

Teams Notification Example

Most of the activities needs to send message to a special team’s channel. In this case, we have a common team’s message handler logic apps which send team’s message. The activities need to send teams message post a message to service bus queue with a payload containing required information.

What team’s handler does is very generic as well. Based on the activity code and template name it fetches the team’s configurations such as icon image, subject, text etc. from the database and send message to the team’s channel.

Conclusion

As you can see this with the examples, event driven approach helps us easily decouple all the activities in the entire process and orchestrate the flows. It is more manageable, maintainable, and independently testable.

If you wish to know more about the details of entire automation pipeline, please feel free to contact us.