This is my last blog post in CI/CD setup for Azure Data Platform. First two parts can be found here:
Data Platform in Use:
The Data platform consists of Data Lake, Azure Data Factory, Azure Databricks, Synapse Analytics and Power BI. Here’s the high-level architecture:
What is Azure DevOps & Why?
1a. What is Azure DevOps?
Azure DevOps provides set of tools and practices to collaborate across teams within the organization to build, integrate and deploy applications/Services.
1b. Why Azure DevOps?
We have opted DevOps component in Azure for the following reasons:
- Easy and tight integration with Azure native components such Azure Data factory, Synapse Analytics, Azure Databricks etc.
- Maintenance-free operations
- Allows to manage security roles and groups
- Elastic scaling
- Azure DevOps Organization and Project should be setup and configured
- Git Repository is already setup in Repos section of DevOps
Azure Synapse Analytics CI/CD with Azure DevOps:
1. Synapse analytics follows similar steps to ADF with minor changes. A project should be setup with an associated git repository in Azure DevOps for Synapse.
2. Navigate to synapse workspace and open Synapse studio. Connect to Git by configuring
Studio -> Manage -> Source control -> Git Configuration
3. Then you would notice that synpase is connected to the master branch:
4. Create a feature branch to create/modify synapse components and then click on commit all to save changes to the respective feature branch.
5. You can merge the changes to master branch before clicking on Publish button.
6. Click Publish to propagate the changes to Live Mode as well as to generate ARM templates in workspace_publish branch in Git.
7. Switch to Azure DevOps to continue with the release process as Publish in synapse studio has taken care of CI part.
8. Click on Releases under Pipelines section to create a new release pipeline
9. Connect to workspace_publish branch from Azure DevOps git to gather artifacts built:
10. Add a stage in which ARM Template deployment should be configured with all required details such as Deployment scope, subscription , resource group etc
10a. While Selecting Azure Resource Manager Connection, It prompts to Authorize and creates a service connection to Azure Data Factory.
- Sign in DevOps as the owner of the Azure Pipelines Organization and the Azure Subscription
- You don't need to further limit the permissions for Azure resources accessed through the service connection.
- When you click on Authorize button, It will Automatically create a service connection.
11. Templates need to be configured with the default template files created in workspace_publish branch.
12. The populated parameter values need to be replaced with Production specific values.
13. Enable continuous deployment trigger to continuously poll for the changes done to workspace_publish branch and invoke the deployment.
14. To have better control on releases, enable pre-deployment conditions as well:
This concludes the CI/CD for Synapse Analytics using Azure DevOps.