Azure Data Factory is Microsoft’s cloud service for moving and transforming data between systems. It connects to your databases, applications, and files, pulls data out, reshapes it with business rules, and delivers it where it’s needed, on a schedule or on demand.
This guide explains how it works, when it’s a good fit, and how to automate it with code.
How Azure Data Factory works
A data factory is built from a few core pieces:
- Linked services: connections to your data sources and destinations, such as databases, APIs, and file storage.
- Datasets: the specific data you want to work with in those sources.
- Activities: individual steps, such as copying data or running a transformation.
- Pipelines: groups of activities that run together to complete a job.
- Triggers: schedules or events that start a pipeline automatically.
Together, these let you build repeatable data workflows: extract raw data from a source, clean and transform it, and load it into a destination for reporting or use in another system.
When to use it
- Data migrations: moving data from a legacy system into a new platform.
- Ongoing syncs: keeping two systems up to date with each other.
- Reporting pipelines: bringing data from several sources into one place for analysis.
- Hybrid environments: working with both cloud and on-premises data.
Azure Data Factory offers a large library of built-in connectors, works with source control, handles cloud and on-premises data, and charges based on usage. It’s a strong fit for organizations already working in Microsoft Azure.
Automating Azure Data Factory with code
Building pipelines by hand in the Azure portal works for a few jobs. When you need many similar pipelines, or want every environment built the same way, automation is the better path. Using the Azure SDK for .NET, you can create data factories, linked services, datasets, and pipelines in code, then run and monitor them programmatically.
Automation brings real advantages:
- Consistency: every environment is built the same way, from development to production.
- Speed: new pipelines can be generated from templates instead of built by hand.
- Version control: pipeline definitions live in source control alongside your other code.
- Repeatability: the same pipeline can be deployed for each new client, office, or data source.
Presentations and sample code
BRIO presented both topics at Philadelphia Cloud Technologies User Group meetups: an introduction to Azure Data Factory, and automating the creation and execution of data factories with .NET. The sample application’s source code is available in our DataFactoryAutomation repository on GitHub.
Planning a data migration or integration? Our data migration checklist walks through the process, and our data intelligence team can help design the pipelines.
Frequently asked questions
What is Azure Data Factory?
Azure Data Factory is Microsoft’s cloud service for moving and transforming data between systems using pipelines that can run on a schedule or on demand.
What is a pipeline in Azure Data Factory?
A pipeline is a group of activities, such as copying and transforming data, that run together to complete a data job.
Can Azure Data Factory be automated?
Yes. Data factories, pipelines, and their components can be created, run, and monitored in code using the Azure SDK, which keeps environments consistent and repeatable.
