📖 What is Azure Data Factory?
Azure Data Factory is a fully managed, serverless data integration service. It orchestrates and automates data movement and transformation processes for hybrid and cloud data warehouses. ADF enables building ETL and ELT pipelines to ingest, prepare, and load data from diverse sources into Azure data stores.
"Understand Data Factory’s role in building data pipelines. Exam questions frequently contrast ADF with other Azure data services like Synapse Pipelines. Focus on its integration capabilities and trigger mechanisms. Be prepared to identify appropriate use cases for ADF versus other services."
📚 Certification: Microsoft Azure Fundamentals (AZ-900)
🔑 What are the Key Concepts of Azure Data Factory?
- ▸ ADF uses pipelines, activities, and datasets to define data movement and transformation workflows, enabling automated data integration.
- ▸ Integration Runtime (IR) is a compute infrastructure used by ADF to execute activities; options include Azure IR, Self-hosted IR, and Azure-SSIS IR.
- ▸ ADF supports a wide range of connectors to various data sources, both on-premises and in the cloud, facilitating hybrid data integration.
- ▸ Triggers initiate pipeline runs based on schedules, events, or tumbling window patterns, automating data processing workflows.
- ▸ ADF is a serverless service, meaning you don't manage underlying infrastructure; it scales automatically based on workload demands.
🎯 How does Azure Data Factory appear on the AZ-900 Exam?
You may be asked to identify the Azure service best suited for orchestrating a complex ETL process involving data from multiple on-premises SQL Servers and Azure Blob Storage.
A scenario might describe a requirement to load data into a data warehouse on a scheduled basis – determine which service provides the necessary scheduling and orchestration features.
Expect questions about choosing between ADF and Azure Synapse Pipelines, focusing on scenarios where ADF’s broader connector support is advantageous.
❓ Frequently Asked Questions
When would I choose a Self-hosted Integration Runtime?
Use a Self-hosted IR when you need to access data sources behind a firewall or on-premises networks that are not directly accessible from Azure. It provides secure connectivity.
How does ADF differ from Azure Databricks?
ADF focuses on data movement and orchestration, while Databricks excels at large-scale data processing and machine learning. They often work together – ADF can trigger Databricks notebooks.
What are common use cases for ADF triggers?
Schedule triggers run pipelines on a defined schedule. Event triggers start pipelines in response to events like blob storage updates. Tumbling Window triggers process data in time-based batches.