Azure Data Factory Interview Questions

The serverless fully managed Azure Data Factory (ADF) is a remedy for ingesting, preparing, and converting all of your data at scale. It enables all businesses across all sectors to use it for a wide range of use cases, including data engineering, operations, and maintenance data integration, analytics, intaking information into information warehouses, and more.

The very next layers of new tech, listed from the greatest abstraction level which you communicate with to the software nearest to the data, are necessary to make ADF actually work for you.

• The graphical user interface called Pipeline, where widgets are placed and data paths are drawn
• Activity is a visual widget that modifies your data.
• Source and Sink, the components of an action that identify the sources and sinks of data
• Data Set, a set of data that is explicitly defined and on which ADF can operate; Linked Service, the link details that permit ADF to access a particular external data resource
• Integration ADF can communicate with software from outside itself through the runtime, a glue or gateway layer.

Clearly, the demand for azure data factory professionals is quite high in the market causing tremendous competition within the industry.  

To make the process of learning easier, we have divided the interview questions into four categories, they are:

• Basic
• Scenario-Based
• Advanced
• FAQs

Frequently Asked Azure Data Factory Interview Questions

1. How can we in the Data Factory utilize code to higher environments?
2. What steps constitute an ETL process?
3. Can we give a pipeline run parameters?
4. What Data Factory constructs are available and useful?
5. What are data flow maps?
6. What various Azure Data Factory activities have you used?
7. What benefit does lookup activity in the Azure Data Factory provide?
8. How can an ADF pipeline be debugged?
9. Describe the azure data factory's data source
10. What are some of ADF's drawbacks?

If you want to enrich your career and become a professional in Azure Data Factory, then enroll in "Azure Data Factory Training". This course will help you to achieve excellence in this domain.

Basic Azure Data Factory Interview Questions and Answers

1. What do Azure Data Factory's ARM Templates do? What do they serve?

The infrastructure and setup for the data factory pipeline, such as linked services, pipeline activities, datasets, etc, are defined in an ARM template, which is a JSON (JavaScript Object Notation) file. The template's code will be nearly identical to that of our pipeline. When we want to move our pipeline code from Development to a higher environment, such as Production or Staging, after we are certain that the code is functioning properly, ARM templates come in handy.

2. How can we in the Data Factory utilize code to higher environments?

At a high level, the following series of actions will help us accomplish this:

• Make a feature branch where our code base will be kept.
• Once we're certain the code belongs in the Dev branch, start creating a request form to merge it.
• Publish the development branch's code to create ARM templates.
• As a result, code can be promoted to higher surroundings like Staging or Production using an automated CI/CD DevOps pipeline.

3. What are the three tasks that Microsoft Azure Data Factory supports?

Data Factory supports the following activities: data movement, transformation, and control activities.

• Movement of data activities: As the name implies, these processes aid in the transfer of data.
• Activities for data transformation: These activities assist in data transformation as the data is loaded into the target or destination.
• Control flow activities: Control (flow) activities aid in regulating any activity's flow through a pipeline.

[ Learn Complete Azure Data Factory Tutorial ]

4. What are the two categories of computing environments that Data Factory supports for the purposes of carrying out transform activities?

The different compute environment types that Data Factory endorses for carrying out transformation activities are listed below: -

• On-Demand Computing Environment: ADF offers this completely managed environment. When performing this kind of calculation, a cluster is created to carry out the transformation activity, and it is automatically deleted once the task is finished.
• Bring Your Own Environment: If you have the infrastructure for on-premises services, you can use ADF to manage the computing environment in this scenario.

5. What steps constitute an ETL process?

The four main steps of the ETL also called as Extract, Transform, and Load process are as follows:

• Integrate and Collect: Attach to the data source(s) and transfer the data to crowdsourced and local data storage.
• Using computing services like HDInsight, Hadoop, Spark, etc., for data transformation.
• Publish: To upload data to Azure Cosmos DB, Azure SQL databases, Azure Data Lake storage, etc.
• Monitor: Pipeline monitoring for Azure Data Factory is built-in and supported by Azure Monitor, API, PowerShell, Azure Monitor logs, and health panels on the Azure portal.

6. Which activity should you use if you need to use the results from running a query?

The output of a query or executable execution can be returned by a look-up activity. The outcome can be a singleton value, an array of attributes, or any transition or control flow activity like the ForEach activity. These outputs can be used in a subsequent copy data activity.

7. Can we give a pipeline run parameters?

Yes, in Data Factory, parameters are a first-class, top-level concept. When running the pipeline on demand or using a trigger, we can plate thickness at the pipeline level and pass arguments.

8. Have you ever used Data Factory's Execute Notebook activity? How do I pass parameters to an activity in my notebook?

To send code to the databricks cluster, we can use the execute notebook activity. Using the baseParameters property, we can supply parameters to a notebook activity. The default values from notebook are used if the parameters are not described or specified in the activity.

9. What Data Factory constructs are available and useful?

• Parameter: Using the @parameter construct, each activity in the pipeline can use the parameter value that was passed to it.
• Coalesce: To gracefully handle null values, we can use the @coalesce construct in the expressions.
• Activity: The @activity construct enables the consumption of an activity output in a subsequent activity.

10. Can CI/CD or Continuous Integration and Continuous Delivery be used with ADF to push code?

The CI/CD of the data pipelines utilizing Azure DevOps and GitHub is fully supported by Data Factory. Before publishing the final product, this allows you to refine and deliver the ETL processes incrementally. Load the data into Azure Cosmos DB, Azure SQL Azure Data Lake, Azure Data Warehouse, or whatever analytics engine the company uses and can point to from one‘s business intelligence tools once the raw data has been transformed into a form that can be consumed by businesses.

[ Related Article: GitHub CI/CD Tutorial ]

Scenario-Based Azure Data Factory Interview Questions

11. What exactly do you mean when you refer to variables in Azure Data Factory?

The Azure Data Factory pipeline's variables offer the capacity to store the values. They are available inside the pipeline and used for the same purposes as factors in any programming language.

Setting or modifying the values of the variables can be done using the set variable and trying to add variable activities. In a data factory, there are two different kinds of variables:

• System variables: these are constant variables that come from the Azure pipeline. The names of pipelines, pipeline IDs, triggers, etc. These are primarily required to obtain any network data that might be required for your use case.
• User variable: You must manually declare a user variable in your code.

12. What are data flow maps?

In Azure Data Factory, mapping flow of data are data transformations that are visually designed. Without writing any code, data engineers can create data transformation logic using data flows. The resulting data flows are carried out in weighted Apache Spark clusters by Azure Data Factory pipelines as activities. Utilizing the scheduling, control flow, and monitoring tools already available in Azure Data Factory, data flow activities could be operationalized.

Data flow mapping offers a completely visual experience without the need for coding. Scaled-out data processing is carried out using execution clusters that are managed by ADF. All of the path optimizations, data flow job execution, and code translation are handled by Azure Data Factory.

13. What does the Azure Data Factory mean by copy activity?

One of the most well-liked and frequently used activities in the Azure data factory is copy. It is employed in ETL, also known as lift and shift, which is the process of moving data through one data source to another. You can transform the data as you copy it. For instance, let's say you read data from a txt/csv file with 12 columns, but you only want to keep seven columns when writing it to the target data source. It can be transformed so that only the necessary number of columns are sent to the target data source.

14. Can you provide more details about the Copy activity?

At a high level, the copy activity completes the following actions:

Take information out of the source data store. Work with the data to perform the following tasks:

• Serialization/deserialization
• Compression/decompression
• Table mapping
• Enter data into the sink or destination data store.

15. What various Azure Data Factory activities have you used?

If you've used some of the key activities in your career, whether it be your job or a college project, you can share them here. Here are some of the most popular pursuits:

• To transfer data between datasets, use the Copy Data Activity.
• ForEach Activity is used to loop.
• Get Metadata Activity can give you information about any data source's metadata.
• To define and start variables within pipelines, set the variable activity.
• Lookup operation to retrieve values from a table or file.
• Wait Activity to wait for a predetermined period of time prior to or following a pipeline run.
• The existence of files inside the dataset will be verified by the validation activity.

16. How do I plan a pipeline?

A pipeline can be scheduled using either the window of time trigger or the scheduler trigger. The trigger utilizes a wall-clock calendar timetable that can schedule pipelines on a recurring basis or periodically. There are three trigger types that the service currently supports:

Trigger for tumbling windows: A trigger that keeps a state while operating at regular intervals.

• Schedule trigger: A trigger that starts a pipeline on a wall-clock timetable is known as a schedule trigger.
• Event-based trigger: A trigger that reacts to an event, such as a file being added to a blob, is known as an event-based trigger. The relationship between pipelines and triggers is many-to-many except for the tumbling window trigger. A single pipeline can be started by multiple triggers, or many pipelines can be started by a single trigger.

17. When is Azure Data Factory the best option?

Consider utilizing Data Factory:

• A data warehouse must be implemented when continuing to work with big data; you may need a cloud-based integrated platform like ADF for the same.
• Not all team members have coding experience, and some might find it easier to work with data using graphical tools.
• We would like to use a single analytics solution, such as ADF, to integrate all of the raw business data that is stored across various data sources, which may be on-premises or in the cloud.
• We prefer to manage our infrastructure lightly and use solutions for data motion and processing that are easily accessible.

18. How are the remaining 90 dataset types in Data Factory used for data access?

Azure Synapse Analytics, Azure SQL Database delimited text files from such an Azure storage account, or Azure Data Lake Storage Gen2 are all supported natively as the source and sink data sources by the mapping data flow feature. Parquet files from blob storage or Data Lake Storage Gen2 are also supported. Data from all other connectors should be staged using the Copy activity before being transformed using a Data Flow activity.

[ Check out Azure Analysis Services ]

19. Is it possible to use the established column from mapping in ADF to determine a value for a new column?

We can create a new column predicated on our desired logic by deriving transformations from the mapping data flow. When trying to generate a derived column, we have the option to add a new one or keep updating an existing one. In the Column textbox, type the name of the new column you're creating. To replace an existing column in the schema, use the column dropdown. To begin writing the expression for the derived column, click the Enter expression textbox. To create your logic, either input it or use the expression builder.

20. What benefit does lookup activity in the Azure Data Factory provide?

The Lookup activity in the ADF pipeline is frequently used for setup lookup needs, and the origin dataset is accessible. Additionally, it is used to extract the data from the source dataset and send it as the activity's output. The output of a lookup activity is typically utilized in the pipeline to make additional decisions or to present any resulting configuration. Simply put, the ADF pipeline uses lookup activity to fetch data. Your pipeline logic would determine how you would use it. Depending on the dataset or query, you may be able to retrieve just the first row or all of the rows.

Advanced Azure Data Factory Interview Questions

21. Give more details about the Azure Data Factory's Get Metadata activity.

Any data in an Azure Data Factory or Synapse pipeline can have its metadata retrieved using the Get Metadata activity. The Get Metadata activity's output can be used in conditional expressions to sample predictions or to consume the metadata in later activities. It receives a dataset as input and outputs metadata details. The following connectors are supported right now, along with the corresponding retrievable metadata. The returned metadata can only be up to 4 MB in size.

22. How can an ADF pipeline be debugged?

One of the most important components of any coding-related task is debugging, which is necessary to test the software for any potential bugs. It also offers the choice of debugging the pipeline without actually running it.

23. What does "the breakpoint in the ADF pipeline" refer to?

For instance, let's say you have a pipeline with three activities and want to focus on debugging the second action only. By setting the cut-off point at the second activity, you can achieve this. You can press the circle at the activity's top to add a breakpoint.

24. What purpose does the ADF Service serve?

The main function of an ADF is to coordinate data copying among numerous relational and non-relational sources of data that are hosted locally, in data centers, or the cloud. Additionally, you can use the ADF Service to transform the information that has been ingested to meet business needs. ADF Service is utilized as an ETL or ELT tool for loading data in the majority of Big Data solutions.

[ Check out Top Open Source ETL Tools ]

25. Describe the azure data factory's data source.

The system from which the data will be used or executed is referred to as the data source. Data can be in binary, text, CSV, JSON, or any other format. It might be an appropriate database, but it could also be an image, video, or audio files.

26. How do I copy data from multiple sheets in an Excel file?

We must specify the name of the sheet from which we must load data when using an Excel connector inside of a data factory. When dealing with data from a single or small number of sheets, this approach is nuanced. However, if we have many sheets (say, 10+), changing the hard-coded sheet name repeatedly can become tedious. To accomplish this, we can use a data factory binary data format plug and point it at the excel file without having to specify which sheet(s) to use. The copy activity will allow us to copy the data from each and every sheet in the file.

27. Is nested looping possible with Azure Data Factory?

The data factory does not directly support nested looping for any looping action (for each / until). One for each and until loop activities, on the other hand, contain execute pipeline activities that may contain loop activities. In this manner, we can achieve nested looping because when we call the loop activity, it will inadvertently call another loop activity.

28. How can I move several tables from one datacenter to another?

An effective strategy for finishing this task would've been:

• Keep a lookup table or file that lists the tables that need to be copied along with their sources.
• After that, we can scan the list using the data retrieval activity and each loop activity.
• To copy multiple tables to the target datastore, we can employ a copy activity or a mapping data flow inside the for each loop activity.

29. What are some of ADF's drawbacks?

Excellent data movement and transition functionalities are offered by Azure Data Factory. There are, however, some restrictions as well.

• If we have enclosed looping activities in our pipeline, we cannot have them in the data factory and must find a solution. This covers all looping activities: Activities include If, Foreach, Switch, and Until.
• A maximum of 5000 rows can be retrieved at once by the lookup activity. Again, in order to achieve this type of organization in the pipeline, we must combine SQL with another loop activity.
• The total number of activities we can have in a single pipeline, along with all inner activities, and containers.

30. Which assimilation runtime should we employ when using Azure Data Factory to copy data from a local SQL Server instance?

We should have installed the self-hosted assimilation runtime on the onsite machine where the SQL Server Instance is offered to host in order to copy data from an on-premises SQL Database using Azure Data Factory.

Most Common Azure Data Factory FAQs

1. What is Azure Data Factory?

Microsoft Azure's Azure Data Factory is a fully managed, serverless, cloud-based ETL and data integration service for automating the transfer of data from its original location to, say, a data lake or centralized data using ETL also called extract-transform-load. You can use it to build and execute data pipelines that move and transform the data as well as execute scheduled pipelines.

2. What type of tool: ETL or ELT tool is Azure Data Factory?

It is a Microsoft cloud-based tool that supports the ETL and ELT paradigms and offers cloud-based information for data analytics at scale.

3. Why do we require ADF?

ADF is a service that really can orchestrate and implement processes to transform vast stores of raw business information into usable business insights, which is necessary given the growing amount of big data.

4. What distinguishes Azure Data Factory from traditional ETL tools?

Due to the following features, Azure Data Factory differs from other ETL tools: -

• Enterprise Readiness: Big Data Analytics, Data Integration at Cloud Scale.
• Enterprise Information Readiness: You can get your data to the Azure cloud from more than 90 different sources.
• Code-Free Transformation: UI-driven dataflows for mapping.
• The capacity to execute code on any Azure calculate resource: practical data transformations
• Three-step process for moving on-premises services to Azure Cloud: On Azure cloud, many SSIS packages are active.
• Streamlining DataOps with source control, automatic vehicle deployment, and straightforward templates.
• Secure Data Integration: Control module networks guard against data espionage, simplifying your networking in the process.

A comprehensive end-to-end framework for data engineers is offered by the collection of interlinked systems that make up Data Factory. The same is summed up in the paragraph below.

5. What are the various pipeline execution methods available in Azure Data Factory?

In Data Factory, we can run a pipeline in one of three ways:

• Debug mode can be useful for testing and troubleshooting our code as well as for trying out pipeline code.
• In a pipeline, selecting the "Trigger now" option initiates manual execution. If you wish to run the pipelines on an as-needed basis, this is helpful.
• Using a trigger, we can programme our pipelines to run at specific times and intervals. There are three different trigger types available in Data Factory.

6. What do Connected Services in Azure Data Factory serve?

In Data Factory, Linked Services are primarily used for two purposes:

• For a representation of a data store, such as an Oracle DB/ SQL Server instance, a file share, or an Azure Blob storage account.
• The underlying VM will carry out the activity specified in the pipeline for Compute representation.

7. Can you provide more information about the Data Factory Integration Runtime?

The computing foundation for Azure Data Factory pipelines is called the Integration Runtime, or IR. It acts as a link between various activities and associated services. It offers the computer environment in which the linked provider or activity can be dispatched or run directly. As a result, the task can be carried out in the area that is closest to the aim data stores or calculate service.

8. What are the types of integration runtime supported by Azure data factory?

One should select an integration runtime based on their network environment requirements and data integration capabilities from the three types supported by Azure Data Factory.

• To duplicate data among cloud data stores and send the activity to different computing services like SQL Server, Azure HDInsight, etc., use the Azure Integration Runtime.
• Self-Hosted Integration Runtime: Used for copy operations between private network data stores and cloud data stores. Similar to the Azure Integration Runtime, self-hosted integration running time is software that is installed on your local computer or virtual machine via a virtual network.
• Run SSIS packages in a controlled environment using Azure SSIS Integration Runtime. Consequently, when we move SSIS bundles to the data factory by lifting them, we use ADF.

[ Related Article: How to Create Package in SSIS? ]

9. What is necessary for an SSIS package to run in a Data Factory?

Before we can run an SSIS package, we must first start creating an SSIS integration runtime and an SSISDB catalog hosted in the Azure SQL server database or an Azure SQL-managed instance.

10. If there is a cap on the quantity of Integration Runtimes, what is it?

Data sets, Pipelines, linked services, triggers, integration runtimes, and private endpoints, all have a default limit of 5000 in a data factory. If necessary, one can submit an online support ticket to increase the restriction to a higher number.

Conclusion

ADF can also perform more complicated transformations by calling webhooks, running PySpark code in Databricks, initiating a custom virtual machine, etc. Data can be delivered to a variety of locations after the movements and transformations are complete, including an outbound FTP, SQL database, and a plain file system.