Study with Databricks : Databricks-Certified-Data-Engineer-Professional Exam Torrent as your best preparation materials

How are the operational aspects of Lakeflow Declarative Pipelines different from Spark Structured Streaming?

• A. Lakeflow Declarative Pipelines manage the orchestration of multi-stage pipelines automatically, while Structured Streaming requires external orchestration for complex dependencies.
• B. Lakeflow Declarative Pipelines automatically handle schema evolution, while Structured Streaming always requires manual schema management.
• C. Structured Streaming can process continuous data streams, while Lakeflow Declarative Pipelines cannot.
• D. Lakeflow Declarative Pipelines can write to Delta Lake format, while Structured Streaming cannot.

A team of data engineer are adding tables to a DLT pipeline that contain repetitive expectations for many of the same data quality checks.
One member of the team suggests reusing these data quality rules across all tables defined for this pipeline.
What approach would allow them to do this?

• A. Maintain data quality rules in a Delta table outside of this pipeline's target schema, providing the schema name as a pipeline parameter.
• B. Add data quality constraints to tables in this pipeline using an external job with access to pipeline configuration files.
• C. Use global Python variables to make expectations visible across DLT notebooks included in the same pipeline.
• D. Maintain data quality rules in a separate Databricks notebook that each DLT notebook of file.

The data engineering team maintains the following code:

Assuming that this code produces logically correct results and the data in the source tables has been de-duplicated and validated, which statement describes what will occur when this code is executed?

• A. The enriched_itemized_orders_by_account table will be overwritten using the current valid version of data in each of the three tables referenced in the join logic.
• B. A batch job will update the enriched_itemized_orders_by_account table, replacing only those rows that have different values than the current version of the table, using accountID as the primary key.
• C. An incremental job will leverage information in the state store to identify unjoined rows in the source tables and write these rows to the enriched_iteinized_orders_by_account table.
• D. An incremental job will detect if new rows have been written to any of the source tables; if new rows are detected, all results will be recalculated and used to overwrite the enriched_itemized_orders_by_account table.
• E. No computation will occur until enriched_itemized_orders_by_account is queried; upon query materialization, results will be calculated using the current valid version of data in each of the three tables referenced in the join logic.

The data engineer team has been tasked with configured connections to an external database that does not have a supported native connector with Databricks. The external database already has data security configured by group membership. These groups map directly to user group already created in Databricks that represent various teams within the company. A new login credential has been created for each group in the external database. The Databricks Utilities Secrets module will be used to make these credentials available to Databricks users. Assuming that all the credentials are configured correctly on the external database and group membership is properly configured on Databricks, which statement describes how teams can be granted the minimum necessary access to using these credentials?

• A. "Read'' permissions should be set on a secret key mapped to those credentials that will be used by a given team.
• B. No additional configuration is necessary as long as all users are configured as administrators in the workspace where secrets have been added.
• C. "Manage" permission should be set on a secret scope containing only those credentials that will be used by a given team.
• D. "Read" permissions should be set on a secret scope containing only those credentials that will be used by a given team.

A data engineer is running a groupBy aggregation on a massive user activity log grouped by user_id. A few users have millions of records, causing task skew and long runtimes. Which technique will fix the skew in this aggregation?

• A. Use salting by adding a random prefix to skewed keys before aggregation, then aggregate again after removing the prefix.
• B. Increase the Spark driver memory and retry.
• C. Filter out the skewed users before the aggregation.
• D. Use reduceByKey instead of groupBy to avoid shuffles.