Databricks Certified Associate Developer for Apache Spark

Identify the advantages and challenges of implementing Spark. Identify the role of core components of Apache Spark Architecture, including cluster, driver node, worker nodes or executors, CPU cores, and memory. Describe the architecture of Apache Spark, including DataFrame and Dataset concepts, SparkSession lifecycle, caching, storage levels, and garbage collection. Explain the Apache Spark Architecture execution hierarchy. Configure Spark partitioning in distributed data processing, including shuffles and partitions. Describe the execution patterns of the Apache Spark engine, including actions, transformations, and lazy evaluation. Identify the features of the Apache Spark Modules, including Core, Spark SQL, DataFrames, Pandas API on Spark, Structured Streaming, and MLlib.

Utilize common data sources such as JDBC, files, and so on, to efficiently read from and write to Spark DataFrames using Spark SQL, including overwriting and partitioning by column. Execute SQL queries directly on files, including ORC files, JSON files, CSV files, Text files, and Delta files, and understand the different save modes for outputting data in Spark SQL. Save data to persistent tables while applying sorting and partitioning to optimize data retrieval. Register DataFrames as temporary views in Spark SQL, allowing them to be queried with SQL syntax.

Manipulate columns, rows, and table structures by adding, dropping, splitting, renaming column names, applying filters, and exploding arrays. Perform data deduplication and validation operations on DataFrames. Perform aggregate operations on DataFrames such as count, approximate count distinct, mean, and summary. Manipulate and utilize Date data type, such as Unix epoch to date string, and extract date component. Combine DataFrames with operations such as inner join, left join, broadcast join, multiple keys, cross join, union, and union all. Manage input and output operations by writing, overwriting, and reading DataFrames with schemas. Perform operations on DataFrames such as sorting, iterating, printing schema, and conversion between DataFrame and sequence or list formats. Create and invoke user-defined functions with or without stateful operators, including StateStores. Describe different types of variables in Spark, including broadcast variables and accumulators. Describe the purpose and implementation of broadcast joins.

Implement performance tuning strategies and optimize cluster utilization, including partitioning, repartitioning, coalescing, identifying data skew, and reducing shuffling. Describe Adaptive Query Execution (AQE) and its benefits. Perform logging and monitoring of Spark applications: publish, customize, and analyze Driver logs and Executor logs to diagnose out-of-memory errors, cluster underutilization, and so on.

Explain the Structured Streaming engine in Spark, including its functions, programming model, micro-batch processing, exactly-once semantics, and fault tolerance mechanisms. Create and write Streaming DataFrames and Streaming Datasets, including the basic output modes and output sinks. Perform basic operations on Streaming DataFrames and Streaming Datasets, such as selection, projection, window and aggregation. Perform Streaming Deduplication in Structured Streaming, both with and without watermark usage.

Describe the features of Spark Connect. Describe the different deployment mode types (Client, Cluster, Local) in the Apache Spark environment.

Explain the advantages of using Pandas API on Spark. Create and invoke Pandas UDF.