Requirements:
Some understanding of Database and SQL queries
Description:
Data Engineering is a vital component of modern data-driven businesses. The ability to process, manage, and analyze large-scale data sets is a core requirement for organizations that want to stay competitive. In this course, you will learn how to build a data pipeline using Apache Spark on Databricks' Lakehouse architecture. This will give you practical experience in working with Spark and Lakehouse concepts, as well as the skills needed to excel as a Data Engineer in a real-world environment.
Throughout the Course, You Will Learn:
Conducting analytics using Python and Scala with Spark.
Applying Spark SQL and Databricks SQL for analytics.
Developing a data pipeline with Apache Spark.
Becoming proficient in Databricks' free edition.
Managing a Delta table by accessing version history, restoring data, and utilizing time travel features.
Unity Catalog Volumes - File Storage and Operations
Optimizing query performance using Delta Cache.
Working with Delta Tables and Databricks File System.
Gaining insights into real-world scenarios from experienced instructors
Chapter 1_Introduction/2.Data Engineering with Spark.mp412.90 MiB
Chapter 1_Introduction/3.What is Databricks.mp47.11 MiB
Chapter 2_Working with Databricks Storage_ DBFS, Volumes, and Delta Tables/10.Important_ Interface Differences Between Databricks Editions.mp416.91 MiB
Chapter 2_Working with Databricks Storage_ DBFS, Volumes, and Delta Tables/4.Getting Started with Databricks Free Edition.mp426.49 MiB
Chapter 2_Working with Databricks Storage_ DBFS, Volumes, and Delta Tables/5-Resources/delta_unity.zip6.06 KiB
Chapter 2_Working with Databricks Storage_ DBFS, Volumes, and Delta Tables/5.Working with Unity Catalog and Delta Tables.mp474.42 MiB
Chapter 2_Working with Databricks Storage_ DBFS, Volumes, and Delta Tables/6-Resources/fx-dbfs-demo.ipynb.zip538.74 KiB
Chapter 2_Working with Databricks Storage_ DBFS, Volumes, and Delta Tables/6.Exploring DBFS and dbutils - Working with Sample Datasets.mp459.60 MiB
Chapter 2_Working with Databricks Storage_ DBFS, Volumes, and Delta Tables/7-Resources/databricks_volume_demo.ipynb28.32 KiB
Chapter 2_Working with Databricks Storage_ DBFS, Volumes, and Delta Tables/7.Unity Catalog Volumes - File Storage and Operations.mp4126.81 MiB
Chapter 2_Working with Databricks Storage_ DBFS, Volumes, and Delta Tables/8-Resources/read_from_db_write_to_delta.ipynb31.59 KiB
Chapter 2_Working with Databricks Storage_ DBFS, Volumes, and Delta Tables/8.Using Generative AI in Databricks for Data Transformation and Querying.mp415.04 MiB
Chapter 2_Working with Databricks Storage_ DBFS, Volumes, and Delta Tables/9.No-Code Data Engineering with Databricks Generative AI.mp421.83 MiB
Chapter 3_Data Engineering with Apache Spark/11-Resources/spark_transformations_python.zip2.46 KiB
Chapter 3_Data Engineering with Apache Spark/11.More Transformations and Actions using PySpark.mp474.18 MiB
Chapter 3_Data Engineering with Apache Spark/12-Resources/scala_transformations.zip589 B
Chapter 3_Data Engineering with Apache Spark/12.Doing the Transformations in Scala.mp459.46 MiB
Chapter 3_Data Engineering with Apache Spark/13-Resources/python_scala_crash_course.zip3.13 KiB
Chapter 3_Data Engineering with Apache Spark/13.Python Scala crash course.mp432.92 MiB
Chapter 3_Data Engineering with Apache Spark/14-Resources/fx_udf.ipynb.zip9.83 KiB
Chapter 3_Data Engineering with Apache Spark/14.Spark User Defined Functions (UDF).mp4218.78 MiB
Chapter 3_Data Engineering with Apache Spark/15-Resources/spark_joins_1.zip5.16 KiB
Chapter 3_Data Engineering with Apache Spark/15-Resources/store_customers_transactions_1.zip817.41 KiB
Chapter 3_Data Engineering with Apache Spark/15.Joining Datasets using DataFrame APIs and Spark SQL.mp4213.26 MiB
Chapter 3_Data Engineering with Apache Spark/16-Resources/spark_joins.zip5.16 KiB
Chapter 3_Data Engineering with Apache Spark/16-Resources/store_customers_transactions.zip817.41 KiB
Chapter 3_Data Engineering with Apache Spark/16.More join operations using Spark.mp451.51 MiB
Chapter 3_Data Engineering with Apache Spark/17.Section summary.mp45.53 MiB
Chapter 4_Dat Lakehouse Delta Lake and Delta Tables deep dive/18.Understanding Data Warehouse, Data Lake and Data Lakehouse.mp421.36 MiB
Chapter 4_Dat Lakehouse Delta Lake and Delta Tables deep dive/19.Databricks Lakehouse Architecture and Delta Lake.mp413.22 MiB
Chapter 4_Dat Lakehouse Delta Lake and Delta Tables deep dive/20.Delta Tables.mp43.20 MiB
Chapter 4_Dat Lakehouse Delta Lake and Delta Tables deep dive/21-Resources/spark_transformations_python_sql.zip6.52 KiB
Chapter 4_Dat Lakehouse Delta Lake and Delta Tables deep dive/21.Storing data in a Delta table, Databricks SQL and time travel.mp4147.21 MiB
Chapter 4_Dat Lakehouse Delta Lake and Delta Tables deep dive/22-Resources/databricks_sql.zip3.57 KiB
Chapter 4_Dat Lakehouse Delta Lake and Delta Tables deep dive/22.Databricks SQL vs Spark SQL.mp473.85 MiB
Chapter 4_Dat Lakehouse Delta Lake and Delta Tables deep dive/23-Resources/delta_table_caching.zip7.70 KiB
Chapter 4_Dat Lakehouse Delta Lake and Delta Tables deep dive/23.Delta Table caching.mp4150.76 MiB
Chapter 4_Dat Lakehouse Delta Lake and Delta Tables deep dive/24-Resources/delta_table_paritioning.zip10.80 KiB
Chapter 4_Dat Lakehouse Delta Lake and Delta Tables deep dive/24.Delta Table partitioning.mp491.31 MiB
Chapter 4_Dat Lakehouse Delta Lake and Delta Tables deep dive/25-Resources/delta_table_paritioning_zordering.zip13.69 KiB
Chapter 4_Dat Lakehouse Delta Lake and Delta Tables deep dive/25.Delta Table Z-ordering.mp427.60 MiB
Chapter 5_Databricks Labs on AWS/26.AWS Account Creation.mp429.73 MiB
Chapter 5_Databricks Labs on AWS/27.Setting up Databricks account on AWS.mp456.78 MiB
Chapter 5_Databricks Labs on AWS/28-Resources/aws-labs.zip65.43 KiB
Chapter 5_Databricks Labs on AWS/28.Running Notebooks Within a Databricks AWS Account.mp4209.15 MiB
Chapter 5_Databricks Labs on AWS/29-Resources/futurex-dlt-sql-demo.zip632 B
Chapter 5_Databricks Labs on AWS/29.Building an ETL pipeline with Delta Live Tables.mp4174.13 MiB
Chapter 6_Bonus Section - AWS Data Engineering Labs/30.AWS Identity and Access Management (IAM).mp4104.01 MiB