1

Introductions & Setup

Registration, Courseware & Q&As

2

Apache Spark Overview

Delta Overview

About Databricks, Spark & Spark Architecture

Delta Lake and how to leverage it for data science/ML applications

3

ML Overview (optional)

Types of Machine Learning, Business applications of ML

(NOTE: this class uses Airbnb's SF rental data to predict things such as price of rental)

4

Data Cleansing

How to deal with null values, outliers, data imputation

5

Data Exploration Lab

Exploring your data, log-normal distribution, determine baseline metric to beat

6

Linear Regression I

Build simple univariate linear regression model

SparkML APIs: transformer vs estimator

1

Linear Regression I Lab

Build multivariate linear regression model

Evaluate RMSE and R2

2

Linear Regression II

How to One Hot Encode in Spark, Pipeline API, Saving and loading models

3

Linear Regression II Lab

Simplify pipeline using RFormula, Build linear regression model to predict on log-scale, then exponentiate prediction and evaluate

4

MLflow Tracking

Use MLflow to track experiments, log metrics, and compare runs

5

MLflow Model Registry

Register a model using MLflow

Deploy that model into production

Update a model in production to new version including a staging phase for testing

Archive and delete models

6

MLflow Lab

Use MLflow to track models and Delta table

Register Model

1

Review

What did we discover yesterday?

2

Decision Trees

Distributed implementation of decision trees and maxBins parameter (why you WILL get different results from sklearn)

Feature importance

3

Hyperparameter Tuning

K-Fold cross-validation

SparkML's Parallelism parameter (introduced in Spark 2.3)

Speed up Pipeline model training by 4x

4

Hyperparameter Tuning/RF Lab

Hyperparameter search with cross-validation on Random Forests

5

Hyperopt

Hyperparameter tuning for MLlib models

1

Hyperopt Lab

Distributed hyperparameter tuning for scikit-learn models with SparkTrials

2

MLlib Deployment Options

Discuss batch, streaming, and real-time use cases and how MLlib/Spark fit into each of those

3

XGBoost

Using 3rd party libraries with Spark

Discuss gradient boosted trees and their variants

4

Inference with Pandas UDFs

Build a single-node ML model, but apply in parallel using mapInPandas (introduced in Spark 3.0)

5

Training with Pandas UDFs

Build a single-node ML model for each IoT Device using applyInPandas (introduced in Spark 3.0) and track it with MLflow

6

Pandas UDFs Lab

Lab to apply a single-node ML model at scale

7

Koalas

Use the new open-source library to write Pandas code that distributes using Spark under the hood

8

Capstone Project & Course Recap

Load dataset into Databricks and create Delta Table

Build a machine learning model

Track model performancew with MLflow

Present models to class (if time)