Notion Vault Portal Backup & Automation Hub

Databricks Cheatsheet

Databricks is a unified, cloud-based data analytics platform built on Apache Spark. It provides an interactive workspace for data engineers, data scientists, and business analysts to collaborate.


1. Delta Lake Optimization & Storage Engine

Delta Lake is the default storage layer in Databricks, providing ACID transactions, version control, and performance enhancements over raw Parquet files.

-- Compact small files automatically in Delta Tables (major performance boost)
OPTIMIZE delta_db.sales_data;

-- Z-Ordering by high-cardinality search columns to accelerate downstream queries
OPTIMIZE delta_db.sales_data
ZORDER BY (customer_id, country);

-- Purge old file versions no longer needed (defaults to keeping 7 days of history)
-- CAUTION: Bypasses the "Time Travel" version restore feature
VACUUM delta_db.sales_data RETAIN 168 HOURS;

Table Properties for Auto-Optimization (Auto-Compact & Optimize Write)

Configure tables to automatically manage files sizes on insert:

ALTER TABLE delta_db.sales_data SET TBLPROPERTIES (
  'delta.autoOptimize.optimizeWrite' = 'true',
  'delta.autoOptimize.autoCompact' = 'true'
);

2. Spark/Delta Table Time Travel Queries

Delta Lake keeps transaction histories, allowing you to query historical table states or roll back errors.

-- Query table exactly as it looked on a specific timestamp
SELECT * FROM delta_db.sales_data TIMESTAMP AS OF '2026-07-10 10:00:00';

-- Query table by specific sequential version number
SELECT * FROM delta_db.sales_data VERSION AS OF 12;

-- Restore a broken table state
RESTORE TABLE delta_db.sales_data TO VERSION AS OF 10;

3. DBUtils Workspace & Storage Utilities

dbutils is a native suite of helper tools inside Databricks to manage file systems, secret scopes, and notebook parameters.

DBUtils File System (fs) Examples (Python)

# List files in a DBFS path
dbutils.fs.ls("/mnt/data-lake/raw/")

# Copy a directory or file recursively
dbutils.fs.cp("/mnt/data-lake/raw/", "/mnt/data-lake/backup/", recurse=True)

# Mount an Azure Blob storage or AWS S3 Bucket
dbutils.fs.mount(
  source = "wasbs://container@storageaccount.blob.core.windows.net",
  mount_point = "/mnt/my-container",
  extra_configs = {"fs.azure.account.key.storageaccount.blob.core.windows.net": dbutils.secrets.get(scope="my-scope", key="storage-key")}
)

DBUtils Secrets Management

# Fetch a password/token safely from Databricks Secrets Scope (bypassing plaintext configs)
api_token = dbutils.secrets.get(scope="finance-scope", key="stripe-api-token")

4. Performance Tuning & Adaptive Query Execution (AQE)

Adaptive Query Execution (AQE) is a Spark 3.x technology that optimizes query plans dynamically at runtime based on intermediate metrics.

# Enable Adaptive Query Execution (Default: True in modern runtimes)
spark.conf.set("spark.sql.adaptive.enabled", "true")

# Dynamically coalesce shuffle partitions to match sizes (prevents thousands of empty tasks)
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")

# Automatically switch to broadcast joins if dynamic partition data is below limit
spark.conf.set("spark.sql.adaptive.localShuffleReader.enabled", "true")

5. MLflow Tracking & Model Registry Integration

MLflow is natively integrated inside Databricks for tracking AI/ML training experiments, hyper-parameters, and model registration.

import mlflow
import mlflow.spark
from sklearn.ensemble import RandomForestRegressor

# Set active experiment path in Workspace
mlflow.set_experiment("/Users/developer@company.com/Sales_Forecasting_RF")

# Start an active training logging run
with mlflow.start_run() as run:
    # Model configuration
    n_estimators = 100
    model = RandomForestRegressor(n_estimators=n_estimators)
    model.fit(X_train, y_train)

    # Log parameters and evaluation metrics
    mlflow.log_param("n_estimators", n_estimators)
    mlflow.log_metric("rmse", 0.1452)

    # Log the trained model object
    mlflow.sklearn.log_model(model, "random-forest-model")

    # Register the logged model directly to the Unity Catalog / Databricks Registry
    model_uri = f"runs:/{run.info.run_id}/random-forest-model"
    mlflow.register_model(model_uri, "prod_sales_forecaster")

6. Unity Catalog (UC) Governance

Unity Catalog is a unified governance solution for files, databases, tables, and AI assets across your Databricks workspace.

-- Create a catalogs structure
CREATE CATALOG IF NOT EXISTS prod_catalog;
USE CATALOG prod_catalog;

-- Create schemas inside the catalogs
CREATE SCHEMA IF NOT EXISTS silver_schema;
USE SCHEMA silver_schema;

-- Grant read permission to an engineering group
GRANT SELECT ON TABLE prod_catalog.silver_schema.sales_facts TO `data-engineering-group`;

-- Grant read/write permissions for specific schemas
GRANT USAGE, CREATE ON SCHEMA prod_catalog.silver_schema TO `data-science-group`;

-- Standardized UC namespace mapping:
-- catalog.schema.table