Module 1 · The platform

Control plane & compute plane

Intermediate 18 min read What runs where, and why it matters

Databricks is split in two. The control plane runs in Databricks' own cloud account: the web UI, notebooks, the job scheduler, cluster management and Unity Catalog's metadata. The compute plane is where your data is actually processed, either on virtual machines in your cloud account (classic compute) or in a pool Databricks runs for you (serverless). Your data itself stays in your cloud storage. Knowing this boundary explains Databricks' security model, its networking, its bill and many of its error messages.

✈️

Air traffic control

The control tower (control plane) plans flights, tracks them and gives instructions, but it never carries passengers. The aircraft (compute plane) do the actual flying, and the passengers (your data) board at your own airport (your storage). The tower can tell a plane to take off, but it cannot open the luggage.

1. The picture

CONTROL PLANE · Databricks' cloud account web app / APIs notebooks (code) jobs scheduler cluster manager Unity Catalog metadata:tables, grants, lineage CLASSIC COMPUTE · your cloud account VMs in your VPC / VNet you pay the cloud for VMs + Databricks DBUs full network control (private links, firewalls) start-up: minutes · you choose instance types SERVERLESS COMPUTE · Databricks' account a warm pool managed by Databricks you pay DBUs only (VM cost included) network via Databricks' serverless controls start-up: seconds · no instance types to pick launches & instructs YOUR CLOUD STORAGE (S3 / ADLS / GCS): the data files
The control plane holds your code, configuration and metadata; the compute plane processes data; your storage holds the data. Serverless moves compute into Databricks' account but data still lives in your storage (or Databricks-managed storage in Free Edition and some default setups).

2. Accounts, workspaces and metastores

LevelWhat it isTypical count
AccountYour company's top-level Databricks tenant: users, groups, billing, metastores1
Metastore (Unity Catalog)The top of the data hierarchy, one per cloud region1 per region
WorkspaceA deployment with its own URL, notebooks, jobs and compute (e.g. dev, prod)several
Catalog → schema → tableThe data hierarchy inside the metastore (lesson 06)many

Several workspaces can attach to the same metastore, so a table created in the dev workspace can be granted to users of the analytics workspace without copying anything. Identity (users, groups, service principals) lives at the account level and is synced from your identity provider (Microsoft Entra ID, Okta) via SCIM.

3. Why the split matters

Security

Data never has to pass through the control plane. Notebook results shown in the UI do, which is why admins can restrict downloading results and why secrets should never be printed.

Networking

Classic clusters in your VPC need to reach the control plane (outbound) and your storage. Private connectivity (PrivateLink / Private Link) keeps that traffic off the public internet.

Billing

Classic: two bills, cloud VMs plus Databricks DBUs. Serverless: one DBU-based bill that includes the machines (lesson 16).

Errors

"Cluster failed to start: cloud provider quota exceeded" is your cloud account's limit. "Permission denied on table" is Unity Catalog, in the control plane.

4. Where data lives

Tables are Delta files (Parquet data files plus a _delta_log folder) in cloud object storage. With Unity Catalog, managed tables live in a storage location that Unity Catalog controls (you never touch the paths), and external tables point at paths you manage. Access to storage is granted through storage credentials and external locations defined in Unity Catalog, not through keys pasted into notebooks. Lesson 06 covers this in detail. 

DESCRIBE DETAIL main.default.orders;      -- shows format = delta, location, numFiles, sizeInBytes
DESCRIBE HISTORY main.default.orders;     -- every write recorded in the transaction log

Recap

  • Control plane (Databricks' account): UI, notebooks, scheduler, cluster manager, Unity Catalog metadata.
  • Compute plane: classic VMs in your account, or serverless in Databricks' account.
  • Data stays in cloud object storage as Delta tables; access is governed by Unity Catalog.
  • Account → metastore (per region) → workspaces; many workspaces can share one metastore.

Checkpoint

1 · A classic cluster fails to start with "insufficient capacity for instance type". Where is the problem?
Classic compute is created in your cloud account, so VM quotas and capacity are your cloud's. Try another instance type or zone, raise the quota, or use serverless.
2 · Why can two workspaces query the same table without copying it?
The metastore is shared across workspaces in a region. Data stays in one storage location, and access is controlled by grants.