Databricks

This guide walks you through setting up a Haute pipeline to deploy to Databricks Model Serving - step by step, from scratch. The Databricks adapter is the end-to-end deployment path: it registers the model and creates or updates its serving endpoint. A CI run performs that action only when your team enables the generated workflow and supplies the required credentials.

What is Databricks Model Serving?

Databricks is a cloud platform for data and AI. Model Serving is a feature that takes a model (in this case, your pricing pipeline) and hosts it as a live web address (called an API) that accepts quote data and returns results. You don't need to manage any servers.

New to the command line?

This guide involves typing a few setup commands in a terminal. If you've never done that before, read Before You Start first - it takes five minutes and will make everything below much clearer.

What this guide covers

This guide has 9 steps. You don't need to do them all in one sitting.

• Steps 1-3 are one-time credential setup - get your workspace URL, create a token, and give them to CI. This takes 10 minutes.
• Steps 4-6 are Databricks infrastructure - set up Unity Catalog, MLflow, and Model Serving. You or your data engineer do these once. If your workspace is already set up, you may be able to skip some of these.
• Steps 7-9 are your ongoing workflow - review your config, deploy by merging to main, and call the API.

---

How it works

The generated CI workflow is designed to run haute deploy; you may use it after reviewing and adapting it. A typical workflow is:

1. Edit your pipeline locally and preview with haute serve
2. Push your changes and open a pull request - CI validates automatically
3. Merge to main - the generated workflow runs a staging deploy, then smoke and impact commands if the deploy succeeds
4. Review the result and use a CI-provider approval/process to decide whether to run the separate production workflow

Set up the configuration (haute.toml), Databricks infrastructure (steps below), CI secrets, and your CI provider's branch/environment protections. Haute does not enforce approval counts or create those protections.

---

Prerequisites

Before you start, you need:

• A Databricks workspace (your organisation probably already has one)
• Python 3.11+ installed on your machine
• Haute installed with Databricks extras (uv add "haute[databricks]" - see Installing Haute)
• A pipeline with an output and test quotes of your own. The starter pipeline is empty, and CI's dry-run deploy fails on it with "No nodes found" ("No output node found" on a pipeline without an output): build your pipeline, ending in a Quote Response, and replace the starter tests/quotes/example.json with your own test quotes before you push
• A repository on your CI provider for the project: Before You Start covers creating one for a new project

If you haven't initialised your project yet, open your VS Code terminal and run:

haute init --target databricks --ci github

Your team may have already done this

If you cloned an existing project that already has a haute.toml file, skip this step - it's already initialised. This command is only needed once per project, and whoever set up the repository probably ran it already.

---

Step 1: Get your Databricks workspace URL

Your workspace URL is the web address you use to log into Databricks. It looks like one of these:

Cloud	URL format
Azure (most common for UK insurance)	https://adb-1234567890123456.12.azuredatabricks.net
AWS	https://dbc-abc12345-1234.cloud.databricks.com
GCP	https://1234567890123456.gcp.databricks.com

If you don't know your workspace URL, open Databricks in your browser and copy the URL from the address bar (just the part before any / path).

Don't have a Databricks workspace?

Ask your IT team - most insurance companies with a data platform already have one. If you need to create one, the cheapest option is Azure Databricks on the Premium tier (required for Model Serving).

---

Step 2: Create a Personal Access Token

A Personal Access Token (PAT) is like a password that Haute uses to talk to Databricks on your behalf. To create one:

1. Open your Databricks workspace in a browser
2. Click your user icon in the top-right corner → Settings
3. Go to Developer → Access tokens
4. Click Manage → Generate new token
5. Fill in:
   • Comment: haute-deploy (so you remember what it's for)
   • Lifetime: 90 days (or whatever your organisation allows)
6. Click Generate
7. Copy the token immediately - you won't be able to see it again

The token looks like: dapi_your_token_here

---

Step 3: Add credentials to CI

Since deployment runs in CI (not on your laptop), your credentials need to be stored as encrypted secrets in your CI provider. This is a one-time setup.

The four values you need are:

Secret name	Value
DATABRICKS_MLFLOW_HOST	The workspace URL that hosts your MLflow experiments and Unity Catalog models
DATABRICKS_MLFLOW_TOKEN	A personal access token whose scopes cover MLflow and the model registry
DATABRICKS_RATING_HOST	Your workspace URL from Step 1
DATABRICKS_RATING_TOKEN	A personal access token whose scopes cover Model Serving

Why separate MLflow and RATING credentials?

Databricks token scopes can be narrow, so three credential spaces stay separate. The deploy logs and registers the model through MLflow with the DATABRICKS_MLFLOW_* pair, then creates or updates the serving endpoint with the DATABRICKS_RATING_* pair. The general DATABRICKS_HOST/DATABRICKS_TOKEN pair is for data access in the editor and is never used by MLflow or by the deploy. The values can be the same workspace URL and token when one token's scopes cover everything - only the names differ, and the deploy fails with a clear error naming any missing variable before it contacts Databricks.

The deploy, smoke, impact and release-tag steps all read these four, and haute deploy names every one that is missing before it does anything else.

On GitHub: let Haute set them

On GitHub, one command stores all four. Copy .env.example to .env and fill in the four values (.env is gitignored, so it never reaches the repository), sign in to the GitHub CLI once with gh auth login, then run this in your project:

haute ci secrets

It checks that Databricks accepts both pairs, then stores the four values as encrypted repository secrets, printing their names and never their values. haute ci secrets --check lists which of the four the repository has.

With the GitHub CLI signed in, the editor does this for you too. The Push that first publishes your project to an empty GitHub repository stores whichever of the four the repository doesn't have, before main arrives. A later Push that finds some missing offers Set from .env in the Git panel. The editor never replaces a secret the repository already has: run haute ci secrets to change one.

By hand, or on another CI provider

How to add them depends on your CI provider:

• GitHub Actions - see Adding GitHub Secrets
• GitLab CI/CD - see Adding CI/CD Variables
• Azure DevOps - see Creating a Variable Group

Don't know which CI provider you're using?

It depends on where your code is hosted. If you access your project on github.com, you're using GitHub. If it's gitlab.com (or a company GitLab server), you're using GitLab. If it's dev.azure.com, you're using Azure DevOps. If you're not sure, ask your IT team or tech lead: "Where is our code repository hosted?" They'll tell you GitHub, GitLab, or Azure DevOps.

You never deploy from your laptop

You never run haute deploy locally: the CI runner has the credentials, and you just merge to main. The .env on your laptop serves the editor and haute ci secrets (which pair does what).

---

Step 4: Set up Unity Catalog

Unity Catalog is where Databricks registers your deployed model. Think of it as a filing system - you need a catalog (like a cabinet) and a schema (like a folder inside it).

Check if Unity Catalog is enabled

1. In your Databricks workspace, click Catalog in the left sidebar
2. If you see a catalog browser with tables and schemas, it's enabled
3. If you don't see it, ask your workspace admin to enable it

Create a schema for your models

Most workspaces already have a main catalog. You just need a schema inside it.

To run the SQL command below, open your Databricks workspace in a browser and do one of the following:

• Option A: SQL Editor - click SQL Editor in the left sidebar, paste the command, and click Run
• Option B: Notebook - click New → Notebook, change the language to SQL (dropdown at the top), paste the command, and click Run

CREATE SCHEMA IF NOT EXISTS main.pricing;

You can name the schema anything you like - pricing is a sensible default.

No main catalog?

Newer workspaces often have no main catalog: theirs is usually called workspace. Use the catalog your workspace has, both in the CREATE SCHEMA command and in haute.toml. haute deploy checks the schema exists before it uploads the model, and if it does not, names the catalogs your token can see.

Update haute.toml

Make sure your haute.toml matches:

[deploy.databricks]
catalog = "main"
schema = "pricing"

---

Step 5: Set up an MLflow experiment

What is MLflow?

MLflow is a logging system built into Databricks that keeps an audit trail of every version you deploy - think of it as a version history for your pricing models. Every time you deploy, Haute creates an MLflow run recording what was deployed, when, and by whom. These runs are grouped inside an experiment (which is just a named folder for runs).

Every time you deploy, Haute logs the deployment as an MLflow run inside an experiment. This gives you a full history of every version you've ever deployed.

Option A: Let Haute create it automatically

If the experiment doesn't exist, Haute will create it on first deploy. Just set the path in haute.toml:

[deploy.databricks]
experiment_name = "/Shared/haute/motor-pricing"

Option B: Create it manually

1. In Databricks, click Experiments in the left sidebar (under Machine Learning)
2. Click Create Experiment
3. Set the name to /Shared/haute/motor-pricing (or whatever matches your pipeline)
4. Click Create

Staging deploys log to the same path with the staging suffix, /Shared/haute/motor-pricing-staging: create that one too, or let Haute create both.

Naming convention

We recommend /Shared/haute/<your-model-name> so all team members can access it:

/Shared/haute/motor-pricing
/Shared/haute/home-pricing
/Shared/haute/commercial-property

---

Step 6: Check that Model Serving is available

Databricks Model Serving is the feature that actually hosts your pipeline as an API. To check it's available:

1. Click Serving in the left sidebar
2. If you see the Serving page, you're good
3. If not, ask your workspace admin - Model Serving requires a Premium tier workspace

Cost

Model Serving is billed per compute-second. With serving_scale_to_zero = true in your config, you only pay when the endpoint is actually receiving requests. A Small workload with occasional dev/test traffic costs little; check your Databricks pricing for the rate.

---

Step 7: Review your full configuration

Here's the haute.toml that haute init writes for a project named motor-pricing:

[project]
name = "motor-pricing"
pipeline = "rating/main.py"

[deploy]
target = "databricks"
model_name = "motor-pricing"
endpoint_name = "motor-pricing"

[deploy.databricks]
experiment_name = "/Shared/haute/motor-pricing"
catalog = "main"
schema = "pricing"
serving_workload_size = "Small"
serving_scale_to_zero = true

[safety]
# `haute impact` compares staging with production on these quotes. Without a
# sample it compares the test quotes; commit a Parquet sample of your book (not
# under data/, which is gitignored) and name it here to compare that instead:
# impact_dataset = "tests/impact/portfolio_sample.parquet"

[safety.approval]
min_approvers = 2

[ci]
provider = "github"

[ci.staging]
endpoint_suffix = "-staging"

What each setting means

Setting	What it does	Example
target	Tells Haute to deploy to Databricks	"databricks"
model_name	The name of your model in the Databricks Model Registry	"motor-pricing"
endpoint_name	The name of the serving endpoint (becomes part of the API URL)	"motor-pricing"
experiment_name	Where MLflow logs each deployment	"/Shared/haute/motor-pricing"
catalog	Unity Catalog name	"main"
schema	Unity Catalog schema	"pricing"
serving_workload_size	How much compute to allocate - Small, Medium, or Large	"Small"
serving_scale_to_zero	Whether the endpoint shuts down when idle (saves money)	true
dir ([test_quotes])	Where the test quotes are, when not in tests/quotes. haute init leaves it out	"tests/quotes"
impact_dataset	The Parquet portfolio sample the impact step scores. Left unset, as haute init leaves it, the impact step compares your test quotes instead. CI can read only files in the repository, and haute init gitignores data/, so commit a sample you are allowed to share somewhere else (for example tests/impact/portfolio_sample.parquet) and point this at it	"tests/impact/portfolio_sample.parquet"
min_approvers	A record of how many approvals your team expects; your CI provider's settings enforce it	2
provider	The CI provider haute init generated workflows for	"github"
endpoint_suffix	Added to the endpoint, model and experiment names for staging deploys. The generated workflows hold it, so after changing it run haute ci update and commit the workflows	"-staging"

---

Step 8: Deploy by merging to main

Once your configuration is set up and CI secrets are in place, you can deploy through your CI process. The starter workflow is a sequence of commands, not an enforced governance policy:

1. Push your changes to a branch and open a pull request
2. CI validates automatically - lints your code, runs tests, and does a dry-run deploy to check your pipeline parses and test quotes pass
3. Merge the PR - the generated staging workflow requests deployment to motor-pricing-staging, then invokes smoke and impact commands if that command succeeds
4. Review the impact report - download it from CI if the impact job ran, and check the premium changes make sense
5. Promote deliberately - use the separate production workflow and any CI-provider approval rule your team configured

haute deploy waits until the endpoint serves the new version before it finishes. A first deploy builds the model's environment and starts its container, which can take 10–20 minutes; later deploys are usually quicker, and the generated deploy job allows an hour. If Databricks cannot build or start the model, the deploy fails with its reason and the end of the build log, and the endpoint keeps serving the previous version. Smoke and impact then run against a ready endpoint.

What does success look like?

After Databricks reports the endpoint Ready and the corresponding smoke and impact commands have run, you should see:

1. In your CI provider - green validation and deploy jobs; smoke and impact can be green only after endpoint readiness and valid endpoint configuration
2. In Databricks - click Serving in the left sidebar and you'll see your endpoint with status Ready and a green indicator: motor-pricing-staging after a merge, and motor-pricing once the production workflow has run
3. In MLflow - click Experiments in the left sidebar, navigate to your experiment (/Shared/haute/motor-pricing-staging for staging, /Shared/haute/motor-pricing for production), and you'll see a new run logged with the deployment details. The staging model is registered as main.pricing.motor-pricing-staging

If you see all three, your pipeline is live and serving premiums.

If CI reports errors during validation, the most common causes are:

• Missing model files - check that the files referenced in your pipeline (e.g. models/freq.cbm) exist
• Test quote schema mismatch - your test quote JSON fields don't match what your pipeline expects
• Pipeline syntax error - there's a bug in your Python file

See your CI provider's setup guide for full details: GitHub Actions, GitLab, or Azure DevOps.

---

Step 9: Call the API

Once the endpoint is live, you can call it from any system that can make HTTP requests. After a merge, the endpoint is the staging one, motor-pricing-staging, which the examples below call. The production endpoint, motor-pricing, answers once the production workflow has run: drop the suffix from the name to call it.

Using Python (recommended)

If you're more comfortable with Python than the command line, this is the easiest way to test:

import requests

# Replace these with your actual values
url = "https://adb-xxx.12.azuredatabricks.net/serving-endpoints/motor-pricing-staging/invocations"
token = "dapi_your_token_here"  # Your Databricks personal access token from Step 2

headers = {
    "Authorization": f"Bearer {token}",
    "Content-Type": "application/json",
}
payload = {
    "dataframe_records": [
        {"IDpol": 99001, "VehPower": 7, "DrivAge": 42, "Area": "C", "VehBrand": "B12"}
    ]
}

response = requests.post(url, json=payload, headers=headers)
print(response.json())

Using the Databricks SDK

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
response = w.serving_endpoints.query(
    name="motor-pricing-staging",
    dataframe_records=[
        {"IDpol": 99001, "VehPower": 7, "DrivAge": 42, "Area": "C", "VehBrand": "B12"}
    ],
)
print(response.predictions)

Using curl (advanced)

curl is a command-line tool for sending web requests. You don't need to use it - the Python examples above are easier and work the same on Windows - but it's included for reference:

curl -X POST `
  "https://<your-workspace-url>/serving-endpoints/motor-pricing-staging/invocations" `
  -H "Authorization: Bearer <your-token-here>" `
  -H "Content-Type: application/json" `
  -d '{\"dataframe_records\": [{\"IDpol\": 99001, \"VehPower\": 7, \"DrivAge\": 42, \"Area\": \"C\", \"VehBrand\": \"B12\"}]}'

Prefer the Python example above

The Python examples are simpler and avoid Windows command-line quoting issues. Use those unless you have a specific reason to use curl.

---

Troubleshooting

"PERMISSION_DENIED" on deploy

Your token needs these permissions:

• Can Manage on the MLflow experiment
• USE CATALOG and USE SCHEMA on Unity Catalog, and CREATE MODEL on the schema to register the model
• Can Manage on serving endpoints (or ask an admin to create the endpoint first)

Ask your Databricks admin to grant these if you see permission errors. This is usually someone in your IT or data engineering team - the person who set up the Databricks workspace.

"Endpoint not found" when calling the API

A deploy that finished has already waited for its endpoint to serve, so waiting longer won't help: the name or the workspace is wrong. Check that you are calling the staging name, motor-pricing-staging, after a merge (the production name, motor-pricing, exists only once the production workflow has run), and that the URL is the workspace the deploy used, DATABRICKS_RATING_HOST. The Serving page in the Databricks left sidebar lists the endpoints the workspace has.

Token expired

Tokens have a lifetime (default 90 days). An expired token fails the deploy before it uploads anything, and the error names the token:

• Databricks returned 401 Unauthorized. Check that your DATABRICKS_RATING_TOKEN is valid and has not expired. - the serving endpoint's token
• Databricks returned 401 Unauthorized to the model registry. Check that DATABRICKS_MLFLOW_TOKEN is valid and has not expired. - the token MLflow registers the model with

Generate a new token (Step 2) and replace that secret in your CI provider. On GitHub, put the new value in .env and run haute ci secrets.

Slow first request (cold start)

With serving_scale_to_zero = true, the endpoint shuts down when it's idle. The first request after idle can take 30–60 seconds. serving_scale_to_zero = false in haute.toml keeps an endpoint running, at the cost of paying for it while idle, and applies to staging and production alike: both deploys read the one [deploy.databricks] value.

To keep only production running, leave haute.toml as it is and set the environment variable HAUTE_SERVING_SCALE_TO_ZERO=false on the production deploy job, which overrides the haute.toml value for that deploy. On GitHub that is an env: entry in .github/workflows/deploy-production.yml. haute ci update rewrites that file and removes the entry, so add it again after an update.

The endpoint fails to build or start

haute deploy fails with Databricks' reason and the end of the endpoint's build log. The serving environment installs, on your project's Python (.python-version):

• the Haute that ran the deploy: haute==<version> from PyPI for a released Haute, or, when your project installs Haute from Git, that Haute itself, packed into the model as a wheel
• MLflow, Polars, pandas and NumPy at the versions the deploy ran, and the packages they need
• the engine each model in the pipeline loads under (CatBoost, XGBoost, LightGBM, EBM, t-boost or the GLM's), at the version installed where the deploy ran

A package that cannot be installed usually means the workspace cannot reach PyPI, or a Python version that Model Serving cannot provide: the build log names which.

"Databricks refused to create serving endpoint"

The deploy registered the model version, but Databricks would not create or update the endpoint. The message ends with Databricks' reason. "Cannot create or update model serving endpoints, please try again later" comes from the workspace, not from your pipeline. Databricks refuses the change before it builds anything, and Haute adds the reason Databricks gives with it:

• "Databricks has marked the workspace inactive": Databricks has deactivated the workspace, so it refuses every compute and serving request, a SQL warehouse start included, and retrying won't help. Sign in to the workspace in a browser, then deploy again: a deactivated Free Edition workspace has come back after a sign-in, while requests from CI did not bring it back. If it's still inactive, ask its account admin or Databricks support.
• No further reason: on Databricks Free Edition, the number of active endpoints is limited, and a workspace that exceeds its usage quota has its compute shut down for the rest of the day. Wait, or delete an endpoint you no longer need on the Serving page, then deploy again.

The registered version stays registered, and the next deploy registers and serves a new one.

---

Keeping the CI files up to date

The workflow files are generated by the Haute version that ran haute init. After upgrading Haute, run:

haute ci update

It rewrites the generated workflow files and .env.example for the installed version, overwriting changes you made to them by hand. Keeping the generated files up to date says what it rewrites and removes, and how to commit the result.

---

Checklist

Before your first deploy, confirm:

• [ ] You have your Databricks workspace URL
• [ ] You have a Personal Access Token
• [ ] All four are added as CI secrets (DATABRICKS_MLFLOW_HOST, DATABRICKS_MLFLOW_TOKEN, DATABRICKS_RATING_HOST, DATABRICKS_RATING_TOKEN): haute ci secrets --check confirms it on GitHub
• [ ] Unity Catalog is enabled with a catalog and schema
• [ ] haute.toml has the correct experiment_name, catalog, and schema
• [ ] Model Serving is available in your workspace
• [ ] Your pipeline ends in a Quote Response, and tests/quotes/ holds your own quotes, in the test quote format, in place of the starter example.json
• [ ] impact_dataset points at a portfolio sample committed to the repository, or is left unset to compare the test quotes
• [ ] CI/CD workflows are committed to your repository, with pyproject.toml and uv.lock
• [ ] CI validation passes on a pull request