~/omprakashjena
6 min read

Modern Python CI/CD on Databricks with uv and Declarative Automation Bundles

PythonuvDatabricksAzure DevOpsCI/CD

Modern Python CI/CD on Databricks with uv and Declarative Automation Bundles

Python dependency management is simple until the same project must work across developer machines, CI agents, and Databricks compute. Then familiar problems appear: inconsistent environments, slow installations, mixed development and runtime dependencies, and builds that behave differently from production.

uv and Databricks Declarative Automation Bundles solve different parts of this problem:

  • uv manages Python versions, dependencies, virtual environments, locking, testing, and package builds.
  • Declarative Automation Bundles define, validate, deploy, and run Databricks resources and artifacts.

Declarative Automation Bundles were previously called Databricks Asset Bundles. Databricks renamed them in March 2026 without changing the existing databricks bundle commands or databricks.yml configuration.

Together, they provide a clean path from local development to governed Databricks deployment.

Where each tool fits

Responsibility Tool
Declare Python dependencies pyproject.toml
Resolve and lock dependencies uv.lock
Create the local environment uv sync
Run tests and quality checks uv run
Build the application wheel uv build
Define jobs, pipelines, and targets Declarative Automation Bundles
Deploy and run Databricks resources Databricks CLI

The overall delivery flow is:

flowchart LR
    A["Develop"] --> B["Lock and test"]
    B --> C["Build wheel"]
    C --> D["Validate bundle"]
    D --> E["Deploy"]
    E --> F["Run on Databricks"]

Project structure

A compact packaged project can use the following structure:

customer-events/
├── databricks.yml
├── pyproject.toml
├── uv.lock
├── resources/
│   └── customer_events.job.yml
├── src/
│   └── customer_events/
│       ├── __init__.py
│       └── main.py
└── tests/
    └── test_main.py

Create it with:

uv init --package customer-events
cd customer-events

A minimal pyproject.toml could be:

[project]
name = "customer-events"
version = "0.1.0"
requires-python = ">=3.11,<3.13"
dependencies = [
    "pydantic>=2",
]

[dependency-groups]
dev = [
    "pytest",
    "ruff",
]

[project.scripts]
customer-events = "customer_events.main:main"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.pytest.ini_options]
testpaths = ["tests"]

[tool.ruff]
line-length = 100
target-version = "py311"

Runtime dependencies remain under project.dependencies. Test and lint tools belong in the dev dependency group, so they do not become application runtime dependencies.

Both pyproject.toml and uv.lock should be committed. The project file defines acceptable dependency ranges; the lockfile records the exact resolution used by developers and CI.

Local development with uv

A developer can recreate the project environment with:

uv sync --locked

The --locked option fails if pyproject.toml and uv.lock disagree, preventing an unnoticed dependency update.

Commands run inside the managed environment without manually activating .venv:

uv run ruff check .
uv run pytest
uv build --wheel

Dependencies can be changed while keeping the project metadata and lockfile synchronized:

uv add httpx
uv add --dev mypy
uv remove pydantic

For pull-request validation, a useful minimum is:

uv lock --check
uv sync --locked
uv run ruff check .
uv run pytest
uv build --wheel

Building the wheel with a bundle

Databricks now recommends uv for managing Python dependencies in bundle projects. The bundle can call uv directly when building its artifact.

# databricks.yml
bundle:
  name: customer_events

include:
  - resources/*.yml

artifacts:
  application_wheel:
    type: whl
    path: .
    build: uv build --wheel

variables:
  cluster_id:
    description: Compute used by the job

targets:
  dev:
    mode: development
    default: true
    variables:
      cluster_id: "<development-cluster-id>"
    workspace:
      host: "https://<development-workspace-url>"

  prod:
    mode: production
    variables:
      cluster_id: "<production-cluster-id>"
    workspace:
      host: "https://<production-workspace-url>"

The job installs and executes the generated wheel:

# resources/customer_events.job.yml
resources:
  jobs:
    customer_events_job:
      name: customer-events-${bundle.target}

      tasks:
        - task_key: process_events
          existing_cluster_id: ${var.cluster_id}

          python_wheel_task:
            package_name: customer_events
            entry_point: customer-events

          libraries:
            - whl: ../dist/*.whl

The example uses existing compute to keep the configuration short. Production workloads should use compute that follows organisational policies, such as serverless compute or a policy-controlled job cluster.

The deployment commands remain simple:

databricks bundle validate -t dev
databricks bundle deploy -t dev
databricks bundle run -t dev customer_events_job

The uv.lock boundary

One detail is easy to miss:

uv.lock controls the developer and CI environments, but Databricks compute does not automatically consume it when installing the application wheel.

A standard wheel contains application code and dependency metadata. If pyproject.toml declares pydantic>=2, the Databricks installer may select a newer compatible version than the one recorded in uv.lock.

There are three common approaches:

  1. Compatible ranges: deploy the wheel and let Databricks resolve dependencies. This is simple but provides less control over exact transitive versions.
  2. Exported runtime lock: generate a requirements file from uv.lock and deploy it alongside the wheel.
  3. Approved wheelhouse: download and scan all required wheels in a network-enabled environment, then install them from a Unity Catalog volume or internal artifact repository.

An exported runtime lock can be generated with:

uv export \
  --frozen \
  --no-dev \
  --no-emit-project \
  --format requirements-txt \
  --output-file requirements.lock.txt

Do not maintain this file manually. Generate it from uv.lock during CI.

Azure DevOps CI example

The pipeline below verifies the lockfile, runs quality checks, builds the wheel, and validates the development bundle. Deployment should be placed in protected stages with environment approvals.

trigger:
  branches:
    include:
      - dev
      - main

pr:
  branches:
    include:
      - dev
      - main

pool:
  vmImage: ubuntu-latest

variables:
  UV_VERSION: "0.11.28"

steps:
  - checkout: self

  - bash: |
      python -m pip install "uv==$(UV_VERSION)"
      curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
    displayName: Install build tools

  - bash: |
      uv lock --check
      uv sync --locked
    displayName: Verify and synchronize dependencies

  - bash: |
      uv run ruff check .
      uv run pytest --junitxml=test-results.xml
    displayName: Run quality checks

  - bash: uv build --wheel
    displayName: Build wheel

  - bash: databricks bundle validate -t dev
    displayName: Validate bundle
    env:
      DATABRICKS_HOST: $(DATABRICKS_HOST)
      DATABRICKS_CLIENT_ID: $(DATABRICKS_CLIENT_ID)
      DATABRICKS_CLIENT_SECRET: $(DATABRICKS_CLIENT_SECRET)

For production automation, pin both uv and the Databricks CLI. Use a service principal or workload identity instead of a developer’s personal token.

A practical branch-to-environment mapping is:

Branch Deployment
Feature branch Pull-request validation only
dev Development
main Pre-production, approval, then production

Private dependencies and restricted compute

uv can use Azure Artifacts without storing credentials in source control:

[[tool.uv.index]]
name = "private-registry"
url = "https://pkgs.dev.azure.com/<organization>/<project>/_packaging/<feed>/pypi/simple/"
explicit = true

Credentials should be supplied through the developer credential helper or CI secret variables:

export UV_INDEX_PRIVATE_REGISTRY_USERNAME="<username>"
export UV_INDEX_PRIVATE_REGISTRY_PASSWORD="<token>"

If Azure DevOps can reach the package feed but Databricks compute cannot, building only the application wheel is insufficient. Its private and third-party dependencies must also reach the runtime.

In that situation, a network-enabled job can build a platform-compatible wheelhouse and store it in a controlled Unity Catalog volume. The production job validates that the required wheel set exists and installs only from that location.

  • Keep dependency metadata in pyproject.toml and commit uv.lock.
  • Separate runtime and development dependencies.
  • Use uv lock --check and uv sync --locked in CI.
  • Build the application as a wheel.
  • Define the wheel build as uv build --wheel in databricks.yml.
  • Validate bundles during pull requests, but deploy only from protected branches.
  • Use separate bundle targets for each environment.
  • Use a service principal for unattended production deployment.
  • Build once and promote the same tested artifact.
  • Use an exported runtime lock or approved wheelhouse when exact runtime reproducibility is required.

Conclusion

uv and Declarative Automation Bundles are complementary rather than competing tools. uv creates a fast, consistent Python development and build workflow. Bundles make Databricks resources and deployments declarative. Azure DevOps connects both with testing, approvals, identity controls, and environment promotion.

The result is a repeatable software engineering workflow from local development to production Databricks deployment.

References