<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://omprakashjena.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://omprakashjena.com/" rel="alternate" type="text/html" /><updated>2026-07-15T22:04:19+01:00</updated><id>https://omprakashjena.com/feed.xml</id><title type="html">Omprakash Jena</title><subtitle>Personal site, portfolio, and blog of Omprakash Jena — GenAI Solutions Architect &amp; Consultant.</subtitle><author><name>Omprakash Jena</name><email>omprakash.jena@hotmail.com</email></author><entry><title type="html">Modern Python CI/CD on Databricks with uv and Declarative Automation Bundles</title><link href="https://omprakashjena.com/uv-python-package-manager/" rel="alternate" type="text/html" title="Modern Python CI/CD on Databricks with uv and Declarative Automation Bundles" /><published>2026-07-13T00:00:00+01:00</published><updated>2026-07-13T00:00:00+01:00</updated><id>https://omprakashjena.com/uv-python-package-manager</id><content type="html" xml:base="https://omprakashjena.com/uv-python-package-manager/"><![CDATA[<h1 id="modern-python-cicd-on-databricks-with-uv-and-declarative-automation-bundles">Modern Python CI/CD on Databricks with uv and Declarative Automation Bundles</h1>

<p>Python dependency management is simple until the same project must work across developer machines, CI agents, and Databricks compute. Then familiar problems appear: inconsistent environments, slow installations, mixed development and runtime dependencies, and builds that behave differently from production.</p>

<p><a href="https://docs.astral.sh/uv/"><code class="language-plaintext highlighter-rouge">uv</code></a> and <a href="https://docs.databricks.com/aws/en/dev-tools/bundles/">Databricks Declarative Automation Bundles</a> solve different parts of this problem:</p>

<ul>
  <li><strong><code class="language-plaintext highlighter-rouge">uv</code></strong> manages Python versions, dependencies, virtual environments, locking, testing, and package builds.</li>
  <li><strong>Declarative Automation Bundles</strong> define, validate, deploy, and run Databricks resources and artifacts.</li>
</ul>

<blockquote>
  <p>Declarative Automation Bundles were previously called <strong>Databricks Asset Bundles</strong>. Databricks renamed them in March 2026 without changing the existing <code class="language-plaintext highlighter-rouge">databricks bundle</code> commands or <code class="language-plaintext highlighter-rouge">databricks.yml</code> configuration.</p>
</blockquote>

<p>Together, they provide a clean path from local development to governed Databricks deployment.</p>

<h2 id="where-each-tool-fits">Where each tool fits</h2>

<table>
  <thead>
    <tr>
      <th>Responsibility</th>
      <th>Tool</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Declare Python dependencies</td>
      <td><code class="language-plaintext highlighter-rouge">pyproject.toml</code></td>
    </tr>
    <tr>
      <td>Resolve and lock dependencies</td>
      <td><code class="language-plaintext highlighter-rouge">uv.lock</code></td>
    </tr>
    <tr>
      <td>Create the local environment</td>
      <td><code class="language-plaintext highlighter-rouge">uv sync</code></td>
    </tr>
    <tr>
      <td>Run tests and quality checks</td>
      <td><code class="language-plaintext highlighter-rouge">uv run</code></td>
    </tr>
    <tr>
      <td>Build the application wheel</td>
      <td><code class="language-plaintext highlighter-rouge">uv build</code></td>
    </tr>
    <tr>
      <td>Define jobs, pipelines, and targets</td>
      <td>Declarative Automation Bundles</td>
    </tr>
    <tr>
      <td>Deploy and run Databricks resources</td>
      <td>Databricks CLI</td>
    </tr>
  </tbody>
</table>

<p>The overall delivery flow is:</p>

<pre><code class="language-mermaid">flowchart LR
    A["Develop"] --&gt; B["Lock and test"]
    B --&gt; C["Build wheel"]
    C --&gt; D["Validate bundle"]
    D --&gt; E["Deploy"]
    E --&gt; F["Run on Databricks"]
</code></pre>

<h2 id="project-structure">Project structure</h2>

<p>A compact packaged project can use the following structure:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>customer-events/
├── databricks.yml
├── pyproject.toml
├── uv.lock
├── resources/
│   └── customer_events.job.yml
├── src/
│   └── customer_events/
│       ├── __init__.py
│       └── main.py
└── tests/
    └── test_main.py
</code></pre></div></div>

<p>Create it with:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>uv init <span class="nt">--package</span> customer-events
<span class="nb">cd </span>customer-events
</code></pre></div></div>

<p>A minimal <code class="language-plaintext highlighter-rouge">pyproject.toml</code> could be:</p>

<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[project]</span>
<span class="py">name</span> <span class="p">=</span> <span class="s">"customer-events"</span>
<span class="py">version</span> <span class="p">=</span> <span class="s">"0.1.0"</span>
<span class="py">requires-python</span> <span class="p">=</span> <span class="py">"&gt;</span><span class="p">=</span><span class="mf">3.11</span><span class="p">,</span><span class="err">&lt;</span><span class="mf">3.13</span><span class="s">"</span><span class="err">
</span><span class="py">dependencies</span> <span class="p">=</span> <span class="p">[</span>
    <span class="py">"pydantic&gt;</span><span class="p">=</span><span class="mi">2</span><span class="s">",</span><span class="err">
</span><span class="p">]</span>

<span class="nn">[dependency-groups]</span>
<span class="py">dev</span> <span class="p">=</span> <span class="p">[</span>
    <span class="s">"pytest"</span><span class="p">,</span>
    <span class="s">"ruff"</span><span class="p">,</span>
<span class="p">]</span>

<span class="nn">[project.scripts]</span>
<span class="py">customer-events</span> <span class="p">=</span> <span class="s">"customer_events.main:main"</span>

<span class="nn">[build-system]</span>
<span class="py">requires</span> <span class="p">=</span> <span class="nn">["hatchling"]</span>
<span class="py">build-backend</span> <span class="p">=</span> <span class="s">"hatchling.build"</span>

<span class="nn">[tool.pytest.ini_options]</span>
<span class="py">testpaths</span> <span class="p">=</span> <span class="nn">["tests"]</span>

<span class="nn">[tool.ruff]</span>
<span class="py">line-length</span> <span class="p">=</span> <span class="mi">100</span>
<span class="py">target-version</span> <span class="p">=</span> <span class="s">"py311"</span>
</code></pre></div></div>

<p>Runtime dependencies remain under <code class="language-plaintext highlighter-rouge">project.dependencies</code>. Test and lint tools belong in the <code class="language-plaintext highlighter-rouge">dev</code> dependency group, so they do not become application runtime dependencies.</p>

<p>Both <code class="language-plaintext highlighter-rouge">pyproject.toml</code> and <code class="language-plaintext highlighter-rouge">uv.lock</code> should be committed. The project file defines acceptable dependency ranges; the lockfile records the exact resolution used by developers and CI.</p>

<h2 id="local-development-with-uv">Local development with uv</h2>

<p>A developer can recreate the project environment with:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>uv <span class="nb">sync</span> <span class="nt">--locked</span>
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">--locked</code> option fails if <code class="language-plaintext highlighter-rouge">pyproject.toml</code> and <code class="language-plaintext highlighter-rouge">uv.lock</code> disagree, preventing an unnoticed dependency update.</p>

<p>Commands run inside the managed environment without manually activating <code class="language-plaintext highlighter-rouge">.venv</code>:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>uv run ruff check <span class="nb">.</span>
uv run pytest
uv build <span class="nt">--wheel</span>
</code></pre></div></div>

<p>Dependencies can be changed while keeping the project metadata and lockfile synchronized:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>uv add httpx
uv add <span class="nt">--dev</span> mypy
uv remove pydantic
</code></pre></div></div>

<p>For pull-request validation, a useful minimum is:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>uv lock <span class="nt">--check</span>
uv <span class="nb">sync</span> <span class="nt">--locked</span>
uv run ruff check <span class="nb">.</span>
uv run pytest
uv build <span class="nt">--wheel</span>
</code></pre></div></div>

<h2 id="building-the-wheel-with-a-bundle">Building the wheel with a bundle</h2>

<p>Databricks now recommends <code class="language-plaintext highlighter-rouge">uv</code> for managing Python dependencies in bundle projects. The bundle can call <code class="language-plaintext highlighter-rouge">uv</code> directly when building its artifact.</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># databricks.yml</span>
<span class="na">bundle</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">customer_events</span>

<span class="na">include</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="s">resources/*.yml</span>

<span class="na">artifacts</span><span class="pi">:</span>
  <span class="na">application_wheel</span><span class="pi">:</span>
    <span class="na">type</span><span class="pi">:</span> <span class="s">whl</span>
    <span class="na">path</span><span class="pi">:</span> <span class="s">.</span>
    <span class="na">build</span><span class="pi">:</span> <span class="s">uv build --wheel</span>

<span class="na">variables</span><span class="pi">:</span>
  <span class="na">cluster_id</span><span class="pi">:</span>
    <span class="na">description</span><span class="pi">:</span> <span class="s">Compute used by the job</span>

<span class="na">targets</span><span class="pi">:</span>
  <span class="na">dev</span><span class="pi">:</span>
    <span class="na">mode</span><span class="pi">:</span> <span class="s">development</span>
    <span class="na">default</span><span class="pi">:</span> <span class="no">true</span>
    <span class="na">variables</span><span class="pi">:</span>
      <span class="na">cluster_id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;development-cluster-id&gt;"</span>
    <span class="na">workspace</span><span class="pi">:</span>
      <span class="na">host</span><span class="pi">:</span> <span class="s2">"</span><span class="s">https://&lt;development-workspace-url&gt;"</span>

  <span class="na">prod</span><span class="pi">:</span>
    <span class="na">mode</span><span class="pi">:</span> <span class="s">production</span>
    <span class="na">variables</span><span class="pi">:</span>
      <span class="na">cluster_id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">&lt;production-cluster-id&gt;"</span>
    <span class="na">workspace</span><span class="pi">:</span>
      <span class="na">host</span><span class="pi">:</span> <span class="s2">"</span><span class="s">https://&lt;production-workspace-url&gt;"</span>
</code></pre></div></div>

<p>The job installs and executes the generated wheel:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># resources/customer_events.job.yml</span>
<span class="na">resources</span><span class="pi">:</span>
  <span class="na">jobs</span><span class="pi">:</span>
    <span class="na">customer_events_job</span><span class="pi">:</span>
      <span class="na">name</span><span class="pi">:</span> <span class="s">customer-events-${bundle.target}</span>

      <span class="na">tasks</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">task_key</span><span class="pi">:</span> <span class="s">process_events</span>
          <span class="na">existing_cluster_id</span><span class="pi">:</span> <span class="s">${var.cluster_id}</span>

          <span class="na">python_wheel_task</span><span class="pi">:</span>
            <span class="na">package_name</span><span class="pi">:</span> <span class="s">customer_events</span>
            <span class="na">entry_point</span><span class="pi">:</span> <span class="s">customer-events</span>

          <span class="na">libraries</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">whl</span><span class="pi">:</span> <span class="s">../dist/*.whl</span>
</code></pre></div></div>

<p>The example uses existing compute to keep the configuration short. Production workloads should use compute that follows organisational policies, such as serverless compute or a policy-controlled job cluster.</p>

<p>The deployment commands remain simple:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>databricks bundle validate <span class="nt">-t</span> dev
databricks bundle deploy <span class="nt">-t</span> dev
databricks bundle run <span class="nt">-t</span> dev customer_events_job
</code></pre></div></div>

<h2 id="the-uvlock-boundary">The uv.lock boundary</h2>

<p>One detail is easy to miss:</p>

<blockquote>
  <p><code class="language-plaintext highlighter-rouge">uv.lock</code> controls the developer and CI environments, but Databricks compute does not automatically consume it when installing the application wheel.</p>
</blockquote>

<p>A standard wheel contains application code and dependency metadata. If <code class="language-plaintext highlighter-rouge">pyproject.toml</code> declares <code class="language-plaintext highlighter-rouge">pydantic&gt;=2</code>, the Databricks installer may select a newer compatible version than the one recorded in <code class="language-plaintext highlighter-rouge">uv.lock</code>.</p>

<p>There are three common approaches:</p>

<ol>
  <li><strong>Compatible ranges:</strong> deploy the wheel and let Databricks resolve dependencies. This is simple but provides less control over exact transitive versions.</li>
  <li><strong>Exported runtime lock:</strong> generate a requirements file from <code class="language-plaintext highlighter-rouge">uv.lock</code> and deploy it alongside the wheel.</li>
  <li><strong>Approved wheelhouse:</strong> download and scan all required wheels in a network-enabled environment, then install them from a Unity Catalog volume or internal artifact repository.</li>
</ol>

<p>An exported runtime lock can be generated with:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>uv <span class="nb">export</span> <span class="se">\</span>
  <span class="nt">--frozen</span> <span class="se">\</span>
  <span class="nt">--no-dev</span> <span class="se">\</span>
  <span class="nt">--no-emit-project</span> <span class="se">\</span>
  <span class="nt">--format</span> requirements-txt <span class="se">\</span>
  <span class="nt">--output-file</span> requirements.lock.txt
</code></pre></div></div>

<p>Do not maintain this file manually. Generate it from <code class="language-plaintext highlighter-rouge">uv.lock</code> during CI.</p>

<h2 id="azure-devops-ci-example">Azure DevOps CI example</h2>

<p>The pipeline below verifies the lockfile, runs quality checks, builds the wheel, and validates the development bundle. Deployment should be placed in protected stages with environment approvals.</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">trigger</span><span class="pi">:</span>
  <span class="na">branches</span><span class="pi">:</span>
    <span class="na">include</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">dev</span>
      <span class="pi">-</span> <span class="s">main</span>

<span class="na">pr</span><span class="pi">:</span>
  <span class="na">branches</span><span class="pi">:</span>
    <span class="na">include</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">dev</span>
      <span class="pi">-</span> <span class="s">main</span>

<span class="na">pool</span><span class="pi">:</span>
  <span class="na">vmImage</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>

<span class="na">variables</span><span class="pi">:</span>
  <span class="na">UV_VERSION</span><span class="pi">:</span> <span class="s2">"</span><span class="s">0.11.28"</span>

<span class="na">steps</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">checkout</span><span class="pi">:</span> <span class="s">self</span>

  <span class="pi">-</span> <span class="na">bash</span><span class="pi">:</span> <span class="pi">|</span>
      <span class="s">python -m pip install "uv==$(UV_VERSION)"</span>
      <span class="s">curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh</span>
    <span class="na">displayName</span><span class="pi">:</span> <span class="s">Install build tools</span>

  <span class="pi">-</span> <span class="na">bash</span><span class="pi">:</span> <span class="pi">|</span>
      <span class="s">uv lock --check</span>
      <span class="s">uv sync --locked</span>
    <span class="na">displayName</span><span class="pi">:</span> <span class="s">Verify and synchronize dependencies</span>

  <span class="pi">-</span> <span class="na">bash</span><span class="pi">:</span> <span class="pi">|</span>
      <span class="s">uv run ruff check .</span>
      <span class="s">uv run pytest --junitxml=test-results.xml</span>
    <span class="na">displayName</span><span class="pi">:</span> <span class="s">Run quality checks</span>

  <span class="pi">-</span> <span class="na">bash</span><span class="pi">:</span> <span class="s">uv build --wheel</span>
    <span class="na">displayName</span><span class="pi">:</span> <span class="s">Build wheel</span>

  <span class="pi">-</span> <span class="na">bash</span><span class="pi">:</span> <span class="s">databricks bundle validate -t dev</span>
    <span class="na">displayName</span><span class="pi">:</span> <span class="s">Validate bundle</span>
    <span class="na">env</span><span class="pi">:</span>
      <span class="na">DATABRICKS_HOST</span><span class="pi">:</span> <span class="s">$(DATABRICKS_HOST)</span>
      <span class="na">DATABRICKS_CLIENT_ID</span><span class="pi">:</span> <span class="s">$(DATABRICKS_CLIENT_ID)</span>
      <span class="na">DATABRICKS_CLIENT_SECRET</span><span class="pi">:</span> <span class="s">$(DATABRICKS_CLIENT_SECRET)</span>
</code></pre></div></div>

<p>For production automation, pin both <code class="language-plaintext highlighter-rouge">uv</code> and the Databricks CLI. Use a service principal or workload identity instead of a developer’s personal token.</p>

<p>A practical branch-to-environment mapping is:</p>

<table>
  <thead>
    <tr>
      <th>Branch</th>
      <th>Deployment</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Feature branch</td>
      <td>Pull-request validation only</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">dev</code></td>
      <td>Development</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">main</code></td>
      <td>Pre-production, approval, then production</td>
    </tr>
  </tbody>
</table>

<h2 id="private-dependencies-and-restricted-compute">Private dependencies and restricted compute</h2>

<p><code class="language-plaintext highlighter-rouge">uv</code> can use Azure Artifacts without storing credentials in source control:</p>

<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[[tool.uv.index]]</span>
<span class="py">name</span> <span class="p">=</span> <span class="s">"private-registry"</span>
<span class="py">url</span> <span class="p">=</span> <span class="s">"https://pkgs.dev.azure.com/&lt;organization&gt;/&lt;project&gt;/_packaging/&lt;feed&gt;/pypi/simple/"</span>
<span class="py">explicit</span> <span class="p">=</span> <span class="kc">true</span>
</code></pre></div></div>

<p>Credentials should be supplied through the developer credential helper or CI secret variables:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span><span class="nv">UV_INDEX_PRIVATE_REGISTRY_USERNAME</span><span class="o">=</span><span class="s2">"&lt;username&gt;"</span>
<span class="nb">export </span><span class="nv">UV_INDEX_PRIVATE_REGISTRY_PASSWORD</span><span class="o">=</span><span class="s2">"&lt;token&gt;"</span>
</code></pre></div></div>

<p>If Azure DevOps can reach the package feed but Databricks compute cannot, building only the application wheel is insufficient. Its private and third-party dependencies must also reach the runtime.</p>

<p>In that situation, a network-enabled job can build a platform-compatible wheelhouse and store it in a controlled Unity Catalog volume. The production job validates that the required wheel set exists and installs only from that location.</p>

<h2 id="recommended-production-baseline">Recommended production baseline</h2>

<ul>
  <li>Keep dependency metadata in <code class="language-plaintext highlighter-rouge">pyproject.toml</code> and commit <code class="language-plaintext highlighter-rouge">uv.lock</code>.</li>
  <li>Separate runtime and development dependencies.</li>
  <li>Use <code class="language-plaintext highlighter-rouge">uv lock --check</code> and <code class="language-plaintext highlighter-rouge">uv sync --locked</code> in CI.</li>
  <li>Build the application as a wheel.</li>
  <li>Define the wheel build as <code class="language-plaintext highlighter-rouge">uv build --wheel</code> in <code class="language-plaintext highlighter-rouge">databricks.yml</code>.</li>
  <li>Validate bundles during pull requests, but deploy only from protected branches.</li>
  <li>Use separate bundle targets for each environment.</li>
  <li>Use a service principal for unattended production deployment.</li>
  <li>Build once and promote the same tested artifact.</li>
  <li>Use an exported runtime lock or approved wheelhouse when exact runtime reproducibility is required.</li>
</ul>

<h2 id="conclusion">Conclusion</h2>

<p><code class="language-plaintext highlighter-rouge">uv</code> and Declarative Automation Bundles are complementary rather than competing tools. <code class="language-plaintext highlighter-rouge">uv</code> creates a fast, consistent Python development and build workflow. Bundles make Databricks resources and deployments declarative. Azure DevOps connects both with testing, approvals, identity controls, and environment promotion.</p>

<p>The result is a repeatable software engineering workflow from local development to production Databricks deployment.</p>

<h2 id="references">References</h2>

<ul>
  <li><a href="https://docs.astral.sh/uv/">uv documentation</a></li>
  <li><a href="https://docs.astral.sh/uv/concepts/projects/sync/">uv locking and syncing</a></li>
  <li><a href="https://docs.astral.sh/uv/guides/integration/azure/">uv with Azure Artifacts</a></li>
  <li><a href="https://docs.databricks.com/aws/en/dev-tools/bundles/">Declarative Automation Bundles</a></li>
  <li><a href="https://docs.databricks.com/aws/en/dev-tools/bundles/library-dependencies">Bundle library dependencies</a></li>
  <li><a href="https://docs.databricks.com/aws/en/dev-tools/bundles/authentication">Bundle authentication</a></li>
  <li><a href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/azure-devops">Azure DevOps CI/CD with Databricks</a></li>
</ul>]]></content><author><name>Omprakash Jena</name></author><category term="Python" /><category term="uv" /><category term="Databricks" /><category term="Azure DevOps" /><category term="CI/CD" /><summary type="html"><![CDATA[Using uv for reproducible Python development, testing, packaging, and Azure DevOps deployment with Databricks Declarative Automation Bundles.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://omprakashjena.com/assets/images/bio-photo.png" /><media:content medium="image" url="https://omprakashjena.com/assets/images/bio-photo.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>