Section 4: Working with Lakeflow Jobs¶

Weighted 16% in the May 2026 exam guide, about seven questions. Lakeflow Jobs (formerly Databricks Workflows, and before that just "Jobs") is the orchestrator: it runs notebooks, SQL, pipelines and other tasks in a dependency graph, on a trigger, with retries and notifications.

The guide's objectives:

• Implement control flow: retries, and conditional tasks for branching and looping.
• Configure common tasks (notebook, SQL query, dashboard, pipeline) and their dependencies in the DAG.
• Schedule jobs, understanding the trigger types: scheduled, file arrival and table update.
• Choose between time-based and data-driven triggers, based on when data is available and on pipeline dependencies.

Source: the 📖 exam page and its linked exam guide. Creating jobs through the REST API and CLI, and serverless job settings, are Professional topics: see Professional Section 1.

Table of Contents¶

1. What a job is
2. Task types
3. Dependencies and the DAG
4. Passing values between tasks
5. Control flow: retries, branching and looping
6. Triggers
7. Time-based or data-driven?
8. Compute, notifications and other job settings
9. How the exam tests this section

What a job is¶

A job has three parts: one or more tasks (units of work), the dependencies between them (a directed acyclic graph, DAG), and a trigger (when it runs). Each execution is a job run, made of task runs. You build jobs in the Jobs & Pipelines UI, with the REST API or CLI, or as YAML in a Declarative Automation Bundle (Section 5). The same job definition can be viewed as YAML in the UI, which is a quick way to move from clicking to code.

flowchart LR
    T((Trigger: file arrival)) --> I[ingest - pipeline task]
    I --> C{check_rows - If/else}
    C -->|true| G[refresh_gold - SQL task]
    C -->|false| N[notify_empty - notebook task]
    G --> D[refresh_dashboard - dashboard task]
    I --> F[For each region: load_region - notebook]
    F --> G

The diagram shows a typical job: a trigger starts an ingestion pipeline, a condition decides whether to build gold or send a notice, a For each task loads regions in parallel, and a dashboard refreshes at the end.

📖 Lakeflow Jobs - concepts, tasks, triggers 📖 Configure and edit tasks - task settings

Task types¶

Two pairings the exam likes: put a pipeline task upstream of a SQL or dashboard task so the dashboard refreshes only after the data it shows has been rebuilt; and run SQL tasks on a SQL warehouse, notebook tasks on jobs compute.

📖 Task types - every type and its settings 📖 Dashboard task - refresh dashboards from a job 📖 Pipeline task - run a pipeline from a job

Dependencies and the DAG¶

Each task lists the tasks it depends on. Tasks with no dependency between them run in parallel; a task with several upstreams waits for all of them (by default). Draw the DAG and the run order falls out of it.

By default a task runs only when all its upstream tasks succeeded. The Run if setting changes that:

When a task fails, its downstream tasks are marked Upstream failed and don't run (unless their Run if says otherwise). After you fix the cause, repair run reruns only the failed and skipped tasks, not the whole job.

📖 Run if dependencies - conditions and examples 📖 Repair a failed job run - rerun only what failed

Passing values between tasks¶

Job parameters are key-value pairs set on the job and passed to every task; you can override them for a single run with Run now with different parameters. Inside a notebook, read them as widgets (dbutils.widgets.get("run_date")). In task settings, dynamic value references in double braces insert values at run time:

Task values let one task hand a small result to another:

# In task "ingest"
dbutils.jobs.taskValues.set(key="row_count", value=n)

# In a later task, or in an If/else condition as {{tasks.ingest.values.row_count}}
n = dbutils.jobs.taskValues.get(taskKey="ingest", key="row_count", default=0)

Task values are for small values (counts, flags, a list of table names), not datasets; data moves between tasks through tables.

📖 Parameterize jobs - job and task parameters 📖 Dynamic value references - the full list 📖 Task values - set and get

Control flow: retries, branching and looping¶

Retries are set per task: a maximum number of retries, a minimum interval between attempts, and whether to retry on timeout. They fix transient failures (a cloud capacity blip, a throttled API); they don't fix bugs, which fail every time. Add a timeout so a hung task fails instead of running forever. Only retry tasks that are safe to repeat: an ingestion task that appends could duplicate rows unless it is idempotent (Auto Loader, COPY INTO and MERGE are).

Branching uses the If/else condition task. It compares two operands with ==, !=, >, >=, < or <=; operands are usually a task value or job parameter and a constant. Its downstream tasks depend on its true or false outcome; the other branch is skipped (which is why tasks after a branch often use Run if "None failed").

- task_key: check_rows
  depends_on: [ { task_key: ingest } ]
  condition_task:
    op: GREATER_THAN
    left: "{{tasks.ingest.values.row_count}}"
    right: "0"
- task_key: refresh_gold
  depends_on: [ { task_key: check_rows, outcome: "true" } ]
  sql_task:
    file: { path: ../sql/refresh_gold.sql }
    warehouse_id: ${var.warehouse_id}

Looping uses the For each task. It runs one nested task once per item of an input list (a JSON array, a job parameter or a task value), with a concurrency setting for how many iterations run at once. Inside, {{input}} is the current item. The classic example: load 40 source tables with one parameterised notebook, 8 at a time, instead of copying the task 40 times.

- task_key: load_each_region
  for_each_task:
    inputs: '["emea", "amer", "apac"]'
    concurrency: 3
    task:
      task_key: load_region
      notebook_task:
        notebook_path: ../src/load_region.ipynb
        base_parameters: { region: "{{input}}" }

Failed iterations are retried on their own (per the nested task's retry policy), and a repair run reruns only failed iterations.

📖 Add branching logic with If/else - operators and outcomes 📖 Use a For each task to loop - inputs and concurrency 📖 Retries and timeouts - per-task policies

Triggers¶

A job has a trigger type that decides when runs start:

trigger:
  pause_status: UNPAUSED
  file_arrival:
    url: /Volumes/main/raw/landing/orders/
    min_time_between_triggers_seconds: 300
    wait_after_last_change_seconds: 120

"Wait after last change" is a debounce: if a vendor drops 50 files over two minutes, a 120-second wait starts one run after the last file instead of fifty runs. "Minimum time between triggers" is a cooldown: at most one run every N seconds.

📖 Automating jobs with schedules and triggers - every trigger type 📖 File arrival triggers - locations, limits, settings 📖 Table update triggers - any or all, supported tables 📖 Continuous jobs - always-on runs

Time-based or data-driven?¶

The guide's last objective is a judgement call. A time-based trigger runs whether or not there is new data. A data-driven trigger (file arrival, table update) runs because data changed.

The tell in exam wording: "files arrive at irregular times", "as soon as new data is available" and "avoid running when nothing changed" point to data-driven triggers. "Every night at 2 AM", "by a fixed time" or "on a regular cadence" point to schedules. A schedule set "a bit after the upstream usually finishes" is the classic wrong answer: it breaks the day the upstream runs late.

Compute, notifications and other job settings¶

• Compute. Each task picks compute: serverless jobs compute (the default choice: no cluster to manage, fast start), a classic jobs cluster created for the run and terminated after (for custom libraries, init scripts or GPUs), or a SQL warehouse for SQL and dashboard tasks. Several tasks can share one jobs cluster to avoid repeated start-ups. Running production jobs on an all-purpose cluster costs more and is the wrong answer.
• Notifications. On start, success, failure, duration warning and streaming backlog, to email, Slack, Microsoft Teams, PagerDuty or a webhook, at the job or task level.
• Duration threshold and timeout. Warn when a run takes longer than expected; fail it after a hard limit.
• Maximum concurrent runs. Default 1, so a slow run delays the next instead of overlapping it; raise only if runs are independent.
• Run as. The identity the job runs as. In production, use a service principal, not a person who might leave.
• Queueing. Lets runs wait when a concurrency limit is reached instead of being skipped.

📖 Notifications - destinations and events 📖 Configure compute for jobs - serverless, jobs clusters, sharing 📖 Manage identities for jobs - run as a service principal

How the exam tests this section¶

• Pick the trigger. Irregular arrivals or "only when new data exists" is file arrival; dependency on another table is table update; fixed time is scheduled; must never stop is continuous.
• Pick the control flow. "Only if the load produced rows" is If/else on a task value. "Same step for each of N items" is For each with concurrency. "Even if something failed" is Run if All done. "Transient failures" is retries, ideally with a fresh jobs cluster or serverless compute, plus pinned library versions.
• Order of tasks. Pipeline before the SQL refresh before the dashboard refresh. Independent tasks run in parallel.
• Values between tasks go through task values and {{tasks.<key>.values.<name>}}; data goes through tables.
• Recovering. Repair run reruns only failed tasks and their dependents.
• Cost. Jobs belong on serverless jobs compute or jobs clusters, not all-purpose clusters.

Continue with Section 5: Implementing CI/CD.