Skip to main content

Example: run Apache Airflow in App Studio

This worked example deploys Apache Airflow 3.3.2, web interface included, as an App Studio app in about ten minutes.

Who can use it: the Builder role or higher.

VideoRun Apache Airflow in App Studio2:09

Recorded on .

Prerequisites​

  • Your platform address, the part of the URL before the first / when you are signed in. This page writes it as your-platform-address.
  • The email address you sign in with.
  • A compute tier with at least 2 cores and 8 GB of memory. Airflow's standalone mode runs four processes at once.

Work out the app's address first​

Airflow must be told its own address before the first deploy. App Studio serves every app at /apps/name-suffix, and the suffix is the first six characters of a SHA-256 hash of your organisation, your sign-in email and the app name, joined by colons. On AI Platform the organisation part is empty, so the text starts with a colon. On AI Cloud it starts with your organisation's short name.

printf "%s" ":you@example.com:airflow" | sha256sum | cut -c1-6

For you@example.com and the app name airflow this prints f9a4df, so the app lives at https://your-platform-address/apps/airflow-f9a4df. The Review step shows the address with a placeholder where the suffix goes, so work the suffix out yourself.

Settings to copy​

Start command. Keep the word streamlit out of it. If it appears, the platform appends Streamlit's own flags to the command.

pip install "apache-airflow==3.3.2" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.2/constraints-3.11.txt" && exec airflow standalone

Environment variables. Replace airflow-f9a4df with your own name and suffix.

KeyValueWhy
AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_ALL_ADMINSTrueTurns off Airflow's own login. Read Limits before you rely on this.
AIRFLOW__API__PORT8501The port a Streamlit app is expected to listen on.
AIRFLOW__API__BASE_URLhttps://your-platform-address/apps/airflow-f9a4dfTells Airflow it is served under that path, so its links and assets resolve.
AIRFLOW__CORE__EXECUTION_API_SERVER_URLhttp://localhost:8501/apps/airflow-f9a4df/execution/Lets task processes reach Airflow inside the pod. Without it they go out through the sign-in page and every task fails.

Deploy Airflow​

  1. Open the wizard

    Switch to Studio, open App Studio and choose New app.

  2. Source

    Choose GitHub repo. Enter https://github.com/octocat/Hello-World with the branch master and leave Access token empty. App Studio needs a Git source to run a start command; the repository's files are never used. A Docker image source does not work here, because the official Airflow image needs the standalone argument and the wizard cannot pass it.

  3. Framework

    Detection says it could not identify a framework, which is expected. Pick Streamlit: it is the framework App Studio serves at the full /apps/ path without rewriting it, which is how Airflow expects to sit behind a proxy. Leave Main file as app.py and paste the start command into Build / start command.

  4. Runtime

    Set App name to airflow, the name you hashed. Choose a Compute tier with 2 cores and 8 GB or more, leave Runtime image on Auto (Python 3.11, which must match the constraints file in the start command), and set Min replicas and Max replicas to 1. Leave storage and the dataset on none. Set Health check path to /api/v2/monitor/health. Choose + Add variable four times and enter the table above, leaving Secret unticked.

  5. Network

    Choose Private (just me). The default is Shared. Leave everything else as it is.

  6. Review and ship

    Check the summary and choose Ship app. The build takes two to three minutes. The first line of the build log names the app it is creating. If that name is not the airflow- name you worked out, the address in your variables is wrong: delete the app and create it again with the right suffix.

Verify​

Choose Open on the app, or go to its address with a trailing slash.

  • The home page shows MetaDatabase, Scheduler, Triggerer and Dag Processor as healthy.
  • Dags lists Airflow's example DAGs.
  • Open example_bash_operator and trigger it. The run finishes as a success, with two tasks skipped by design: one exits with code 99, which skips it and the task after it.

Limits​

This is a demonstration, not a production Airflow.

  • Access. With Airflow's own login turned off, the app's visibility is the only gate. Private hides the app from other users in App Studio, but sign-in on its address is enforced only where your cluster enables it, and the Network step warns you when it does not. If you see that warning, remove the first variable so Airflow keeps its own login. Never choose Public.
  • No persistence. Airflow's database lives inside the pod, so a restart loses run history, connections and variables.
  • Slow restarts. Every pod start installs Airflow again, which takes about two minutes.
  • Your own DAGs need a real Git repository or a mounted Storage dataset. This example sets up neither.
  • Cost. The app is billed for its compute tier while it runs; Review shows the estimate. Stop or delete the app in App Studio when you are done.

The Airflow documentation for 3.3.2 explains each setting: see its pages on running behind a reverse proxy and its configuration reference.

Describes the platform as of .

Was this page helpful?