Command Line Interface (CLI)

All Gluepy projects have a CLI exposed through the manage.py file located in project root. This command line interface allow you to execute your DAGs, retry previous runs and manage your project in various ways.

The CLI comes with a set of commands out of the box, but it is also extendable and allow you to Create your own CLI commands.

Gluepy Commands

dag command

This command allow you to run a DAG registered in the DAG Registry by its label attribute. The label attribute can either be set explicitally, or it default to the lowercase string of the DAG class name.

startproject command

startproject(*args: Any, **kwargs: Any) → Any[source]

Command used to initiate a new Gluepy project.

Parameters:

project (str) – project name

startmodule command

startmodule(*args: Any, **kwargs: Any) → Any[source]

Command used to initiate a new Module in existing Gluepy project.

Parameters:

module (str) – module name

airflow generate command

This command is used to generate DAG files in Airflow format to be used to run Gluepy DAGs using Airflow as an orchestrator. The command sits under the airflow CLI command group and is used by executing ./manage.py airflow generate.

The command is leveraging the following settings:

  • AIRFLOW_TEMPLATE. Path to .j2 Jinja file that contains template of how the Gluepy DAG is transformed into Airflow format. Defaults to template that generate Airflow DAG using KubernetesPodOperator.

  • AIRFLOW_DAG_PREFIX. Prefix in each Airflow DAG file. E.g. your Gluepy DAG is named forecaster and with AIRFLOW_DAG_PREFIX set to "myproject" the resulting Airflow DAGs would be named myproject_forecaster.

  • AIRFLOW_IMAGE. Image name to be used in template using the KubernetesPodOperator.

  • AIRFLOW_CONFIGMAPS. Configmaps to populate environment variables used in the KubernetesPodOperator.

  • AIRFLOW_POD_RESOURCES. Resource limits and requests to be used in the KubernetesPodOperator.

  • KUBERNETES_CONFIG. Optional path to the kubernetes config that allow connection to cluster, used in KubernetesPodOperator.

Create your own CLI commands

To write your own custom CLI commands in your project, see our tutorial Part 4: Custom CLI commands