(An Autonomous Institute of the Department of Social Justice and Special Assistance, Government of Maharashtra)
Caste Certificate Verification Information System (CCVIS)
For explicit control, use xcom_push with a meaningful key:
Suppose we have a workflow that involves processing customer data. We can use XCom to share data between tasks, enabling data-driven decision-making.
# Task A: Push def push_task(**context): return "data": [1, 2, 3], "user": "admin"
from airflow.decorators import dag, task from airflow.utils.dates import days_ago from airflow.models import XCom from airflow.utils.session import provide_session from datetime import datetime, timedelta @dag(start_date=datetime(2026, 1, 1), schedule="@weekly", catchup=False, tags=["maintenance"]) def xcom_cleanup_dag(): @task @provide_session def purge_old_xcoms(session=None): # Define retention boundary (e.g., delete xcoms older than 30 days) retention_date = datetime.now() - timedelta(days=30) deleted_rows = session.query(XCom).filter(XCom.timestamp < retention_date).delete(synchronize_session=False) print(f"Successfully purged deleted_rows older XCom rows from the database.") purge_old_xcoms() xcom_cleanup_dag() Use code with caution.
Airflow keeps XCom data forever by default. Over months of operation, old XCom data can clog up your database. Exclusive pipeline designs include a cleanup task at the end of the DAG. This task deletes XCom rows for that specific run once the pipeline completes successfully. 3. Explicit Cloud Paths
: It handles XComs automatically and results in cleaner, more maintainable code.
: XComs are automatically cleaned up when DAG runs complete, but consider custom cleanup logic for custom backends.
For explicit control, use xcom_push with a meaningful key:
Suppose we have a workflow that involves processing customer data. We can use XCom to share data between tasks, enabling data-driven decision-making. airflow xcom exclusive
# Task A: Push def push_task(**context): return "data": [1, 2, 3], "user": "admin" For explicit control, use xcom_push with a meaningful
from airflow.decorators import dag, task from airflow.utils.dates import days_ago from airflow.models import XCom from airflow.utils.session import provide_session from datetime import datetime, timedelta @dag(start_date=datetime(2026, 1, 1), schedule="@weekly", catchup=False, tags=["maintenance"]) def xcom_cleanup_dag(): @task @provide_session def purge_old_xcoms(session=None): # Define retention boundary (e.g., delete xcoms older than 30 days) retention_date = datetime.now() - timedelta(days=30) deleted_rows = session.query(XCom).filter(XCom.timestamp < retention_date).delete(synchronize_session=False) print(f"Successfully purged deleted_rows older XCom rows from the database.") purge_old_xcoms() xcom_cleanup_dag() Use code with caution. Over months of operation, old XCom data can
Airflow keeps XCom data forever by default. Over months of operation, old XCom data can clog up your database. Exclusive pipeline designs include a cleanup task at the end of the DAG. This task deletes XCom rows for that specific run once the pipeline completes successfully. 3. Explicit Cloud Paths
: It handles XComs automatically and results in cleaner, more maintainable code.
: XComs are automatically cleaned up when DAG runs complete, but consider custom cleanup logic for custom backends.
No. of Visitors51820706