Scheduling pipelines across projects and jobs with Asset Sensors
Asset sensors in Dagster provide a powerful mechanism for monitoring asset materializations and triggering downstream computations or notifications based on those events.
There are many use cases for asset sensors, however the most common is to trigger a job when an asset is materialized in a different job or code location, or to provide custom logic to determine when to trigger a run following an asset materialization.
Cross-job and cross-code location dependencies
Asset Sensors enable dependencies across different jobs and even different code locations. This flexibility allows for more modular and decoupled workflows.
Here's a minimal example of an asset sensor that triggers a job when an asset is materialized. The daily_sales_data asset is in the same code location for this example, but the same pattern can be applied to assets in different code locations.
Loading...
Custom evaluation logic
The evaluation function of an asset sensor can be customized to include custom logic for determining when to trigger a run. This allows for fine-grained control over the conditions under which downstream jobs are executed.
In this example, the @asset_sensor decorator allows us to define a custom evaluation function that returns a RunRequest object when the asset is materialized and certain metadata is present,
otherwise it skips the run.
Loading...
Trigger a job with configuration
By providing a configuration to the RunRequest object, you can trigger a job with a specific configuration. This is useful when you want to trigger a job with custom parameters based on custom logic you define. For example, you might use a sensor to trigger a job when an asset is materialized, but also pass metadata about that materialization to the job.
Loading...
Monitor multiple assets
This example shows how to use a multi-asset sensor to monitor multiple assets and trigger a job when any of the monitored assets are materialized.
Loading...