Transformations: Improve the scheduling of different frequency transformations
To explain, I'll use an example:
Let's say we have 2 sources - Source X and Y.
If Source X syncs every 30 minutes, and Source Y syncs every 1 hour, and we have models that only use either X or Y, and one model that uses both, what should happen is:
- The model that uses both X and Y should be triggered every 30 minutes
- The model that uses only X should also be triggered every 30 minutes
- The model that uses only Y should be triggered every 1 hour
So ideally, we would see a dbt run that includes the first 2 models every 30 minutes, but every other run will also include the third model.
Right now, all models will just trigger based on the fastest frequency, causing transformations to run more frequent than they are intended to, which incurs processing costs.
Please sign in to leave a comment.
Comments
0 comments