Other: Publishing Fivetran audit updates when there were 0 rows updated
Answered
We use the Fivetran audit table to help us determine when we should run jobs downstream of the tables written to by our connectors. Some of these tables aren't always expected to have new writes, but we'd still like to know differentiate between a successful sync with no new rows vs no new rows due to a failed sync. As the documentation states currently, "If there is no update to a table, we do not yet publish 0 rows updated to your table." Hopefully that "yet" means this might already be planned, but if not, would highly appreciate it!
We did look at using the Fivetran Log connector's log table to just check for a successful sync on the connector as a whole, but it doesn't quite give us the per table granularity we'd like. For example, if we uncheck particular tables to stop syncing (when a connector has that option), we want to catch that a downstream job was actually dependent on it and not just let it pass because the connector successfully synced.
-
Official comment
Hi Daniel,
Thanks for the details on your use case. We are planning to deprecate the Fivetran audit table and recommend the Fivetran Log Connector in its place. So let's consider this challenge from that perspective if we may.
Could you share more about these downstream jobs? Are they for data modeling, data integrity validation, or perhaps something else? Ultimately it sounds like your jobs are dependent on specific tables and should be run or not run depending on the sync outcome.- Sync completes and updates table
- Sync completes without updating table
- Sync fails (without updating table)
We can already detect sync success vs failure with the log connector, but we can't tell which tables were updated. Is it that you don't want the jobs to run when there are no table updates? Please let me know!
As an idea, have you considered checking the tables directly for updates via the warehouse?
Please sign in to leave a comment.
Comments
1 comment