Built-in Data Count & Row Check Comparison for Pipeline Observability
We would like to see source-to-destination data count and row check comparison reporting added as a core Fivetran capability.
We sync millions of records daily and have encountered situations where records fail to sync, or tables are silently dropped due to various reasons. In most cases, these issues go undetected until we investigate proactively or a problem is reported by an end user. To address this gap, we have had to implement an external application to validate row counts and perform data comparisons between source and destination.
It would be a significant improvement to see Fivetran natively support these checks as part of its observability offering. Having built-in source and destination row count validation and alerting would give teams much greater confidence in their data pipelines and reduce the risk of silent data loss going unnoticed.
We believe this would be a valuable addition for many Fivetran customers operating at scale.
-
Official comment
Hi Subhankar,
This is Ryan, the PM for data quality at Fivetran. We currently have a few data quality initiatives ongoing to make sure that we replicate all your data accurately. We are looking into the best way to offer data checks so you can be confident that we've replicated all your data. It's currently on our long-term roadmap. We'll provide an update here when we have something more to share.
In the meantime, I'd love to learn more about what you're doing as a workaround. Would you be open to chatting? If so, please reach out to me at ryan.waldorf@fivetran.com.
Best,
Ryan
Please sign in to leave a comment.
Comments
1 comment