Other: Validate ELT is working
Answered
As a customer I want to know when ELT was not succesful. I discovered that prior to Feb, almost 33% of the data is missing from Snowflake compared to Dynamodb.
A simple cloud function could do that. For instance, the lambda keeps track of the number of records on dynamodb once every 24 hours. and then 15 minutes later(that is our fivetran sync frequency), it checks how many records have been created on snowflake in the preceding 24 hours. If there is divergence of more than a certain threshold it creates an alert.
FiveTran can create this once for each source, destination combo and ask its customers to use it.
-
Official comment
Omer thanks so much for the request! This is something that we've been thinking about but it is not yet prioritized in our roadmap. As a part of our discovery and research, we'll reach out to you to learn more about the specific case and share ideas for solving this problem systematically.
-
Couldn't you create this using a post extraction transformation? In the case of a data warehouse as a destination, you can create your own triggered event which performs this query and creates this alert.
Certainly I would be in favor of Fivetran creating this for users of the platform. I also wouldn't be opposed to creating it (internally) and releasing either as open source – perhaps a collaborative effort of sorts.
-
Also, agreed -> a cloud function is a good solution --> but requires more infrastructure than desired for a fully managed platform as a solution.
Please sign in to leave a comment.
Comments
3 comments