Skip to main content

Community

Connector Improvement: Volume Cap for Tables

Please sign in to leave a comment.

Comments

3 comments

  • Official comment

    Hi Kirk, Shira the Product Team here!

    Thank you for sharing your request. I have added your request to feature improvements backlog.

    I am curious to learn if any other alerts on connector-specific MAR spikes would have helped in your case.

     

     

    Hi Shira, 

    I am afraid that Alerts simply don't cut the mustard in these runaway train scenarios. Again, I would say that this is a deal breaker for using Fivetran to migrate Data Warehouses. Data Warehouses are messy things that usually have a sordid history, or that is, a more sordid history in terms of data integrity than transactional databases. All sorts of strange things are done in Data Warehouses, like additions of columns and recreation of tables. It is completely unrealistic to assume that a Data Warehouse should conform to a third normal form (3NF) design pattern, because this has never been the case in Data Warehouse-Land, where the concepts of dimensions and cubes have traditionally ruled the day. So, no Alert in the world is good enough if the train is allowed to runaway, crashing into the population and causing devastation in its wake - there needs to be a Pre-Cook phase (or Pre-Fivetran phase - where 'to Fivetran' is a verb) that has a few logical steps: 1. Estimate the amount change of a a given table that is 'ticked' for replication; 2. Check that estimate against a percentage growth tolerance against the migration volume trend for that table; 3. Stop that table migration Dead In Its Tracks before it can cause any monetary or psychological harm to the Fivetran customer if it is larger than trend * acceptable growth %; 4. Notify said customer that Fivetran has once again prevented the loss and suffering that is incurred by these disasters - Yayyy for Fivetran and its happy customers.

    Kirk, Thank you for that insight!