[Snowflake] Why is there an increase in fail-safe backup storage when using Fivetran to copy data to the Snowflake warehouse?

Gulab

January 05, 2021 00:55
Updated

Question

Why is there an increase in fail-safe backup storage when using Fivetran to copy data to the Snowflake warehouse?

Environment

Destination: Snowflake

Answer

The solution is to use transient tables.

New Connectors

Registering Fivetran with a transient database will ensure all Fivetran created tables are transient by default. This will help in reducing the additional costs and storage space associated with Fail-safe. However, transient tables cannot be recovered in case of operational or system failures.

Existing Connectors

1. Pause the Fivetran database connector

2. Clone the target table into a transient table. This will result in an identical, transient table (make sure that the Schema names, Table Names, Column names, and the column data type are the same).

3. Swap tables

4. Resume Fivetran DB connector

5. Confirmed the sync is working fine

6. Drop the swapped out, permanent table

See Snowflake's Fail-safe documentation for more information.

Cause

Snowflake defaults to Fail-safe. Fail-safe is a backup/storage mechanism from Snowflake which ensures historical data is protected in the event of a system failure or other catastrophic event.