Transformations: Add option to not filter Quickstart Transforms on Deleted flag
Quickstart transforms filter on the _FIVETRAN_DELETED column. All Transforms have identical code. Here's a code sample:
FROM "RAW"."SALESFORCE"."USER" AS mt
WHERE NOT mt."_FIVETRAN_DELETED"
Our problem is that we're reading from the Quickstart views and building Slowly Changing Dimensions (Type 2) models. When we see a record added or modified, we make the appropriate change to our models. But when the record is soft-deleted, it just disappears from Quickstart views, so we don't know it was deleted. Right now we have records in our models that look like they're active, but have been deleted, the _FIVETRAN_DELETED is True.
We'd like to have the Transformation filter on _FIVETRAN_DELETED to be optional. When we choose to not filter, the records with _FIVETRAN_DELETED set to True would be visible in Quickstart tables/views.
For our purposes, one Y/N option for all Transforms would be perfect. But a separate option for each table would also be fine.
Please sign in to leave a comment.
Comments
0 comments