Transformations: Need to be able to have a transformation that creates a view of the most recently updated rows

We currently use the fivetran_audit tables to retrieve the rows that were updated in the most recent run of a connector. The transform creates a view that is used in processing those rows for a business critical application.

We doing this with a join on the sync timestamps

```

select

max(update_started) - interval '5 minutes' as update_started,

max(done) + interval '5 minutes' as update_ended

from {schema}.fivetran_audit where "table" = 'raw_data'

)

select

v.*

from audit

inner join {schema}.raw_data v on v._fivetran_synced between audit.update_started

and audit.update_ended

```

This is not without some problems (the sync timestamps in the audit table appear to be only approximate, for example) but it works.

I noticed a comment recently that the audit tables are being deprecated and will be removed soon. I don't see any other way of identifying the rows updated in the most recent run using the fivetran data, unless I update my connectors to inject a field. That would require us to have custom connectors for all external sources -- which we would like to avoid.

Will there be a way to identify the rows that were updated by a particular sync after you remove the audit tables?

Community

Transformations: Need to be able to have a transformation that creates a view of the most recently updated rows

Comments