Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Add ability to sync database tables based on simple SQL script instead of transactional logs only

Answered
Peter Clausen User

Would be nice to have an option to sync on a specific field in a table, vs. being forced to using the transactional logs.  For example, we have a 3rd party Postgres database that we can't connect to directly with Fivetran, but only a backup of that database that is performed daily.  No transactional logs are part of the backup.  So each day when we sync, Fivetran marks all existing rows in all tables as _FIVETRAN_DELETED = TRUE, and pulls in all the rows again.

Each tables that we sync contains a LAST_MODIFIED_DATE field, so if we could have a simple SQL filter to only upsert records where the LAST_MODIFIED_DATE is greater than yesterday, that would be fantastic!

Please sign in to leave a comment.

Comments

1 comment

Date Votes
  • Official comment
    Kevin Kim Fivetranner

    Hi Peter,

    Thanks for your post on your feature request portal. We will soon be launching a feature, row filtering, which allows users to specify conditions for which rows are synced by a connector. In the upcoming launch, we won't support relative functions like currentDate() - 1, however you can set specific date ranges.

    We are considering relative date functions on our roadmap but I don't have a clear timeline for that yet.