Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector improvement : Destination AWS Redshift & RDS

Not planned
Naynesh Parmar User

Hi Support team,

As per Fivetran support document "fivetran_synced (UTC TIMESTAMP) indicates the start time of the job that synced this row."

Considering practical scenario I have faced with fivetran, one of the sync went longer than 24+ hours. Due to this scenario _fivetra_synced column contain value of when sync started w.r.t specific tables. Now when I want to extract delta (T-1) in Data Warehouse process, there are few records missed due to long running sync & _fivetran_synced column's value. And that required manual fix to the data which is complex and time consuming task.

So, I would recommend that Fivetran should add additional column to Redshift / RDS tables and value of that column must be actual current timestamp when individual record is inserted to destination (Redshift / RDS). This will help to resolve this corner case data mismatch situations. 

Please revert me back if you need any further information.

Please sign in to leave a comment.

Comments

5 comments

Date Votes
  • Official comment
    Kevin Kim Fivetranner

    Hi Naynesh,

    Thanks for your question here! This sounds like a valid feature request, as the sync time and the insert time can be two different times, as you noted.

    Could you elaborate on the use case for the insert timestamp? Are you querying the destination using _fivetran_synced to export the data to another warehouse, or another table possibly?

    I can flag this as a feature request for the team. We don't currently have this planned, but will discuss it among other feature request priorities!

  • Naynesh Parmar User

    Any update ??

  • Naynesh Parmar User

    Hi Support team,

    Please find answers :

    Could you elaborate on the use case for the insert timestamp?

    • As we need to use delta for maintaining history in Data Warehouse, every single transaction actually inserted in Data Warehouse using fivetran matters a lot.

    Are you querying the destination using _fivetran_synced to export the data to another warehouse, or another table possibly?

    • We already had a logic with _fivetran_synced column for delta extraction, but it is causing data missing issue when any particular sync ran longer or any other issues. So rather than using dated column when data read started from logminer, it will be more accurate to use column when record inserted into Data Warehouse.

    Please revert me if you need any further information from my end. Also kindly keep me updating about your tentative timeline when this feature will be available in fivetran.

    Thank you.

  • Kevin Kim Fivetranner

    Hey Naynesh,

    Thanks for your feedback and follow-up to my questions! I understand that since some syncs can take long, using a specific timeframe for the extraction may cause missing data in your use case.

    I will discuss this with our engineering team, but please note this is not planned or committed on our product roadmap. We do not currently have a timeline that we can share.

  • Naynesh Parmar User

    Hi Fivetran team,

    It's been 6 months I have raised a feature request. I would like to followup on this feature request, whether it has been keep in planned activity in your roadmap? Is there any tentative timeline for it ?? 

    cc: rhiannon.fowler@fivetran.com