Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Generate an S3 touch file (Empty file) immediately following the successful completion of a Fivetran connector syn

Answered
Shivanand Bekinal User

Hi,

we are in need of generating Touchfile - generate an S3 touch file (Empty file) immediately following the successful completion of a Fivetran connector syn. 

we had raised the support request #359317 and we are asked to submit feature request as this currently is not available in Fivetran,

This is very critical for downstream applications to trigger their jobs post the ingestion sync is completed. Please take this on priority  

 

Regards,

Shivanand

Please sign in to leave a comment.

Comments

3 comments

Date Votes
  • Official comment
    Eric O'Connor Fivetranner

    Hi Shivanand,

    Are you using our S3 Managed Data Lake, or is this an HVR issue? (https://fivetran.com/docs/destinations/managed-data-lake-service) What Fivetran feature are you using to write data into S3? 

    I think with regards to MDLS, we notify downstream processes about updates using standard Iceberg catalog updates, not a touch file. 

    Thanks,

    Eric

  • Shivanand Bekinal User

    Hi Eric,

    We use Fivetran Hybrid model with fusion connector. which is to extract data from Fusion and load in to our Redshift table. the downstream job needs a trigger once the data loaded to Redshift. if we have option of generating empty file in S3 (once the data loaded in Redshift table) so that the downstream job can pick the s3 event to trigger itself that's the option we are looking at.  

    Let me know if you need more clarity

     

    Regards,

    Shivanand

  • Eric O'Connor Fivetranner

    Hi Shivanand,

    That use case makes sense. We don't have plans to support doing this with an S3 touch file, but we do have two alternatives I can recommend: 

    1. Register a webhook using our webhook delivery service to notify an AWS Lambda function, AWS Step Function (or any other HTTP service) when the sync completes
    2. Set up external logging to AWS CloudWatch, and then trigger an alert that notifies downstream processes via CloudWatch alerting. 

    I think Option 1 would be my top recommendation for your use case, since it integrates well with other AWS services.

    Thanks,

    Eric