Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Separate S3 Folders for Full Load (Refresh) and CDC in Fivetran HVR

Answered
Naresh Kumar Grandi User

Current Behavior

Fivetran HVR currently generates a single S3 folder/location for both:

  • Full Load (Refresh) data & Change Data Capture (CDC) data

    Problem

    Storing Refresh and CDC data in a single folder creates several operational and governance challenges:

    • Difficult to distinguish full load vs incremental changes
    • Complicates downstream processing logic
    • Increases risk of accidental reprocessing or data duplication
    • Harder to apply different retention, lifecycle, or archival policies
    • Reduced clarity for auditing, debugging, and monitoring

    • Proposed Enhancement

      Provide the ability to configure separate S3 folders/paths for:

      • Full Load (Refresh)
      • CDC (Incremental changes)
      • Eg: s3://<bucket>/<base-path>/full_load/
        s3://<bucket>/<base-path>/cdc/

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Official comment
    Edwin Fivetranner

    Hi Naresh,

    Thank you for your feature request.

    I understand that storing Full Load and CDC data in the same S3 path creates challenges for processing, governance, and retention management. Separating them into distinct folders such as /full_load/ and /cdc/ would improve control and clarity. We will review this request internally with product and engineering to assess feasibility and potential workarounds, and we will follow up with an update.

    Best regards,

    Edwin

  • Naresh Kumar Grandi User

    Expected Benefits

    • Clear logical separation of data types
    • Simplified downstream ETL/ELT pipelines
    • Improved data governance and lifecycle management
    • Reduced operational risk during refreshes
    • Better alignment with enterprise data lake best practices

    Suggested Implementation (Optional)

    • Configuration option at the channel or location level
    • Default behavior remains unchanged for backward compatibility
    • Optional naming convention configurable by the customer

    Business Impact

    This enhancement improves data manageability, clarity, and reliability, especially for customers running frequent refreshes alongside continuous CDC streams.