Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Destination Improvement: Primary Key for Individual Topics

Answered
Ryan Rua User

Which destination?:

AWS MSK

Additional details:

We have Fivetran syncing the data into AWS MSK, 1 table to 1 topic. And the Avro schema in Glue.

We are trying to relay the primary key information so that we can pick it up in a separate process consuming from the Kafka topic.

Is there a way for Fivetran to add that primary key information to the Avro schema in Glue? Or add the metadata onto the message?

We see in the documentation it adds the primary key information automatically if Fivetran writes multiple tables to a single topic, but no information if it can do the same for 1 table to 1 topic.

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Egidio Terra Fivetranner

    Hi Ryan,

    Thanks for submitting this request.

    We'd like to understand your architecture a bit better:

    1. Which sources are in the pipeline, and how frequently does each sync?
    2. What downstream processes consume the data from your Kafka topics?

    Best,

  • Ryan Rua User

    Hi Egidio,

     

    Currently the data source being used is DB2 for i that is currently on a 1hr frequency. As the environment is being built out we will ingest different sources such as Oracle, SQL, SFTP, etc with variable sync frequencies but all will land in AWS MSK.