Apache Kafka is a distributed streaming platform.
Fivetran supports consuming Kafka messages from:
Fivetran creates one table for each topic.
IMPORTANT: You can choose which topics to sync on the Schema tab in your Fivetran dashboard.
For each table it creates
offset are the primary keys.
timestamp column may contain either
log_append_time as per the server configuration.
You can select to sync
packed messages, Fivetran syncs the message in
unpacked messages must be in
For all the first level
JSON elements, Fivetran creates a separate column.
The column names are formed using
After making the connection, Fivetran starts syncing all available messages from the Kafka topics. It goes to the earliest available offset for each partition of a topic and starts consuming the messages. It loads the messages into the warehouse. After the retention period the messages are deleted from the Kafka topics. The deleted messages won’t be synced so if you happened to resync the connector it would only fetch the current available messages.