New Connector: Redshift Source Connector
AnsweredHi Team,
We have a requirement to consistently ingest data from Amazon Redshift into Databricks, with each batch ranging between 200–300 GB and approximately 4–5 batches per day.
We understand that Fivetran was previously working on a Redshift source connector, which was pulled from preview sometime last year. We’d like to check if this connector is still on the roadmap.
To address our performance needs, we’re exploring an approach where the connector could orchestrate a Redshift UNLOAD operation to a transient S3 bucket, followed by a native S3-to-Databricks ingestion. This could potentially streamline the process and improve performance. Currently, we manage this as a two-step workflow.
Given our strategic direction to unify all data movement workloads under Fivetran, we sincerely request your support in enabling this connector or advising on alternative solutions that align with our architecture.
Looking forward to your guidance.
-
Official comment
Hi Yash,
Redshift does not provide an easily accessible log to capture updates from the source table. We recommend using the Connector SDK, leveraging your knowledge of the data model. You can find a Redshift-specific example here: https://github.com/fivetran/fivetran_connector_sdk/tree/main/examples/source_examples/redshift
Best,
Please sign in to leave a comment.
Comments
1 comment