Connector Improvement: Configure S3 Connector to Fail on Schema Change
CompletedWe have several connectors that periodically copy data from CSV files in S3 to Redshift. The data is placed in S3 by an automated, standardized process. Thus, we don't expect any schema changes to this data. If we observed a schema change in these files, it would be a sign that an upstream process had unexpectedly changed and we would like to be alerted to it by sync failure. Instead, at the moment our Fivetran connector automatically accommodates such schema changes, which we then have to troubleshoot in our downstream data models.
We would like to have a configuration option in S3 connectors that would allow us to define an "on schema change" behavior. Options could be: error, warning, or automatically accommodate the schema change with no notification.
-
Official comment
Hi Sarah, thanks for submitting this request! Data type control is something we are considering for our file connector roadmap. We will continue to collect customer demand and keep you posted.
Thank you,
Parmeet -
Please sign in to leave a comment.
Comments
2 comments