Connector Improvement: Schema changes on SFTP ingestions + alerting on schema changes
Answered
Schema changes set to BLOCK does not behave as expected. I ran some tests cases and found the following changes still occur even when this setting is set to block:
Specifically when a column name is changed this is not blocked.
Additionally, when schema changes are blocked there is no notification or alert. Generally this could result in stale data or problems in downstream processes so we need to have an alert on blocked schema changes to come in and address what caused the issue and make sure data is getting ingested.
-
Official comment
Hi Brian, thanks for submitting this request! I agree that column names changes should be blocked when schema changes are blocked. Same about notifications. We will look into both concerns.
Were the number of columns the same in the files before and after the column name change?
Best,
Parmeet -
Hi Parmeet,
Thanks for your comment on this request, I think I missed it sometime ago. I do believe the number of columns was the same in the example. I did a number of tests including with same number and different number of columns as well as with and without changes to new columns, modified column names/types, etc.
Has there been any update on your end from a development standpoint?
Thank you,
Brian
Please sign in to leave a comment.
Comments
2 comments