Connector Improvement: S3 .csv connectors should create cols that are empty
PlannedRight now S3 .csv connectors will only create destination table cols if the files contain a non-empty value in them, even when the col is defined. I think this is more of a bug than anything, because the columns show up in the schema definition, but they are not created in the destination table if they are empty. They should be created with null values.
From my support request:
We understand your concern and truly appreciate you bringing this to our attention. At the moment, the Amazon S3 connector does not support creating empty columns in the destination. As a result, columns without data will not appear in the destination schema tables. This is a known limitation and is documented in the connector documentation (please refer to the screenshot below for reference).
We understand that this behaviour may sometimes impact downstream queries. Based on your request and feedback, we kindly ask you to submit a feature request using the link provided. Once submitted, the team will review and evaluate it based on feasibility and priority. They will share updates on the same thread. If approved, the team will work on implementing and releasing the enhancement to create all columns regardless of data presence.
Although this may seem like a small change, any updates that affect the connector schema configuration must go through the formal feature request process. This ensures proper evaluation, communication, and notification to other customers, as such changes could impact downstream queries. I hope this explanation is helpful. Please let us know if you have any further questions, we will be happy to assist you.
-
Official comment
Hi Cody,
Thanks for flagging this. We’re addressing this limitation so that columns are created in the destination table even when the source files contain only empty values. We expect this to be available by the end of this month. If you’d like early access, please reach out to your account manager.
We’ll post updates here as we get closer to broader release.
Best,
Parmeet
Please sign in to leave a comment.
Comments
1 comment