In the event that it is not possible to obtain a historical file for the initial connector setup, the connector will skip over NULL columns in the file when creating the table in the destination which causes issues for downstream processes that will not automatically update based on the base object updating. It would be nice to have a toggle in the connector that would allow you to create those columns if you so choose during the initial sync.

At this time, our workaround for this is creating dummy data in a file in order to create the columns initially, then truncating the table and loading the actual file. This is less than efficient when a file has hundreds of columns.

 

Syncing empty columns

The Amazon S3 connector doesn't support the creation of empty columns in your destination.

We create a column in the destination table only if the column in your source file is not empty.