Skip to main content

Community

Destination Improvement: S3 Data Lake - Missing tables causing queries to fail

Please sign in to leave a comment.

Comments

2 comments

    Hi David, 

    Thanks for reaching out. I would love to better understand your use case! Are you asking about the Fivetran Log Connector? 

    Hi,

    Our use case is that we ingest data from ad platforms like Google Ads and Facebook Ads. Sometimes our clients set up connectors as they have an account on those platforms, but haven't actually used it for over 3 months. They are planning to use it so want the connector set up and ready to automatically start ingesting data.

    The way our internal system works is that we run the connector once a day to get the latest data. Then internally we read that data from the destination and ingest it into our platform. We use the S3 Data Lake destination which generates appropriate databases and tables in AWS Glue and we can query then using AWS Athena.

    We see problems when a client hasn't used the ad platform for over 3 months and we set up a connector to get data from the past 3 months. This results in the AWS Glue database missing tables and columns that our internal systems are expected to query. This in turn causes code exceptions that we have to handle.

    This support request is to see if it is possible to have the schema selected tables and columns created in AWS Glue even if there's no data returned from the ad platform. This will mean no more exceptions to handle.

    Thanks