Destination Improvement: S3 Data Lake - Missing tables causing queries to fail
We sometimes only ingest the last 3 months of data. So for Connectors such as Facebook Ads we want to pull in cost data for the last 3 months but occasionally there is no cost data to ingest. This results in the table not being created in AWS Glue which means all of our queries fail with the exception "MISSING TABLE".
Would it be possible to create the table even if there is no data returned from an API?
Thanks
-
Hi David,
Thanks for reaching out. I would love to better understand your use case! Are you asking about the Fivetran Log Connector?
-
Hi,
Our use case is that we ingest data from ad platforms like Google Ads and Facebook Ads. Sometimes our clients set up connectors as they have an account on those platforms, but haven't actually used it for over 3 months. They are planning to use it so want the connector set up and ready to automatically start ingesting data.
The way our internal system works is that we run the connector once a day to get the latest data. Then internally we read that data from the destination and ingest it into our platform. We use the S3 Data Lake destination which generates appropriate databases and tables in AWS Glue and we can query then using AWS Athena.
We see problems when a client hasn't used the ad platform for over 3 months and we set up a connector to get data from the past 3 months. This results in the AWS Glue database missing tables and columns that our internal systems are expected to query. This in turn causes code exceptions that we have to handle.
This support request is to see if it is possible to have the schema selected tables and columns created in AWS Glue even if there's no data returned from the ad platform. This will mean no more exceptions to handle.
Thanks
Please sign in to leave a comment.
Comments
2 comments