Connector Improvement: Lambda connector Sync through S3 file key inconsitency
The Lambda connectors with sync through s3 options sends the s3 key (file) in the request sent to the lambda (event). The file key initially was "fivetran_aws_lambda_connector/fivetran/aws_lambda/{file_name}.json" but it was recently changed to "fivetran_aws_lambda_connector/{destination_id}/{connector_name}/{file_name}.json".
It would be better if the s3 key where the lambda uploads the file be sent in the lambda response once the lambda successfully uploads the file. This would let the connector author decide on the s3_prefix and better manage objects in s3.
-
Official comment
Hi Abu,
Thank you for the feature improvement suggestion. I've explored the idea with our engineering and although we can see some benefits we don't think the benefits out way the risks/costs.
When it comes to better managing the file objects we delete the file once we have read it so these objects are only around to be managed for 15-30 mins.Keeping track of which files are where is such a key part of the connector actually working we have lowered the risk of a file name conflict that would be very hard to debug by selecting the file names ourselves.
Best regards
Alison
-
Support ticket ref: https://support.fivetran.com/hc/en-us/requests/109702
Currently when lambda is invoked from fivetran the below even is sent:
Since the file key is sent in the request itself, the function connector doesn’t get to choose the file key in s3. Hence it would be better to send the key in the response by the connector, which would help the connector author to better manage objects in s3. It would also help fivetran to look for file only when its uploaded to s3 and the file key is sent in the response. Below is an illustration of the suggested response that could be sent by the connector.
Please sign in to leave a comment.
Comments
2 comments