Connector Improvement: Dynamo connector is not scaling effectively: number of tables is bottleneck, not data quantity
Answered
We noticed that the time it takes for our dynamo tables to finish incremental syncs of 100-300mb of data is proportional to the number of tables we have set up and not the data quantity. It appears that each table is taking ~2 min to sync and we have 65 tables connected to this connector. Our syncs are taking ~1.9 hours.
This leads us to believe the extraction is being done in synchronously when it should be done asynchronously.
Entire Sync Duration: 1 hour, 42 minutes
Sync Start: 7:35:18 AM
Sync End: 9:17:38 AM
Status: Completed
Extract Time: 1 hour, 40 minutes
Process Time: 1 minute, 46 seconds
Load Time: 1 minute, 45 seconds
Data Volume: 294 MB
As you can see the extract time is an hour and 40 minutes where the process and load time are only about a minute and 45 seconds.
-
Official comment
Hi Danny,
Apologies for the delay in response here. For a performance related concern, I recommend filing a support ticket so that our team can analyze the performance of your connector.
We are aware of several potential performance optimizations for DynamoDB, and have been tracking these on the backlog. Database performance is a key strategic priority for us, and we will continue working towards that goal!
We have projects on our backlog to improve performance, and will prioritize these projects based on other items on our roadmap.
Please sign in to leave a comment.
Comments
1 comment