Connector Improvement: Extract data from databases with pagination
Planned
Currently database connectors using Teleport sync method (tested on Aurora MySQL connector) extract each table's data in one batch, reserving temporary storage in full size of the table for the sync. In services like Aurora where temporary storage size is not customizable this creates situations where a table's size can exceed temporary storage size when reading data from a reader instance, which makes it impossible to sync such tables.
Please improve connectors to allow to sync such tables using pagination or multiple batches without taking up all temporary storage of the database.
-
Official comment
Hi Kirill,
Thanks for your post in our feature request portal!
In fact, we have been planning to implement pagination for Teleport, and have planned to build this in Q2. We will automatically page in order to optimize the Teleport query and to be more cautious with source database resources.
Additionally, we are also committed to improving Teleport specifically for Aurora in Q2 as well. We have noticed that with Aurora, the query performance is not as optimal as with other database services like RDS.
Please if you have any other feedback about Teleport do share it with us.
Please sign in to leave a comment.
Comments
1 comment