Connector Improvement: Allow disabling order by in ingestion queries when running initial sync
Not planned
It would be beneficial if we can avoid adding an order by to every query that hits the source database when running initial syncs. Our destination is snowflake, which is capable of optimizing the storage in micropartitions, such that order by while fetching the data would not be required. This clause is also adding an overhead and slowing down the ingestion process where it took us 19 hours to sync our largest table.
-
Official comment
Hi Ritika,
Thank you for taking the time to share your request to allow disabling "ORDER BY" in ingestion queries during initial sync. We appreciate your input and always welcome suggestions to improve our connectors. At this time, this feature is not on our product roadmap. We prioritize new functionality based on demand, which we gauge through feedback and upvotes on requests like yours.
To provide more background, we rely on the source connector’s database capabilities to keep data as fresh as possible. This can sometimes mean that the organization in the destination isn’t fully optimized, since each destination handles data differently and may not perfectly align with how it’s extracted from the source.
Every upvote this request receives helps strengthen the case for us to consider its implementation in the future.
Thanks again for your suggestion!
Best regards,
Egidio Terra
Please sign in to leave a comment.
Comments
1 comment