Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Allow change to degree of parallelism/threads used by connector

Gary Tucker User

Currently Postgres connectors use a multi-threaded approach to pull data from source database. This has caused some of databases to get an alert that disk i/o credits are nearly exhausted. We need the ability to modify the degree of parallelism or the number of threads used per connector. 

Please sign in to leave a comment.

Comments

1 comment

Date Votes
  • Official comment
    Kevin Kim Fivetranner

    Hi Gary,

    Thanks for your post in our feature request portal. As you mentioned, we have recently started rolling out multithreading to various connectors to improve performance. That being said, we don't want to risk customer's systems resources when running syncs.

    Fivetran's key product principles is to minimize configuration for customers. We have been discussing ways to reduce the number of threads used for syncs to reduce impact to the source. We are thinking that this would be a dynamic approach such that if we see high load or low resources available on the database, we would scale down the sync, possibly even to be single threaded again.

    Also, we have recently reduced the max number of threads used for Postgres to 4 sessions at one time. Out of interest, what would be the ideal number of threads used for syncs?