Connector Improvement: SFTP connector improvements
Completed
Refer to ticket ID #282086 for more info but in summary:
When our OPS team refreshes the SSH we need to manually reconfirm SSH on 80+ SFTP connectors. There needs to be a bulk option (as there is for bulk pause, sync, etc).
Second request is that when we have an error in an SFTP connector, in order to get past the error we need to have a file in the SFTP location. I understand that for setup there must be a file present but in our use case these are connectors that have already been setup. We have a 2 week data retention policy so there are times when an SFTP location can be empty but we still need the connector to re-establish the connection.
I have noticed there is no manual way to do this. What does work is catching this in time so that the SSH key can be refreshed and then leaving the connector in a failed state so the Fivetran auto-retry logic grabs it and gets it back to an active (no error) state. However, there is a risk that we miss this window and without a file present cannot get past the error.
-
Official comment
Hi Brian, thank you for filing this request!
Can I ask you to expand on what your Ops team does to reconfirm SSH for each connection? One way of solving a per connection operation would be to consolidate your SFTP connections now that you can define multiple file patterns/tables in a single connection - https://fivetran.com/docs/connectors/files#mergemode. Note that to consolidate connections, apart from the file path and table name, all other configuration (host, file type, destination schema, format configuration, etc.) needs to be the same.
For the second request, I can see how that is a problem in this case. I'll get back to you with a timeline once we ideate on a solution.Thank you,
Parmeet -
Hi Brian, you should now be able to edit and save SFTP connections without a file present in the source. Thank you for filing this request!
Best,
Parmeet
Please sign in to leave a comment.
Comments
2 comments