Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: SFTP Pre-Validation

Answered
Vignesh Meyyappan User

This is request raised as part of support ticket: 229729

We would like to have some pre-validation in SFTP connectors to ensure we do not fully rely on producers to keep the right amount of files. 

For example we expected 7 files to be present inside a gz file every time when we perform the load but this time producers who are external to our org have placed 35 files (5x duplicates) and we end up paying for unnecessary MAR which are just mere duplicates. So we would like to have options to perform number of files in a GZ before initiating the load. If we can have some custom shell wrapper execution that offers a great flexibility too, but I'd let the product team decide on this. Some of these validations could be,

  • Number of files
  • Number of expected rows (Range)
  • Number of columns inside a file

Thanks

Skyscanner. 

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Official comment
    Parmeet Kohli Fivetranner

    Hi Vignesh, thank you for submitting this request. Which file mode are you using, Magic or Merge? In Merge Mode you are able to block new columns by using the Block All schema change setting - https://fivetran.com/docs/using-fivetran/features/data-blocking-column-hashing/config#configureschemachangesettingsforexistingconnectors
    For number of files, can that controlled using a file pattern?

    Thank you,
    Parmeet

  • Vignesh Meyyappan User

    Hey Parmeet,

    This is not about new columns coming in, source provider placed duplicated data files in different file name but same schema (Timestamped) ie., instead of 7 files, 14 files were kept in a GZ and fivetran processed everything accounting into MAR calculation which affected our budget. Ask here is control this either by limiting the number of expected rows for a connector or number of files expected from a source.

    Thanks

    Vignesh