Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: [Email Connector] Allow users to accept/reject non-conforming files from email connector

Brandon Beidel User

As part of our regular workflows, we accept files from our clients and vendors that should adhere to some basic assumptions, such as having a consistent file format, a minimum set of columns, and ensuring that those values are populated.  When those incoming files do not meet our expectations we would like to be able to take one of the following actions.

1. [Reject] Delete the file from the ingestion queue
2. [Accept] Allow the file to proceed through the queue, and land in the destination

The validation could be as simple as ensuring that column names exist, or that the contents of a file are of a particular data type or range of values.  As of today, we have very little recourse to deal with misconfigured files that are sent to the email connectors.

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Official comment
    Alison Fivetranner

    Hello Brandon

    Thank you for taking the time to let us know about this need. I'm curious about how you would want to let us know about the correct format to expect? Or would you want to manually review every file before it gets processed?

    I'm wondering how common is this problem and what you end up having to do to deal with it when it does occur? 
    I'm trying to get a better feel of the severity of the problem to help with our prioritization on the issue.

    Regards

    Alison

  • Dalton Malaby User

    Agree with Brandon - generally find the email connector to be very inflexible. If a new column comes in or a file is sent by error, it's very difficult to debug or adjust the files. Inferred data types can also pose issues. I end up having to recreate the sync and generate a new schema, which then requires a full backfill from the previous connector which can cause headaches. The ability to handle these changes over time and remove faulty files would be great!