Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

S3 Files: Header/Trailer Row options for ingestion

Answered
Brittany Young User

Which connector?: Amazon S3

 

Additional details:

  1. Toggle buttons on S3 file connectors to indicate if there is a header and/or trailer row
    1. Toggle button to ingest the row(s)
    2. Free text box to indicate the delimiter of those types of rows
    3. For trailer rows: How many rows make up the trailer
    4. For trailer rows: Is there a character or set of characters that indicate the row has started (free text to indicate that pattern)

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Official comment
    Parmeet Kohli Fivetranner

    Hi Brittany, thank you for raising this request! While we do have toggles to indicate that the file has a header and/or footer, we only provide for skipping them - https://fivetran.com/docs/connectors/files#skipheaderlinesoptional

    To assess this request further, could you elaborate on the specific problem you are trying to solve with these options? For example, what is in the header row? Does it match the data rows in terms of structure or would you want to ingest it to a separate table? A sample file would be super helpful. 

    Best,
    Parmeet

  • Brittany Young User

    Header / Trailer toggles

    • We would like the option to load these rows and indicate their format when not consistent with the rest of the file. I think having a _hdr and _trl or _ftr table that aligns with the table holding the data would be best with the filename and rownum associated for joining purposes.
    • Some of our files have headers that have the business date in them along with expected row counts, in the event a file arrives late, we need a way to identify the business date for process dependencies. It would also be beneficial to be able to use any row counts or field totals during pipeline validations.
    • We also have GL data files that have multiple headers (1, 2) and the data rows are preceeded with a value indicating the header that it applies to. Currently, we have to parse those files into 2 and then drop them for Fivetran to load. It would be helpful if we had a way to do that without "altering" what is sent to us.
    • Some header examples are below: