Native Support for Fixed-Width (Positional) Files in SFTP/File Connectors
AnsweredWhich connector?:
Currently, Fivetran SFTP and file-based connectors do not provide native support for fixed-width (positional) files.
Organizations receiving large volumes of mainframe, legacy, and enterprise-generated files frequently exchange data using positional formats where field boundaries are defined by start position and length rather than delimiters.
To ingest these files today, we must:
- Load each record as a single RAW string column.
- Maintain external metadata describing field positions.
- Implement extensive parsing logic in dbt/Snowflake using SUBSTR operations.
- Rebuild schemas manually whenever layouts change.
This approach introduces unnecessary complexity, increases development effort, impacts maintainability, and makes lineage and schema evolution more difficult than delimited formats.
Add native support for Fixed-Width / Positional Files in SFTP connector.
Suggested capabilities:
- Fixed-width file type selection during connector setup.
- Field definition through UI.
- Start position and length configuration per field.
- Support for STRING, NUMBER, DATE, TIMESTAMP, BOOLEAN data types.
- Optional schema import from metadata files (JSON, YAML, XML, COBOL Copybook).
- Automatic column generation in destination warehouse.
- Schema evolution support when layouts change.
- Validation for record length mismatches.
- Error reporting for malformed records.
- Preview functionality during connector configuration.
Customer improvement:
- Reduced transformation logic in dbt and destination warehouses.
- Faster onboarding of legacy data sources.
- Lower operational and maintenance costs.
- Improved lineage and observability.
- Better support for financial, insurance, telecommunications, retail, and public sector customers where positional files remain common.
- Reduced warehouse compute consumption currently spent on parsing operations.
-
Official comment
Hi Juan,
Thank you for raising this feature request. Native support of fixed-width files in our File connectors is something we are considering. We recognize that handling fixed-width formats is a common need, particularly when integrating with legacy systems or mainframe outputs.Please let us know if you'd be willing to share some sample fixed width files which will help us better assess requirements.
Best,
Parmeet
Please sign in to leave a comment.
Comments
1 comment