Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Support gzipped response from Azure Functions

Answered
Sven Slijkoord User

Currently the Azure Functions connector doesn't accept gzipped responses. We would greatly benefit from gzip support as Azure Functions doesn't seem to handle responses larger than 100MB. When retrieving these responses gzipped they are far smaller and have no problem being processed by Azure.

Please sign in to leave a comment.

Comments

3 comments

Date Votes
  • Official comment
    Alison Fivetranner
    • Edited

    Thanks Sven,

    Handling large responses efficiently is one of the things that led us to introduce our new Connector SDK. Getting the data from the source and passing to Fivetran while checkpointing regularly should allow Connector SDK to much more easily address this type of very large use case.

    We will continue to consider prioritizing gzip support for function connectors in our roadmap.

    Best regards
    Alison

  • Alison Fivetranner
    • Edited

    Hi Sven,

    Thank you for taking the time to submit the request. Can you share more about the connector you are writing that is producing responses over 100MB and gzipped and your full use case?

    This will help us prioritize the ask in our roadmap.

    Best regards

    Alison

  • Sven Slijkoord User

    Hi Alison,

    Thank you for your reply!

    The connector we were trying to write is an Azure Function that extracts data from a remote API. The application behind this API uses event-sourcing (registers all changes to all models). The event-sourcing and just sheer amount of data ends up being a lot of data.

    We already configured the function to only retrieve data on a day-to-day basis but then it still is going over the ~100MB limit.

    Does this answer your question?

    Many thanks,
    Sven