Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Opt out for API nested array unroll for API based connectors

Answered
Braun Reyes User

We are currently using the Optimizely connector and recently had to turn off the sync of decision and conversion data because the unrolling of nested arrays was burning through all of our credits.

We do not think we should be forced into this behavior.  For example, if there is an API call that returns 5 items and each item has a nested array of 25 items and your connector is unrolling that array into a supplemental tables as part of your schematization, then we are now using 5 * 25 MARs.

I think we should be able to choose to retain the nested data and turn off the unrolling.  It is much cheaper to do this as a transformation job.

This should definitely be implemented for Optimizely and potentially and consideration for all API based connectors going forward.

Please sign in to leave a comment.

Comments

1 comment

Date Votes
  • Official comment
    Drew Fivetranner

    Hi Braun Reyes, Drew from the Product Team here!

    Thanks for bringing this to our attention. We have been working through our connectors and making changes to our JSON unnesting behavior where we see areas for improvement.

    We'll definitely dig into this deeper and may potentially provide this change alongside a dbt package for our Optimizely connector. I'm putting it in the backlog for now, but I'll update once we prioritize it.