Other: REST API: GET Schema and PATCH Schema performance improvements
Answered
We have an oracle DB that contains many schemas (5000) with many tables (300) in each schema. To do a "GET schema" on the connector, it takes roughly 10 mins and the response size is 197MB. This is way too long as we are trying to automate the creation of connectors and selection of schemas for each connector. The "PATCH schema" is also taking this long as the response size is also way to big. Is it possible to split the responses so it isn't as big?
for example respond with only enabled schemas and tables
GET 'https://api.fivetran.com/v1/connectors/otter_bacteria/schemas?enabled_schema=true&enabled_tables=true'
and PATCH response contains with only the changed fields
-
Official comment
Hey Shath,
This endpoint might help your team: https://fivetran.com/docs/rest-api/api-reference/connector-schema/pre-create-connector-schema-config
This allows you to pass in the schema config if you already know it before hand and will be immediate, because the response is just what you passed in. This has been useful for other teams with super big schemas, let me know if it helps!
Jimmy
-
I confirm patch endpoint to update schemas takes too much time to complete.
I have an open ticket with this issue as well, Fivetran Support team suggested to use individual tables patch endpoint but even using that endpoint, takes the same amount of time to complete for individual table update cause endpoint is returning all schemas and tables in response
-
Adding to Nestors comment, we are also facing a similar issue with the Terraform code where it takes 23 mins to update the schema/table config where the connector has a large set of schemas and tables.
-
Hi Jimmy,
This "pre-create" endpoint works immediately after the connector creation, but we have use cases where we will need to update an existing connector to enable a schema or a table in the future. This is where it will become problematic.
Please sign in to leave a comment.
Comments
4 comments