Connector Improvement: Select schema/tables for MongoDB Connector
I can programatically create a MongoDB Connector with the Fivetran API, but I can't seem to configure the schemas and tables to enable/disable for syncing when I create the Connector. This means that after I create the connector, I need to manually choose which schemas/tables should or should not be synced. I'd prefer to do this programmatically because there will be hundreds of schemas & tables in the source database(s), and I will know in advance which ones should be disabled (or enabled) - doing this manually does not scale.
I can configure other types of connectors (such as Postgres) with database/schema information at creation time, but I can't see how to do this with MongoDB.
I think that this can be done with the REST API (https://fivetran.com/docs/rest-api/connectors#modifyaconnectorschemaconfig), however all of my code is already written with Terraform/Pulumi - I'd need to write a separate script that accesses the REST API directly in order to disable the schemas & tables I don't want to sync.
I am also wondering if allowing each schema to have its own prefix for the destination would be useful too.
I'm picturing a part of the request to create the connector looking something like this:
{
"schemas": [
"schema_1": {
"database_prefix": "dev",
"databases": ["db1", db3"]
},
"schema_4": {
"database_prefix": "qa_1",
"databases": ["db1", "db2", "db3", "db4"]
}
]
}
I am using the Terraform provider to interact with Fivetran.
Please sign in to leave a comment.
Comments
0 comments