This Quick Start Guide is a curated set of instructions and best practices for implementing the Powered by Fivetran solution.
It’s designed to be consumed in two ways:
-
As a step-by-step instruction manual read from start to finish (recommended).
-
As a handy reference in the event you get stuck or would like to know our best practices for various parts of the implementation process.
Upon completion of this guide, you will have created one or more data pipelines with Fivetran and implemented a repeatable process for onboarding data from your end-users.
NOTE: For the purposes of this guide, end-user refers to any customer, group, or person—internal or external to your organization—that you will be collecting data from.
Prerequisiteslink
-
An active Fivetran Account (sign up for a free 14-day trial)
-
Data —real or sample data— stored in an application, database, Google Sheets spreadsheet or in any of our other supported connectors
-
Credentials or connection details for the data source(s) you want to access
-
Experience using web APIs and token-based authentication
-
REST API client (e.g., Postman)
What is Powered by Fivetran?link
Powered by Fivetran is an implementation of standard Fivetran designed to be embedded into web applications and analytics portals. It includes all of the data replication and automation technology found in our core offering, as well as specific components designed to support the unique requirements of collecting data from an end-user.
Once implemented, Powered by Fivetran offers a reliable, self-service way for your end-users to onboard data into your platform.
TIP: If you’ve ever connected your bank or credit card account to a third-party personal finance application (e.g., Mint, Paypal, Robinhood), then you have experienced this type of functionality first-hand!
Obtain your API keylink
-
Log in to Fivetran.
-
Click the toggle button in the left corner of the dashboard.
-
Click Manage Account.
-
Click on the Settings tab.
-
Scroll down to API Config.
NOTE: You must be an Account Administrator on a Trial, Standard Select, Standard, Enterprise, or Business Critical plan to access this info. If you are not an Account Administrator, ask someone who is an active Account Administrator to update your role by visiting the Users tab on the Manage Account page.
-
Click the Generate button to create a Secret Key. Make a note of your API Key and the Secret Key. You will need them to configure Fivetran.
-
Go to this website and encode the API key and Secret Key. The format should be as follows:
API Key:Secret Key
. Make note of the encoded string. This will beYOUR-API-KEY
for the next steps.
TIP: Having trouble? Check out our REST API documentation for troubleshooting tips and for more detailed instructions on how to connect to the API.
Connect to a data warehouselink
Before you connect to data, you must first decide where you would like Fivetran to deliver it to. Fivetran supports a number of data warehouses and storage services that you can sync data to (see the full list here. These are collectively referred to as destinations.
Assuming you have plans to onboard data from multiple end-users, you have a few options for how to set up your destination(s) based on your needs and your end-users’ needs:
-
Send data to a single destination shared by multiple end-users
-
Send data to a separate destination for each end-user (either your destination or a destination owned by your end-user)
Recommended approaches for setting up destinationslink
Expand to learn more about our recommended approaches for setting up destinationsIf you have plans to onboard data from multiple end-users, you have a few options for how to set up your destination(s) based on your and your end-users’ needs:
-
Send data to a single destination shared by multiple end-users
-
Send data to a separate destination for each end-user (either your destination or your end-user’s destination)
Which data architecture is right for you depends on a number of factors including, first and foremost, your existing data environment.
The following table offers a high-level comparison of the primary approaches:
Single Destination vs. Multiple Destinations
Name | Single destination | Single destination v2 | Multiple destinations |
---|---|---|---|
Description | Deliver data to a single data warehouse shared by all of your end-users | Deliver data to a single data warehouse shared by all of your end-users | Deliver data to multiple data warehouses; one for each of your end-users |
Schema organization | One schema per end-user (e.g., all data for End-user A is sent to Schema A; all data for End-user B is sent to Schema B) | One schema per connector (e.g., Salesforce data for End-user A and End-user B is sent to Schema A; Facebook data for End-user A and End-user B is sent to Schema B) | One schema per connector |
Create destination(s)link
Like many actions in Fivetran, destinations can either be created using the Fivetran user interface (UI) or using the Fivetran REST API. As you navigate this guide, we recommend first completing these activities using the Fivetran UI. This will help you visually understand how objects are represented and organized in Fivetran.
Create destination(s) using the UIlink
Expand for instructionsNOTE: If you’ve already finished creating a destination during your initial setup, you can skip this section. If not, continue to step 1.
-
From the Fivetran dashboard, open the top left dropdown menu and click Manage Account.
-
Click +DESTINATION.
-
Enter a name for your destination and click ADD DESTINATION.
-
Select your destination.
-
Complete the form using the Setup Guide on the right for instructions related to your specific destination. When you’re finished, click SAVE & TEST.
-
Once connection tests pass successfully, click VIEW DESTINATION.
-
That’s it. You’ve created a new destination!
-
If necessary, repeat steps 1 - 7 to create additional destinations within your Fivetran account.
TIP: To toggle between destinations, open the top left drop down and select the desired destination
Create destination(s) using the APIlink
Expand for instructionsNOTE: Before creating a destination using the API, you must first create a group (shown in the next section).
Request
POST https://api.fivetran.com/v1/destinations
{
"group_id":"target_group_id",
"service":"snowflake",
"region":"GCP_US_EAST4",
"time_zone_offset":"-5",
"config":{
"host":"your-account.snowflakecomputing.com",
"port":443,
"database":"fivetran",
"user":"fivetran_user",
"password":"123456"
}
}
Payload parameters
Name | Description | Possible Values |
---|---|---|
group_id (required) |
The unique identifier for the group within the Fivetran system. Find you group_id by fetching your account’s group list |
|
service (required) |
The name for the destination type within the Fivetran system. | Possible destination types |
region |
Data processing location. This is where Fivetran will operate and run computation on data. | GCP_US_EAST4 , GCP_US_WEST1 , GCP_EUROPE_WEST3 , GCP_AUSTRALIA_SOUTHEAST1 , GCP_NORTHAMERICA_NORTHEAST1 , GCP_EUROPE_WEST2 , GCP_ASIA_SOUTHEAST1 , AWS_US_EAST_1 , AWS_US_EAST_2 , AWS_US_WEST_2 , AWS_AP_SOUTHEAST_2 , AWS_EU_CENTRAL_1 , AWS_EU_WEST_1 , AWS_EU_WEST_2 , AZURE_EASTUS2 , AZURE_AUSTRALIAEAST , GCP_ASIA_SOUTH1 . The default value is GCP_US_EAST4 . US , EU , APAC , UK , CANADA , and SINGAPORE are deprecated. |
time_zone_offset (required) |
Determines the time zone for the Fivetran sync schedule. | -11, 10 ... ,0 , ... +11, +12 . |
config (required) |
Destination setup configuration. The format is specific for each destination | |
run_setup_tests |
Specifies whether setup tests should be run automatically. | true or false . The default value is true (tests run automatically by default). |
Response
HTTP 201 Created
{
"code":"Success",
"message":"Destination has been created",
"data":{
"id":"decent_dropsy",
"group_id":"decent_dropsy",
"service":"snowflake",
"region":"GCP_US_EAST4",
"time_zone_offset":"-5",
"setup_status":"connected",
"setup_tests":[
{
"title":"Host Connection",
"status":"PASSED",
"message":""
},
{
"title":"Database Connection",
"status":"PASSED",
"message":""
},
{
"title":"Permission Test",
"status":"PASSED",
"message":""
}
],
"config":{
"host":"your-account.snowflakecomputing.com",
"port":443,
"database":"fivetran",
"user":"fivetran_user",
"password":"******"
}
}
}
If the setup tests pass successfully the setup_status
field will contain connected
.
HTTP 201 Created
{
"code":"Success",
"message":"Destination has been created",
"data":{
"id":"decent_dropsy",
"group_id":"decent_dropsy",
"service":"snowflake",
"region":"GCP_US_EAST4",
"time_zone_offset":"-5",
"setup_status":"incomplete",
"setup_tests":[
{
"title":"Host Connection",
"status":"FAILED",
"message":"Host address is private and not accessible via the internet"
}
],
"config":{
"host":"your-account.snowflakecomputing.com",
"port":443,
"database":"fivetran",
"user":"fivetran_user",
"password":"******"
}
}
}
If the tests fail the setup_status
field will display incomplete
.
HTTP 201 Created
{
"code":"Success",
"message":"Destination has been created",
"data":{
"id":"decent_dropsy",
"group_id":"decent_dropsy",
"service":"snowflake",
"region":"GCP_US_EAST4",
"time_zone_offset":"-5",
"setup_status":"incomplete",
"config":{
"host":"your-account.snowflakecomputing.com",
"port":443,
"database":"fivetran",
"user":"fivetran_user",
"password":"******"
}
}
}
If run_setup_tests
is set to false
in the request the response will not contain the setup_tests
field and the setup_status
field will always contain incomplete
.
Organize your end-users and their datalink
With your destination(s) created, Fivetran now knows where to send data. But before you start creating connectors and syncing data from your end-users, let’s quickly talk about organization.
Stay organized with groupslink
If you have multiple end-users, each with a data source or set of data sources that you will be collecting data from, you’re going to want a way to keep everything organized. This is especially true if you plan to send data across all of your end-users to a single, shared destination.
As we saw in the previous section, your Fivetran account can have multiple destinations associated with it and we call those groups. From each group, you can provision users and connectors.
An obvious and common approach to using groups is to have a separate group for each end-user. Using this approach, users and connectors associated with a particular end-user are added to a single group. This makes it easy to “group” all of an end-user’s connectors in a single view and to provision users with access to the contents related to that specific end-user.
Recommended approaches for setting up groupslink
Expand to learn more about our recommended approaches for setting up groupsThere are several ways groups can be implemented to help you stay organized. To understand your options, consider the following scenarios:
Scenario 1: One group for everything
This is the simplest way to get started with Fivetran groups. Using this approach, all end-users are added to a single group. Often, organization is achieved by naming each created connector after the end-user it is associated with. For example, the connector for Acme Company’s Salesforce account would be acme_company_salesforce
.
Scenario 2: Dedicated group per end-user
Organizing groups by end-user works well for scenarios where end-user segmentation is desired in the destination. For example, each end-user is given their own warehouse in the shared or specific destination platform account.
Scenario 3: Dedicated group per connector type
This scenario works best when you deliver standardized data products built specific to respective data sources which you can segment in Fivetran by the data source connector type.
Scenario 4: Dedicated group per end-user (separate destinations)
Organizing groups by end-user to their own destination warehouses works well for scenarios where each end-user is given/has their own warehouse in a specific destination platform account.
Create a group using the UIlink
Expand for instructionsSince a group is created automatically when a new destination is added, to create a group you simply need to create a destination.
Create a group using the APIlink
Expand for instructionsCreate a new group in your Fivetran account.
Request
POST https://api.fivetran.com/v1/groups
{
"name": "Above_Average_Group_Name"
}
Payload parameters
Name | Description |
---|---|
name (required) |
The group name within the account. The name must start with a letter or underscore and can only contain letters, numbers, or underscores. |
Response
HTTP 200 OK
{
"code": "Success",
"message": "Group has been created",
"data": {
"id": "decent_dropsy",
"name": "Above_Average_Group_Name",
"created_at": "2020-05-25T15:26:47.306509Z"
}
}
NOTE: Changes may have been made to this process since this guide was last updated. See the Fivetran API docs for the most-up-date instructions on Creating a Group using the API.
Capture group IDlink
Once you’ve created your first group, it’s time to make your first request to the Fivetran REST API (that is, if you haven’t already). Capturing your group ID is the starting point for interacting with the REST API.
Capture group ID using the API
Expand for instructions-
Open your preferred
REST API Client
or Command Line Interface (CLI) +cURL
-
Send request
GET https://api.fivetran.com/v1/groups
- Before sending, include an Authorization HTTP header in your request using your encoded API key from earlier. It should look like this:
Basic {YOUR-API-KEY}
- Before sending, include an Authorization HTTP header in your request using your encoded API key from earlier. It should look like this:
-
Locate your group id(s) in the response
TIP: Group IDs are composed of two randomly generated strings separated by an underscore. We hope you find them as comical as we do.
-
Pick a group ID from the response and make note of it. You’ll need this in the next section.
TIP: Group IDs are unique identifiers that allow you to access details about a particular group within the Fivetran system.
Capture group ID using the UI
Expand for instructionsTIP: As you’ll see below, locating a single group ID from the UI only requires a few clicks. However, if you have many groups in your account (and thus many group IDs) you may find it easier to accomplish this task using the API. This method will allow you to capture all group IDs in a single API request.
-
From the Fivetran Dashboard, click Destination in the left menu
-
Find the group ID associated with the destination you are currently viewing in the Fivetran dashboard.
Ingest data from your end-userslink
Now that you have your destination(s) / group(s) set up, you are ready to start collecting data.
With Powered by Fivetran, there are two methods for getting access to end-user data:
Powered by Fivetran Data Onboarding Optionslink
Name | PBF Embedded | PBF Headless |
---|---|---|
DESCRIPTION / USE CASE | End-user uses an embeddable Fivetran setup form called a ‘Connect Card’ to authenticate a connection between their data source(s) and Fivetran themselves. | End-user uses your native user interface to provide credentials necessary to authenticate a connection between their data source(s) and Fivetran themselves. |
END-USER EXPERIENCE - how optimized is the end-user experience? | done done | done done done |
TIME TO ONBOARD DATA - once implemented, how long does it take to onboard data from end-users? | schedule | schedule |
EFFORT TO SETUP - how much effort is required to implement? | schedule | schedule schedule schedule |
SECURITY - how safe is the process? | lock lock lock | lock lock lock |
Requires use of Fivetran REST API? | Yes | Yes |
Before we explore each of these options in more detail, let’s start with the simple exercise of creating a connector. The data you use for this exercise can be from your end-user or from one of your own internal data sources.
TIP: If this is your first time creating a connector with Fivetran, we recommend completing this exercise using the UI. This will help you visually understand how connectors are organized and represented in Fivetran. However, once you’ve created a connector using the UI, it is recommended that subsequent connectors are created using the API. Using the Fivetran API is a requirement to implement the PBF Embedded and PBF Headless data ingestion methods.
PBF Embedded (uses Connect Cards)link
Expand to show detailsWhat is a Connect Card?
A Connect Card is a Fivetran popup that is designed to appear when an end-user attempts to connect one of their data sources to your app or portal. This popup prompts the end-user to authorize a connection between their data source and Fivetran themselves, avoiding the need for any credential sharing between you and your end-user.
NOTE: Configuration of the connector will still be completed using the API. The only action the user will take within the Connect Card is providing authentication information.
What do Connector Cards look like?
The easiest way to understand Connect Cards is through the perspective of an end-user:
A simplified illustration of the Connect Card experience
How do Connector Cards work?
Implementing Connect Cards
To use Connect Card, complete the following steps:
Create a connector using the create a connector API endpoint and get the connector ID from the API response.
TIP: For Powered by Fivetran use cases, since you’ll be ingesting data from your end-users at scale, you’ll likely want to create a connector via the API. Using the API directly (without Connect Cards) will require you to manage the OAuth refresh token experience.
Creates a new connector within a specified group in your Fivetran account. Runs setup tests and returns testing results.
Request
POST https://api.fivetran.com/v1/connectors
{
"service": "criteo",
"group_id": "projected_sickle",
"trust_certificates": true,
"run_setup_tests": true,
"paused": true,
"config": {
"schema": "criteo",
"username": "myuser",
"password": "mypassword",
"app_token": "myapptoken"
},
"auth": {
"client_access": {
"client_id": "my_client_id",
"client_secret": "my_client_secret"
},
"refresh_token": "my_refresh_token"
}
}
Payload parameters
Name | Description |
---|---|
group_id (required) |
The unique identifier for the group within the Fivetran system. Find your group_id by fetching your account’s group list. |
service (required) |
The name for the connector type within the Fivetran system. |
config (required) |
The connector setup configuration. The format is specific for each connector. |
auth |
The connector authorization settings. Can be used to authorize a connector using your external client credentials. The format is specific for each connector. |
paused |
The boolean specifying whether the connector should be paused. |
trust_certificates |
Specifies whether we should trust the certificate automatically. Applicable only for database connectors. The default value is FALSE. If a certificate is not trusted automatically, it has to be approved with Certificates Management API Approve a certificate. |
trust_fingerprints |
Specifies whether we should trust the SSH fingerprint automatically. Applicable only for database connectors. The default value is FALSE. If a fingerprint is not trusted automatically, it has to be approved with Certificates Management API Approve a fingerprint. |
run_setup_tests |
Specifies whether the setup tests should be run automatically. The default value is TRUE . |
Responses
HTTP 201 Created
{
"code": "Success",
"message": "Connector has been created",
"data": {
"id": "speak_inexpensive",
"group_id": "projected_sickle",
"service": "criteo",
"service_version": 0,
"schema": "criteo",
"connected_by": "interment_burdensome",
"created_at": "2018-12-01T15:43:29.013729Z",
"succeeded_at": null,
"failed_at": null,
"sync_frequency":60,
"status": {
"setup_state": "incomplete",
"sync_state": "scheduled",
"update_state": "on_schedule",
"is_historical_sync": true,
"tasks": [],
"warnings": []
},
"setup_tests": [{
"title": "Validate Login",
"status": "FAILED",
"message": "Invalid login credentials"
}],
"config": {
"username": "myuser",
"password": "******",
"api_token": "******",
"service_version": "0"
}
}
}
NOTE: The connector setup configuration (
config
field) is different for each connector type. To help get you started, see the Fivetran REST API docs for an example for your specific connector(s).
TIP: Since the connector is created without specifying authorization parameters, we strongly recommend setting the value of the
run_setup_tests
parameter toFALSE
. This will greatly speed up creating connectors through the API.
OPTIONAL: Set schema configuration using the Modify a Connector Schema Config API endpoint. This will allow you to define exactly what data you bring in from your end-user’s data source
NOTE: For connectors that require a report to be configured (such as our Facebook Ad Insights Connector), review the source system documentation to understand the Measures & Dimensions to be configured to retrieve the required data to support those analytic models.
Best practices:
- Set everything you don’t want the end-user to customize or configure (e.g., tables to enable / disable, column hashing for PII, sync frequency, etc.)
- Set
enable_new_by_default
toFALSE
. This will ensure new tables added to an end-user’s data source don’t automatically start flowing through to the warehouse - For certain apps (like Salesforce), you can choose default schema configuration or leave it empty. Leaving it empty will allow an end-user to modify (block tables, columns, etc.) in the authorization step later
Create a Connect card token using the connector ID. The lifetime of this token is ONE DAY.
Request
POST https://api.fivetran.com/v1/connectors/{connector_id}/connect-card-token
Payload
The request body must be empty.
Response
HTTP 200 OK
Name | Description |
---|---|
token |
The connect-card auth token. |
connector_id |
The connector identifier. |
Redirect your end-user to the Connect Card URL in your user interface, which can be formed in the following way:
https://fivetran.com/connect-card/setup?redirect_uri={redirect_uri}&auth={connect_card_token}
https://fivetran.com/connect-card/setup?redirect_uri={redirect_uri}&auth={connect_card_token}&hide_setup_guide=true
Query parameters
Name | Description |
---|---|
redirect_uri (required) |
The URI on your site where we will redirect the end-user after successful setup. The URI must start with the https or http prefix. |
connect_card_token (required) |
Connect card access token requested in step 3. |
hide_setup_guide (optional) |
Optional parameter that lets you hide the embedded setup guide in the Connect Card UI |
Your end-user will see the following page that allows them to authorize the connector:
NOTE: How to render API responses in the UI of your web app or portal is beyond the scope of this guide. For an introduction to key concepts in this subject area, check out this article.
After authorization, the end-user will click Save & Test.
If the tests pass, the page will be redirected to redirect_uri
specified in step 4 with the query parameter redirect_uri={redirect_uri}
. If the tests fail, the page will not be redirected and the end-user can attempt to reauthorize the connector.
After successful connector authorization.
You can update connector settings through the Fivetran dashboard or through the Modify a Connector public API endpoint.
TIP: Having trouble? Visit our documentation on Connect Cards for the latest information.
Example of an actual Connect Card Implementation
The Connect Card end-user flow in CaliberMind’s application
PBF Headlesslink
Expand to show detailsWhat is PBF Headless?
PBF Headless is our solution for software development teams that want to embed Fivetran in their app but don’t want to use Fivetran Connect Cards.
Instead, developers leverage the Fivetran API behind the scenes and build their own user experience on top of it. This allows you to leverage the powerful functionality of Fivetran while retaining complete control over the front-end experience of your application.
PBF Headless enables software development teams to control the entire experience on the front-end of their application while relying on the Fivetran API to deliver functionality on the back-end
When implementing PBF Headless, you will have to create the equivalent of Fivetran Connect Cards.
How does PBF Headless work?
Unlike Connect Cards, implementing PBF Headless requires you to build a user experience on top of the Fivetran API. This user experience must collect credentials from the user that are required to authenticate a connection between their data source and Fivetran.
Implementing PBF Headless
NOTE: Unlike using Connect Cards, implementing PBF Headless requires you to manage the OAuth token exchange process between the end-user’s application and Fivetran.
-
Confirm your desired connector(s) can be authorized using the Fivetran API
-
OPTIONAL: Take note of our supported authorization methods.
You can use several authorization methods using the API. Each method involves defining the auth
section in the API request (e.g., client_id
, client_secret
, refresh_token
).
{
"service": "salesforce",
"group_id": "tranquil_playmaker",
"config": {
"schema": "testsfdc1",
"username": "dude@fivetran.com.5t",
"password": "XaBXXXXXXXXXX_XXXXXXXX_XXXXXXXXXXX_XXX"
}
}
Method 1: providing the credentials using the login and password fields in the API request.
"auth": {
"client_access": {
"client_id": "my_client_id",
"client_secret": "my_client_secret"
},
"refresh_token": "my_refresh_token"
}
}
Method 2: providing the credentials using the token field and related field(s) in the API request.
{
"service": "airtable",
"group_id": "target_group_id",
"config": {
"schema_prefix" : "test_airtable",
"api_key" : "****"
}
}
Method 3: providing the credentials using the key field and related field(s) in the API request.
-
Review the setup configuration for your target data source(s) and which authorization methods it supports (see the “Authorization” section after clicking on your data source).
TIP: Many of our web application connectors include support for OAuth tokens. Using such a connector, a token can be passed from an end-user’s application to the Fivetran API to authorize a connection.
-
Pick the authorization method(s) you’d like to use for your implementation.
-
Design a user experience in your application that captures the data source credentials required to support the authorization method you selected in Step 3.
-
OPTIONAL: Design a schema configuration experience on top of the Modify a Connector Schema Config API endpoint. This experience would allow a user to define exactly what data is brought in from their data source.
Want to collect data from a database? The following resources outline specific strategies and preparations for connecting to end-user databases:
Connect to your end-user’s data
Expand for detailsIf you have multiple end-users with multiple data sources, chances are they will have different security and networking requirements for how data needs to be ingested by 3rd-party sources such as Fivetran.
This is a quick guide for which connection type is appropriate for which scenario.
Direct Connection
Simple and most relevant to Cloud or Public-Facing subnets.
(Forward) SSH Tunnel
Most security with the least complexity; best suited to end-users with private subnets who can easily set up and maintain an SSH Host and can provide port access.
(Reverse) SSH Tunnel
Parity of security with a Forward SSH Tunnel, more complexity but control over the invocation of the SSH connection; best suited to end-users with private subnets who can easily set up and maintain an SSH Host & who cannot provide direct port access.
VPN Tunnel
The highest level of security, but also the highest level of complexity to set up and maintain; best-suited to end-users who cannot provide any access to their environment or who have complex network configurations that cannot be solved with only an SSH Host.
Name | Direct Connection | (Forward) SSH | (Reverse) SSH | VPN |
---|---|---|---|---|
Security | lock | lock lock | lock lock lock | lock lock lock lock |
Complexity | 1 | 2 | 3 | 4 |
End-User Effort | schedule | schedule schedule | schedule schedule schedule | schedule schedule schedule schedule |
Prepare your end-users to prepare their data
Expand for detailsData Design
It is important that you understand what data you want to sync through Fivetran, as this will have a knock-on effect on the optimal configuration of Fivetran.
What Fivetran Does | What your End-Users Should Do |
---|---|
The first initial sync or ‘initial historic sync’ will replicate the entire database data into your destination warehouse for all tables the ‘fivetran’ user was given SELECT permissions to except those excluded from the sync within the Fivetran dashboard | Review the data you currently hold within the database to ensure you are only syncing data that you want to land in your data warehouse |
Database Connection: Master vs Read Replica
You should consider which copy of your database Fivetran connects to because there can be knock-on effects to load under usage depending on which copy you choose. The incremental update strategy you choose - or indeed, already have enabled - will also determine which copy you proceed with.
What Fivetran Does | What your End-Users Should Do |
---|---|
Fivetran can connect to either a Master or Read Replica database depending on the database type | Review and decide on which copy of your database you want to connect to and take into account the options around Change Log configurations per-database dependent on the instance you proceed with |
Change Log Configuration
The incremental update mechanism you choose to implement or have already implemented will have an impact on both the Fivetran configuration and the knock-on effect to your database.
What Fivetran Does | What your End-Users Should Do |
---|---|
Each database has its own Change Log mechanism and associated configuration best-practices; however, there are also shared commonalities | Ensure that the Change Log retention period is longer than the sync frequency of your Connector in Fivetran; for example, if your sync frequency in Fivetran runs every 15 minutes your retention period cannot be less than 15 minutes long |
Connector Setup: Number of Connectors
This concern is linked to the make-up of your database and will impact the throughput of your Fivetran data pipeline.
What Fivetran Does | What your End-Users Should Do |
---|---|
Creating one single connector is easier to manage, whereas creating separate connectors lets you track and isolate changes coming to the database and keep them independent of each other | Be mindful of how you may want to structure your Connectors; examples could be to split out dynamic from more static tables, or separating out problematic tables for debugging |
Fivetran API Referencelink
As you’ve learned in this guide, Fivetran offers a REST API to provide developers with a way to programmatically interact with the Fivetran platform.
This section offers a short exercise to help you get acquainted with the API and outlines a subset of its endpoints that are most relevant to the getting started process for PBF.
TIP: Before starting the exercise, or using the API in general, we highly recommend leveraging our Postman Collection. This Collection contains templates to help you create API calls for all endpoints supported by Fivetran. See instructions for installing it into your Postman environment here.
Exercise: Starting out with the Fivetran APIlink
Expand to see the exerciseTo help you get familiar with the API, particularly as it relates to Powered by Fivetran, complete the following exercise:
Open items
Open your preferred REST API Client
or Command Line Interface (CLI) + cURL
(for this exercise, we will use Postman).
Retrieve your group ID
Why start here? Connectors and users are often organized by groups. And many PBF implementations use a “one group per end-user” policy. For both scenarios, retrieving your group ID gives you the ability to specify which set of connectors or which similar end-user you’d like to interact with using the API.
Expand for instructions-
Send
GET https://api.fivetran.com/v1/groups
request to the Fivetran API.NOTE: Before you submit the request, include an Authorization header with the following value:
Basic {Your-Base-64-encoded-API-Key}
. Don’t have your Base64 encoded API key? See instructions for obtaining it, here. -
Find a group ID in the response that is associated with a group containing one or more connectors. Take note of that group ID.
List all of the connectors associated with your group ID
Expand for instructions-
Send
GET https://api.fivetran.com/v1/groups/{your_group_id}/connectors
request to the Fivetran API. Make sure to incorporate the group ID you noted in the previous step. -
The response will return a list of details about each of the connectors in the specified group.
NOTE: You can display connector metadata from this response in the UI of your web app or analytics portal. This will give end-users information about their connectors that they’d likely be interested in—such as the status of each of their connectors and the last time their data was synced successfully.
-
Take note of the connector ID for one of the connectors listed.
User Managementlink
Expand to show API requestsList All Users
Returns a list of all users within your Fivetran account.
Invite a User
Invites a new user to your Fivetran account. The invited user will have access to the account only after accepting the invitation. Invited user details are still accessible through the API.
Group Managementlink
Expand to show API requestsList All Groups
Returns a list of all groups within your Fivetran account.
List All Connectors within a Group
Returns a list of information about all connectors within a group in your Fivetran account
Destination Managementlink
Expand to show API requestsCreate a Destination
Creates a new destination within a specified group in your Fivetran account.
See all destination management actions supported by the REST API
Connector Managementlink
Expand to show API requestsRetrieve Connector Details
Returns a connector object if a valid identifier was provided.
Create a Connector
Creates a new connector within a specified group in your Fivetran account. Runs setup tests and returns testing results.
NOTE: The connector setup configuration (
config
field) is different for each connector. Click here to find the specific setup configuration for your connector.
Sync Connector Data
Triggers a data sync for an existing connector within your Fivetran account without waiting for the next scheduled sync. This action does not override the standard sync frequency you defined in the Fivetran dashboard.
If there is a data sync already in progress, that sync is immediately stopped and restarted. If the connector is paused, the data sync will be scheduled to be performed when the connector is re-enabled.
Retrieve a Connector Schema Config
Returns the connector schema config for an existing connector within your Fivetran account.
Modify a Connector Schema Config
Updates the schema config for an existing connector within your Fivetran account (for a single schema for a connector with multiple schemas).
Modify a Connector Database Schema Config
Updates the database schema config for an existing connector within your Fivetran account (for a single schema within a connector with multiple schemas).
Create a new connector based on an existing connector’s parameters
You can create new connectors using the parameters of an existing connector.
See all connector management actions supported by the REST API
TIP: Visit the Fivetran REST API docs to see a comprehensive and up-to-date account of the entire Fivetran REST API in our documentation.
FAQslink
Why does it say “Fivetran” on connect cards?
GDPR requires it or you need to disclose this in your privacy policies including Fivetran as a subprocessor. We’re the registered owner of the OAuth relationship that powers many of the connectors - by disclosing our brand and providing the user with context you can build trust with users!
What if all my connectors will have the same setup?
Just use two of our REST API calls to speed things up! Manually setting this up within the dashboard would be unnecessarily tedious and cumbersome. See how to create a new connector based on an existing connector’s parameters.
What kind of businesses use Powered By Fivetran?
- Companies in the Martech, Ad Tech, Media buying and Agency space. All companies in this industry are funded by their clients to operate as a service or technology layer in MKTG efforts Ex: Starcom, Publicis, Media Associates, Epsilon, Varick Media, MDC Partners, GroupM
- Companies with analytics and insights at their core offering. These are typically companies that present dashboards, Bl tools, or the output of AI & Machine Learning models to their customers (e.g., Latent View Analytics, Stardog, Exist BI, R Systems, Flywheel Technologies)
- Software platforms built to activate data which relies on customer data input to function. These platforms are created to be embedded into existing infrastructure (e.g., Splunk, Tableau, Databricks, Alteryx, ActionIQ, Lytics)
Is there a different cost for Powered By Fivetran vs. a standard Fivetran account?
There is no difference in cost for using Powered By Fivetran, however you need to be on the Standard Select, Standard, Enterprise, or Business Critical plan in order to access our API. Any Fivetran pricing plans can be purchased depending on your functional needs.
Can I get Powered By Fivetran features and services with a Fivetran account set up using Partner Connect in my data warehouse or other platform?
Yes! Powered By Fivetran features and services are available for any Fivetran customer looking to support multiple end-users and/or embed Fivetran into their own product or workflow.
What kind of support is available with Powered By Fivetran?
During initial evaluation and once signed on as customers, Powered By Fivetran customers have access to a dedicated account and support team including account executives, sales engineers, customer success managers, technical success managers and support staff.
Are there limits to how many connectors a Powered By Fivetran account can have?
There are no limits to the amount of connectors a Powered By Fivetran account can have; however, we do have rate limits in our REST API.
What are Connect Cards?
Fivetran Connect Card is an embeddable setup form that enables users to securely pass and authenticate credentials directly to Fivetran using a secure HTTPS connection. Your end-users can leverage 150+ Fivetran connectors without credentials ever touching your own servers. Click here for more info on Connect Cards.
Do Powered By Fivetran customers have to use Connect Cards?
No, Connect Cards are an optional feature to allow an end-user self-authorization workflow. See the data ingestion section of this guide to learn about other options.
Do Powered By Fivetran customers have to use the Fivetran REST API?
The REST API is another optional, but highly useful feature that allows Powered By Fivetran customers to programmatically perform all of the operational and administrative features of Fivetran data pipelines.
Are Transformations included in the Fivetran REST API?
At this time, Fivetran transformation tools are not available through the REST API.
What kind of security comes with Powered By Fivetran?
Powered By Fivetran has the same world-class security that comes with the Fivetran platform. All data is encrypted at rest and in transit. Fivetran is SOC2 Type2 certified, GDPR compliant, HIPAA BAA compliant and is regularly audited and certified for data industry best practices.