A webhook is a user-defined HTTP callback triggered by an event on your website or in your application. A webhook allows you to send real-time HTTP notifications from one application to another whenever an event you specify occurs. A webhook URL receives JSON objects as HTTP POST requests.
An event is a JSON object that describes something that happened. As the event occurs, it triggers a small piece of code embedded in the web app that makes an HTTP request (callback) to the URL you configured in the webhook.
With our Webhooks connector, you will get a unique Fivetran URL endpoint that you need to configure into your web app as a callback. You have to register the webhook on your web app to start receiving events. During the registration, provide a name, description, the Fivetran URL, and specify the list of event types you want to capture and sync. Any event that occurs will trigger an HTTP request (containing event data) to the URL endpoint. Our webhook service captures the events. We buffer the events and then push the events to your destination.
Webhooks are predefined instructions and don’t need you to make requests to the web app for data regularly. You just have to set up the webhook once, and your web app sends data whenever it’s triggered, meaning you will have almost real-time updates.
For example, if you have a GitHub repository and want to track the various operations happening on the repository. Configure a GitHub webhook with the Fivetran URL. Whenever any user performs any action like push or pull, an event is triggered, GitHub will POST an HTTP request to the pre-configured URL. Our webhook service processes and stores these events in your destination. You can build dashboards using the data and analyze the event data for insights into your repository.
NOTES:
- If an API that we support uses a webhook, we recommend using the API to connect because webhooks circumvents the transformation of the ETL process.
- We have not optimized our Webhooks connector for a specific source because we built it to be flexible to support and connect with as many APIs as possible.
- We only store the events that we receive after you configure the webhook and cannot fetch historical event data from the web app.
- If there’s a service that we don’t support (have a connector), but the service has a webhooks API, you can use the service’s webhooks to send event data to the Fivetran URL endpoint and we will sync the data to your destination.
Featureslink
| Feature Name | Supported | Notes |
|---|---|---|
| Capture deletes | ||
| Custom data | check | There is no defined schema; it is all custom data. |
| Data blocking | check | Column level |
| Column hashing | check | |
| Re-sync | check | Connector level |
| History | ||
| API configurable | check | |
| Priority-first sync | ||
| Fivetran data models | ||
| Private networking |
Setup guidelink
Follow our step-by-step Webhooks setup guide to connect Webhooks with your destination using Fivetran connectors.
Sync overviewlink
Historical datalink
Webhooks do not allow Fivetran to sync your historical data. We can only capture data from your connection date forward.
Packed vs. Unpacked Datalink
Fivetran can deliver your data in either packed or unpacked format. Suppose you send the following payload to Fivetran:
{
"foo": 1,
"nested": {
"bar": 2
}
}
Your data can appear in the following two ways:
Unpacked
Fivetran unpacks one layer of nested fields and infer types. This produces a table that looks like:
| _id INTEGER | foo INTEGER | nested JSON |
|---|---|---|
| ? | 1 | {"bar":2} |
Packed
Fivetran loads all your data into a single column. This produces a table that looks like:
| _id INTEGER | data JSON |
|---|---|
| ? | {"foo":1,"nested":{"bar":2}} |
Batch datalink
Fivetran can process JSON arrays and deliver separate rows for each array element. Suppose you send the following payload to Fivetran:
[ {
“foo” : 1,
“name” : “John Doe”
}, {
“foo” : 2,
“name” : “Tom Doe”
}, {
“foo” : 3,
“name” : “Emma Doe”
} ]
Fivetran loads 3 rows in to the destination. This produces a table that looks like:
Unpacked
| _id INTEGER | foo INTEGER | name String |
|---|---|---|
| ? | 1 | John Doe |
| ? | 2 | Tom Doe |
| ? | 3 | Emma Doe |
Packed
| _id INTEGER | data JSON |
|---|---|
| ? | {"foo":1,"name":"John Doe"} |
| ? | {"foo":2,"name":"Tom Doe"} |
| ? | {"foo":3,"name":"Emma Doe"} |
Schema informationlink
Because the events you send are completely customizable, you determine how the schema appears. We also support integrating with webhooks through Segment. For every webhooks integration, we’ll make a single table, but you can create multiple webhooks connections to have multiple tables in a single schema.
Type transformations and mappinglink
You can decide the column types and we will faithfully recreate every column that you send to us. Fivetran has some rules for finding the proper data type for a column, detailed below.
-
Fivetran differentiates between a few types - DATES, INTEGERS, DOUBLES, STRINGS, AND BOOLEANS. We parse the information and make our best guess of the types. If we establish a type during one sync of the integration, and then find that the type was too narrow (for instance, we loaded only integers, but the next sync found decimals), we would “promote” the types to the next-most-narrow type that could logically contain both. In the case of INTEGER and DOUBLE, the DOUBLE type will encapsulate both. See below for a concrete example of Type promotion
-
We only write ISO 8601 format dates as TIMESTAMP - everything else gets interpreted as a string. For instance, “2016-10-04T16:05:30Z” will be correctly written as a timestamp in the destination, but “2016-10-4 4:05:30PM” will be written as a string.
Type Promotions Example:link
In the following example, assume you had the following values in a column: 0 2 3 We would initially sync this as an integer column. Then, if you loaded in a new value, 27.54, it would be promoted to a double column. Next, suppose you sent along the value “hello”. We’d promote one last time up to string. String is the “largest” type - every value can have a string representation.
Excluding source datalink
You have total control over what is sent to your endpoint, so you can exclude source data by simply not generating unwanted records.