Fivetran generates structured log events from connectors, dashboard user actions, and Fivetran API calls. The Fivetran Log Connector delivers your logs and account metadata to a schema in your destination. This metadata includes granular Fivetran consumption information.
Featureslink
Feature Name | Supported | Notes |
---|---|---|
Capture deletes | check | USER and CONNECTOR tables |
Custom data | ||
Data blocking | check | Column level |
Column hashing | check | |
Re-sync | check | Connector level |
History | ||
API configurable | check | |
Priority-first sync | ||
Private networking | ||
Fivetran data models | check | Get the model |
Setup guidelink
Follow our step-by-step setup guide to set up your Fivetran Log Connector.
Sync overviewlink
The Fivetran Log Connector syncs the latest logs generated from your connectors to your destination. It syncs additional information about the account, destination, connectors, users, and membership present in your Fivetran account. We sync all these details at a destination level.
If you are an Account Administrator, you can also choose to sync all the connector logs for all the destinations in your account. By limiting this option to Account Administrators, we ensure that Powered By Fivetran (PBF) customers cannot access logs that belong to other PBF customers on the same account.
Log eventslink
Fivetran generates multiple log events depending on your use case enabling you to monitor your connectors and track your usage. For more information about the different log events that Fivetran generates, see our Events documentation.
Formatlink
We write the log events in a standardized JSON format to your destination:
{
"event": <Event name>,
"created": <Event creation timestamp in UTC>,
"connector_type": <Connector type>,
"connector_id": <Connector ID>,
"connector_name": <Connector name>,
"data": {
// Event specific data. This section varies for each event
},
"exception_id": <Fivetran error identifier>
}
We write the logs to the LOG
table in the destination. See our LOG
table documentation for more information.
Schema informationlink
This schema applies to all Fivetran Log Connectors.
To zoom, open the ERD in a new window.Important tableslink
You can query the data in the different destination tables to monitor different metrics such as connector performance and MAR usage:
ACTIVE_VOLUME tablelink
The ACTIVE_VOLUME
table provides details of the monthly active rows (MAR) for each destination schema, the associated tables, and the time when the MAR is calculated.
Column Name | Description |
---|---|
connector_id |
The ID of the connector for which the MAR is calculated. |
monthly_active_rows |
The number of distinct primary keys synced for the connector. |
measured_at |
The timestamp when the MAR is calculated. |
schema_name |
The destination schema name for which the MAR is calculated. |
table_name |
The table name associated with the MAR. |
LOG tablelink
The LOG
table provides a record of events that have occurred for a particular connector along with the event details.
Column Name | Description | Data Type |
---|---|---|
id |
The ID of the log event (internal). | STRING |
time_stamp |
Indicates the time when the log event is created. | TIMESTAMP |
_fivetran_synced |
Indicates the time when Fivetran last successfully synced the row. | TIMESTAMP |
connector_id |
The ID of the connector for which the event is logged. | STRING |
event |
The event type. Events can be classified as a warning, an error, or just an information log. | STRING |
message_event |
The routine involved in the log. | STRING |
message_data |
The details of the event in JSON format. | STRING |
transformation_id |
The ID of the transformation (internal). | STRING |
USAGE_COST tablelink
The USAGE_COST
table provides information about the monthly usage summary.
Column Name | Description |
---|---|
destination_id |
The ID of the destination for which the usage occured. |
measured_month |
The month in which the usage is measured. |
amount |
The total dollar amount spent during the month. |
NOTE: If you are using a credits-based plan, see our
CREDITS_USED
table documentation.
Error logslink
These logs contain information about errors that connectors encounter. You can track events, such as a failure or an authentication error from the logs.
The following table lists the errors with a SEVERE
severity level:
SEVERITY | EVENT_TYPE | PROPERTY_NAME | VALUE |
---|---|---|---|
SEVERE | status | status | FAILURE |
SEVERE | status | status | FAILURE_WITH_TASK |
SEVERE | error | type | dynamically obtained from response |
SEVERE | error | type | open_file |
SEVERE | error | type | null_primary_key |
SEVERE | error | type | authentication_error |
SEVERE | error | type | permission_denied |
Destination querieslink
You can query the data in your destination using simple SQL queries.
For example, to track schema and table changes in your destination, query the LOG
table for the following events:
create_schema
create_table
drop_table
alter_table
To learn more about the queries you can use, see our Sample Queries page.
Sample query formatlink
SELECT * FROM `project_id.schema_name.table_name`
In this example,
project_id
is your destination’s project ID. To find the project ID, on your Fivetran dashboard, go to the Destination section.schema_name
is the destination schema name you chose while configuring your connector. The default isfivetran_log
.table_name
is the name of the table you want to query. For example,LOG
.
CREDITS_USED tablelink
You can query the data in the CREDITS_USED
table for information about the monthly credit usage summary.
Column Name | Description |
---|---|
destination_id |
The ID of the destination for which the credits are used. |
measured_month |
The month in which the credits are used. |
credits_consumed |
The number of credits used during the month. |