Log to Fivetran Log Connector Overview

Fivetran generates structured log events from connectors, dashboard user actions, and Fivetran API calls. The Fivetran Log Connector delivers your logs and account metadata to a schema in your destination. This metadata includes granular Fivetran consumption information.

Featureslink

Setup guidelink

Follow our step-by-step setup guide to set up your Fivetran Log Connector.

Sync overviewlink

The Fivetran Log Connector syncs the latest logs generated from your connectors to your destination. It syncs additional information about the account, destination, connectors, users, and membership present in your Fivetran account. We sync all these details at a destination level.

If you are an account owner, you can also choose to sync all the connector logs for all the destinations in your account. By limiting this option to account owners, we ensure that Powered By Fivetran (PBF) customers cannot access logs that belong to other PBF customers on the same account.

Log eventslink

Fivetran generates multiple log events depending on your use case enabling you to monitor your connectors and track your usage. For more information about the different log events that Fivetran generates, see our Events documentation.

Formatlink

We write the log events in a standardized JSON format to your destination:

{
    "event": <Event name>,
    "created": <Event creation timestamp in UTC>,
    "connector_type": <Connector type>,
    "connector_id": <Connector ID>,
    "connector_name": <Connector name>,
    "data": {
        // Event specific data. This section varies for each event
    },
    "exception_id": <Fivetran error identifier>
}

We write the logs to the LOG table in the destination. See our LOG table documentation for more information.

Schema informationlink

This schema applies to all Fivetran Log Connectors.

Important tableslink

You can query the data in the different destination tables to monitor different metrics such as connector performance and MAR usage:

ACTIVE_VOLUME tablelink

The ACTIVE_VOLUME table provides details of the monthly active rows (MAR) for each destination schema, the associated tables, and the time when the MAR is calculated.

LOG tablelink

The LOG table provides a record of events that have occurred for a particular connector along with the event details.

CREDITS_USED tablelink

The CREDITS_USED table provides information about the monthly credit usage summary.

Error logslink

These logs contain information about errors that connectors encounter. You can track events, such as a failure or an authentication error from the logs.

The following table lists the errors with a SEVERE severity level:

Destination querieslink

You can query the data in your destination using simple SQL queries. For example, to track schema and table changes in your destination, query the LOG table for the following events:

• create_schema
• create_table
• drop_table
• alter_table

To learn more about the queries you can use, see our Sample Queries page.

Sample query formatlink

SELECT * FROM `project_id.schema_name.table_name`

In this example,

• project_id is your destination’s project ID. To find the project ID, on your Fivetran dashboard, go to the Destination section.
• schema_name is the destination schema name you chose while configuring your connector. The default is fivetran_log.
• table_name is the name of the table you want to query. For example, LOG.