Logs Overview

Fivetran generates structured log events from connectors, dashboard user actions, and Fivetran API calls. The logs provide an internal view of the operations happening with each of your connectors. You can use these logs in the following ways:

• in your Fivetran dashboard
• in your destination using the Fivetran Log Connector
• in an external logging service

In your Fivetran dashboardlink

You can view both user actions and logs in your Fivetran dashboard.

• View user actions on the Status tab of the Connector Details page. The User Actions list contains a complete user action audit.
• View logs on the Logs tab of the Connector Details page. You can search up to one week of log records.

Learn more about the Fivetran dashboard’s user actions and logs in our Sync troubleshooting documentation.

In your destinationlink

The Fivetran Log Connector is a free connector that delivers your logs and account metadata to a schema in your destination. This metadata includes granular Fivetran consumption information. The Fivetran Log Connector is available on all plans. Learn more in our Fivetran Log Connector documentation.

In an external logging servicelink

Important: You must be on an Enterprise plan to use an external logging service.

Fivetran can send logs to the following external services:

• AWS CloudWatch
• Datadog PRIVATE PREVIEW
• Google Stackdriver
• Azure Log Analytics
• Splunk

You can connect one logging service per destination. Fivetran will write log events for all connectors in the destination group to the connected service. If there is a logging service that you would like but that is not yet supported, please let us know.

Log eventslink

The log events are in a standardized JSON format:

{
    "event": <Event name>,
    "created": <Event creation timestamp in UTC>,
    "connector_type": <Connector type>,
    "connector_id": <Connector Id>
    "data": {
        // Event specific data. This section varies for each event
    },
    "exception_id": <Fivetran Error Identifier>
}

The field exception_id will only be present if Fivetran encountered an unexpected problem.

Eventslink

Fivetran connectors generate the following events:

Dashboard activities generate the following events:

Basic SQL transformations generate the following events:

dbt transformations generate the following events:

Connector stages and related log eventslink

This section describes the life cycle of a connector. It lists the corresponding log event generated at each stage of the connector, or connector-related dashboard activities. This will help you recognize the events in the logs and understand how their ordering relates to the operations of the connector.

All listed connector-related log events for a particular connector are displayed on the connector’s Logs tab in your Fivetran dashboard. The connector-related log events are included in the logs captured by the Fivetran Log Connector and external logging services.

NOTE: The connector life cycle stages as are listed in chronological order where possible.

1. Connector creation initializedlink

When you create a connector, Fivetran writes its ID in Fivetran’s database and assigns the connector status “New.”

NOTE: Fivetran writes encrypted credentials in its database.

If you create a connector in your Fivetran dashboard, you need to specify the required properties and authorization credentials.

If you create a connector using the Create a Connector API endpoint, you need to specify the required properties. However, you can omit the authorization credentials and authorize the connector later using the Connect Card.

For this stage, Fivetran generates the following log events:

2. Connector setup tests runlink

During these tests, Fivetran verifies that credentials such as authentication details, paths, and IDs are correct and valid, and resources are available and accessible.

When you click Save & Test while creating a connector in your Fivetran dashboard, Fivetran runs the setup tests.
Also, you can run the setup tests by using the Run Connector Setup Tests endpoint. If the setup tests have succeeded, it means the connector has been successfully created.

For this stage, Fivetran generates the following log events:

3. Connector successfully createdlink

After the setup tests have succeeded, Fivetran records the Connection Created user action in the User Actions Log. At this stage, the connector is paused. It does not extract, process, or load any data from the source to the destination.

After the connector has been successfully created, you can trigger the historical sync.

For this stage, Fivetran generates the following log events:

4. Connector schema changedlink

Connector schema changes include changing the connector schema, tables and table columns.

You change your connector’s schema in the following cases:

• You want to change what data your connector syncs.
• You need to fix a broken connector.
• You need to change a connector schema as part of the schema review. The schema review after you have created a connector and before you run the historical sync is required for particular connector types.

You can change your schema in the following ways:

• In your Fivetran dashboard
• With the Fivetran REST API using any of the following endpoints:
  • Modify a connector schema config
  • Modify a connector database schema config
  • Modify a connector table config
  • Modify a connector column config

For this stage, Fivetran generates the following log events:

NOTE: For an un-pause connector, changing the schema will trigger a sync to run. If a sync is already running, Fivetran will cancel the running sync and immediately initiate a new one with the new schema configuration.

5. Sync triggeredlink

You need to run the historical sync for the connector to start working as intended. The first historical sync that Fivetran does for a connector is called the initial sync. During the historical sync, we extract and process all the historical data from the selected tables in the source. Periodically we will load data into the destination.

After a successful historical sync, the connector runs in the incremental sync mode. In this mode, whenever possible, only data that has been modified or added - incremental changes - is extracted, processed, and loaded on schedule. We will reimport tables where it is not possible to only fetch incremental changes. We use cursors to record the history of the syncs.

The connector sync frequency that you set in your Fivetran dashboard or by using the Modify a Connector endpoint defines how often the incremental sync is run.

IMPORTANT: Incremental sync runs on schedule at a set sync frequency only when the connector’s sync scheduling type is set to auto in our REST API. Setting the scheduling type to manual effectively disables the schedule. You can trigger a manual sync to sync the connector in this case.

NOTE: Some connector types support the priority-first sync mode.

For this stage, Fivetran generates the following log events:

6. Connector paused/resumedlink

When you have just created a connector, it is paused, which means it does not extract, process, or load data. After you successfully run the setup tests, the connector becomes enabled/resumed. After the successful initial sync, it starts working in incremental sync mode.

You can pause and resume the connector either in your Fivetran dashboard, or by using various Connector endpoints in our API.

For this stage, Fivetran generates the following log events:

NOTE: Resuming a connector will trigger either the initial sync, or an incremental sync, depending on the stage the connector was at when it was paused.

7. Re-sync triggeredlink

In some cases you may need to re-run a historical sync to fix a data integrity error. This is called a re-sync. We sync all historical data in the tables and their columns in the source as selected in the connector configuration.

You can trigger a re-sync from your Fivetran dashboard. You can also trigger a re-sync by using the Modify a Connector endpoint.

For connectors that support table re-sync, you can trigger it either in the dashboard or by using the Re-sync connector table data endpoint.

For this stage, Fivetran generates the following log events:

8. Manual sync triggeredlink

You can trigger an incremental sync manually without waiting for the scheduled incremental sync.

You can do this either by clicking Sync Now in your Fivetran dashboard or by using the Sync Connector Data endpoint.

For this stage, Fivetran generates the following log events:

9. Sync endedlink

A sync can end in four states:

• Succeeded - the sync was completed without issue and data in the destination is up-to-date.
• Failure - the sync failed due to a unknown issue.
• Failure with Task - the sync failed due to a known issue that requires the user to take actions to be fixed. A Task is generated and displayed on the Alerts page in the dashboard.
• Rescheduled - the sync was unable to complete at this time and will automatically resume syncing when it can complete. This is most commonly caused by hitting API quotas.

When the sync ends, Fivetran generates a status log event:

10. Connector brokenlink

IMPORTANT: This is an abnormal state for a connector. It commonly happens due to transient networking or server errors and most often resolves itself with no action on your part.

A connector is considered to be broken when during its sync it fails to either extract, process, or load data.

You can see a connector in the Connector List is broken when it has the red Broken label.

In the Connector dashboard, a broken connector also has the red Broken Connection label. Its log in the Logs tab will display the status log event with the red pictogram. Click on the arrow button to expand the log event and see its details.

If we know the breaking issue, we create a corresponding Task and notify you by email with instructions on how to resolve the issue. The Task is displayed on the Alerts page in your Fivetran dashboard. You need to take the actions listed in the Task message to fix the connector. We resend the Task email every seven hours until the Task is completed.

If your connector breaks and the breaking issue is unknown, we do not generate a Task in your dashboard. Instead, we notify you about the broken connector by email after three failed syncs in a row. After a connector is broken without a Task for 48 hours, Fivetran automatically escalates the issue to an engineering team.

For this stage, Fivetran generates the following log events:

11. Connector modifiedlink

You can change the connector credentials, incremental sync frequency, delay notification period and other connector-specific details of a connector. You can modify the connector in your Fivetran dashboard or by using the Modify a Connector endpoint.

For this stage, Fivetran generates the following log events:

NOTE: After you have modified and saved your connector, Fivetran automatically runs the setup tests.

12. Connector deletedlink

When you delete a connector, we delete all of its data from Fivetran’s database. You can delete connectors both in your Fivetran dashboard and by using the Delete a Connector endpoint.

For this stage, Fivetran generates the following log events: