Fivetran generates structured log events from connectors, dashboard user actions, and Fivetran API calls. The logs provide an internal view of the operations happening with each of your connectors. You can use these logs in the following ways:
- in your Fivetran dashboard
- in your destination using the Fivetran Log Connector
- in an external logging service
In your Fivetran dashboardlink
You can view both user actions and logs in your Fivetran dashboard.
- View user actions on the Status tab of the Connector Details page. The User Actions list contains a complete user action audit.
- View logs on the Logs tab of the Connector Details page. You can search up to one week of log records.
Learn more about the Fivetran dashboard’s user actions and logs in our Sync troubleshooting documentation.
In your destinationlink
The Fivetran Log Connector is a free connector that delivers your logs and account metadata to a schema in your destination. This metadata includes granular Fivetran consumption information. The Fivetran Log Connector is available on all plans. Learn more in our Fivetran Log Connector documentation.
In an external logging servicelink
Important: You must be on an Enterprise plan to use an external logging service.
Fivetran can send logs to the following external services:
You can connect one logging service per destination. Fivetran will write log events for all connectors in the destination group to the connected service. If there is a logging service that you would like but that is not yet supported, please let us know.
Log eventslink
The log events are in a standardized JSON format:
{
"event": <Event name>,
"created": <Event creation timestamp in UTC>,
"connector_type": <Connector type>,
"connector_id": <Connector ID>,
"connector_name": <Connector name>,
"data": {
// Event specific data. This section varies for each event
},
"exception_id": <Fivetran error identifier>
}
The field exception_id will only be present if Fivetran encountered an unexpected problem.
Eventslink
Fivetran connectors generate the following events:
| Connector Event Name | Description |
|---|---|
| api_call | API call made to a source service |
| sql_query | SQL query executed on a source database |
| create_schema | Schema created in destination |
| create_table | Table created in destination |
| drop_table | Table dropped from destination |
| alter_table | Table columns added to, modified in or dropped from destination table |
| connection_successful | Successfully established connection with source system |
| connection_failure | Failed to establish connection with source system |
| sync_start | Connector started syncing data |
| sync_end | Data syncing completed |
| import_progress | Rough estimate of import progress |
| processed_records | Number of records read from source system |
| write_to_table_start | Started writing records to destination table |
| write_to_table_end | Finished writing records to destination table |
| schema_migration_start | Schema migration started |
| schema_migration_success | Schema migration succeeded |
| schema_migration_failure | Schema migration failed |
| schema_migration_end | Schema migration ended |
| copy_rows | Data copied to staging table |
| delete_rows | Stale rows deleted from main table |
| insert_rows | Updated rows inserted in main table |
| update_rows | Existing rows in main table updated with new values |
| create_staging_table | Temporary staging table created in destination |
| records_modified | Number of records modified during sync |
| status | Sync update status (for example, success, fail, rescheduled, etc.) |
| sync_stats | Current sync metadata |
| json_value_too_long | A JSON value was too long for your destination and had to be truncated |
| info | Info during data syncing |
| warning | Warning during data syncing |
| error | Error during data syncing |
| forced_resync_table | Forced re-sync of a table |
| forced_resync_connector | Forced re-sync of a connector |
| change_schema_config_via_sync | Schema configuration updated during a sync |
| diagnostic_access_expired | Data access expired |
| diagnostic_access_used | Data accessed by Fivetran support for diagnostic purposes |
Dashboard activities generate the following events:
| Dashboard Activity Event Name | Description |
|---|---|
| change_schema_config | Schema configuration updated |
| migrate_schema_config | Schema configuration migration created |
| connect_bi_tool | BI tool is connected |
| update_bi_tool | BI tool configuration is updated |
| disconnect_bi_tool | BI tool is disconnected |
| create_connector | New connector is created |
| edit_connector | Connector’s credential, sync period or delay notification period is edited |
| delete_connector | Connector is deleted |
| pause_connector | Connector is paused |
| resume_connector | Connector is resumed |
| resync_connector | Connector’s re-sync is triggered |
| resync_table | Connector’s table re-sync is triggered |
| force_update_connector | Trigger manual update for connector |
| dismiss_warning | Warning(s) is dismissed |
| update_logger | Logging service credential is updated |
| pause_logger | Logging service is paused |
| resume_logger | Logging service is resumed |
| disconnect_logger | Logging service is disconnected |
| signin_warehouse | Sign in to destination |
| signout_warehouse | Sign out from destination |
| create_warehouse | Destination is created |
| update_warehouse | Destination configuration is updated |
| delete_warehouse | Destination is deleted |
| test_connection | Connector test(s) run |
| upload_file | File is uploaded |
| create_file_upload_table | Table created to load records from uploaded file |
| remove_file_upload_table | Table dropped for previously uploaded file |
| add_user | User is added to group |
| remove_user | User is removed from group |
| update_user | User details are updated |
| create_custom_role | Role is created |
| update_custom_role | Role details are updated |
| delete_custom_role | Role is removed from account |
| create_transformation | New transformation is added |
| update_transformation | Transformation is updated |
| delete_transformation | Transformation is deleted |
| diagnostic_access_approved | Data access granted for 21 days |
| diagnostic_access_denied | Data access denied |
| diagnostic_access_revoked | Data access revoked |
| generate_api_secret | API secret is generated on Account Settings page |
REST API calls generate the following events:
| API Call Event Name | Description |
|---|---|
| change_schema_config_via_api | Schema configuration updated using API call |
Basic SQL transformations generate the following events:
| Basic Transformation Event Name | Description |
|---|---|
| transformation_run_start | Transformation started |
| transformation_run_succeeded | Transformation successfully finished |
| transformation_run_failed | Transformation failed |
dbt transformations generate the following events:
| dbt Transformation Event Name | Description |
|---|---|
| dbt_run_start | The dbt transformation started. |
| dbt_run_succeeded | The dbt transformation was successfully finished. |
| dbt_run_failed | The dbt transformation failed. |
Connector stages and related log eventslink
This section describes the life cycle of a connector. It lists the corresponding log event generated at each stage of the connector, or connector-related dashboard activities. This will help you recognize the events in the logs and understand how their ordering relates to the operations of the connector.
All listed connector-related log events for a particular connector are displayed on the connector’s Logs tab in your Fivetran dashboard. The connector-related log events are included in the logs captured by the Fivetran Log Connector and external logging services.
NOTE: The connector life cycle stages as are listed in chronological order where possible.
1. Connector creation initializedlink
When you create a connector, Fivetran writes its ID in Fivetran’s database and assigns the connector status “New.”
NOTE: Fivetran writes encrypted credentials in its database.
If you create a connector in your Fivetran dashboard, you need to specify the required properties and authorization credentials.
If you create a connector using the Create a Connector API endpoint, you need to specify the required properties. However, you can omit the authorization credentials and authorize the connector later using the Connect Card.
For this stage, Fivetran generates the following log events:
| Event name | Description |
|---|---|
| create_connector | New connector is created |
2. Connector setup tests runlink
During these tests, Fivetran verifies that credentials such as authentication details, paths, and IDs are correct and valid, and resources are available and accessible.
When you click Save & Test while creating a connector in your Fivetran dashboard, Fivetran runs the setup tests.
Also, you can run the setup tests by using the Run Connector Setup Tests endpoint. If the setup tests have succeeded, it means the connector has been successfully created.
For this stage, Fivetran generates the following log events:
| Event name | Description |
|---|---|
| test_connection | Connector test(s) run |
| api_call | API call made to a source service |
| sql_query | SQL query executed on a source database |
3. Connector successfully createdlink
After the setup tests have succeeded, Fivetran records the Connection Created user action in the User Actions Log. At this stage, the connector is paused. It does not extract, process, or load any data from the source to the destination.
After the connector has been successfully created, you can trigger the historical sync.
For this stage, Fivetran generates the following log events:
| Event name | Description |
|---|---|
| connection_successful | Successfully established connection with source system |
| connection_failure | Failed to establish connection with source system |
4. Connector schema changedlink
Connector schema changes include changing the connector schema, tables and table columns.
You change your connector’s schema in the following cases:
- You want to switch the sync mode
- You want to change what data your connector syncs, which includes using the column blocking and column hashing features.
- You need to fix a broken connector.
- You need to change a connector schema as part of the schema review. The schema review after you have created a connector and before you run the historical sync is required for particular connector types.
You can change your schema in the following ways:
- In your Fivetran dashboard
- With the Fivetran REST API using any of the following endpoints:
For this stage, Fivetran generates the following log events:
| Event name | Description |
|---|---|
| change_schema_config | Schema configuration updated |
| migrate_schema_config | Schema configuration migration created |
NOTE: For an un-paused connector, changing the schema will trigger a sync to run. If a sync is already running, Fivetran will cancel the running sync and immediately initiate a new one with the new schema configuration.
5. Sync triggeredlink
You need to run the historical sync for the connector to start working as intended. The first historical sync that Fivetran does for a connector is called the initial sync. During the historical sync, we extract and process all the historical data from the selected tables in the source. Periodically we will load data into the destination.
After a successful historical sync, the connector runs in the incremental sync mode. In this mode, whenever possible, only data that has been modified or added - incremental changes - is extracted, processed, and loaded on schedule. We will reimport tables where it is not possible to only fetch incremental changes. We use cursors to record the history of the syncs.
The connector sync frequency that you set in your Fivetran dashboard or by using the Modify a Connector endpoint defines how often the incremental sync is run.
IMPORTANT: Incremental sync runs on schedule at a set sync frequency only when the connector’s sync scheduling type is set to
autoin our REST API. Setting the scheduling type tomanualeffectively disables the schedule. You can trigger a manual sync to sync the connector in this case.
NOTE: Some connector types support the priority-first sync mode.
For this stage, Fivetran generates the following log events:
| Event name | Description | Step |
|---|---|---|
| sync_start | Connector started syncing data | Extract |
| api_call | API call made to a source service | Extract |
| sql_query | SQL query executed on a source database | Extract |
| schema_migration_start | Schema migration started | Load |
| schema_migration_success | Schema migration succeeded | Load |
| schema_migration_failure | Schema migration failed | Load |
| schema_migration_end | Schema migration ended | Load |
| write_to_table_start | Started writing records to destination table | Load |
| copy_rows | Data copied to staging table | Load |
| delete_rows | Stale rows deleted from main table | Load |
| insert_rows | Updated rows inserted in main table | Load |
| update_rows | Existing rows in main table updated with new values | Load |
| create_staging_table | Temporary staging table created in destination | Load |
| create_schema | Schema created in destination | Load |
| create_table | Table created in destination | Load |
| json_value_too_long | A JSON value was too long for your destination and had to be truncated | Process |
| drop_table | Table dropped from destination | Load |
| alter_table | Table columns added to, modified in or dropped from destination table | Load |
| change_schema_config_via_sync | Schema configuration updated during a sync. Updates are done when a new table was created during the sync and the user selected to automatically include new tables in the schema. | Process |
| write_to_table_end | Finished writing records to destination table | Load |
| records_modified | Number of records modified during sync | Load |
| sync_end | Data syncing completed | Load |
| status | Sync update status e.g. SUCCESS, FAILURE, RESCHEDULED, etc. |
6. Connector paused/resumedlink
When you have just created a connector, it is paused, which means it does not extract, process, or load data. After you successfully run the setup tests, the connector becomes enabled/resumed. After the successful initial sync, it starts working in incremental sync mode.
You can pause and resume the connector either in your Fivetran dashboard, or by using various Connector endpoints in our API.
For this stage, Fivetran generates the following log events:
| Event name | Description |
|---|---|
| pause_connector | The connector was paused |
| resume_connector | The connector was resumed |
NOTE: Resuming a connector will trigger either the initial sync, or an incremental sync, depending on the stage the connector was at when it was paused.
7. Re-sync triggeredlink
In some cases you may need to re-run a historical sync to fix a data integrity error. This is called a re-sync. We sync all historical data in the tables and their columns in the source as selected in the connector configuration.
You can trigger a re-sync from your Fivetran dashboard. You can also trigger a re-sync by using the Modify a Connector endpoint.
For connectors that support table re-sync, you can trigger it either in the dashboard or by using the Re-sync connector table data endpoint.
For this stage, Fivetran generates the following log events:
| Event name | Description |
|---|---|
| resync_connector | Connector’s re-sync is triggered |
| resync_table | Connector’s table re-sync is triggered |
8. Manual sync triggeredlink
You can trigger an incremental sync manually without waiting for the scheduled incremental sync.
You can do this either by clicking Sync Now in your Fivetran dashboard or by using the Sync Connector Data endpoint.
For this stage, Fivetran generates the following log events:
| Event name | Description |
|---|---|
| force_update_connector | Trigger manual update for connector |
9. Sync endedlink
A sync can end in four states:
- Succeeded - the sync was completed without issue and data in the destination is up-to-date.
- Failure - the sync failed due to a unknown issue.
- Failure with Task - the sync failed due to a known issue that requires the user to take actions to be fixed. A Task is generated and displayed on the Alerts page in the dashboard.
- Rescheduled - the sync was unable to complete at this time and will automatically resume syncing when it can complete. This is most commonly caused by hitting API quotas.
When the sync ends, Fivetran generates a status log event:
| Data fields | Explanation |
|---|---|
status |
Final status of sync. Valid values: - SUCCESS - sync succeeded - FAILURE - sync failed - FAILURE_WITH_TASK - sync failed with Task - RESCHEDULED - sync was rescheduled |
reason (optional) |
If status is FAILURE, this is the description of the reason why the sync failed. If status is FAILURE_WITH_TASK, this is the description of the Task. If status is RESCHEDULED, this is the description of the reason why the sync is rescheduled. |
task_type (optional) |
If status is FAILURE_WITH_TASK, this field displays the type of the Task, e.g., reconnect. |
rescheduledAt(optional) |
If status is RESCHEDULED, this field time is scheduled to resume. The scheduled time depends on the reason it was rescheduled for. |
10. Connector brokenlink
IMPORTANT: This is an abnormal state for a connector. It commonly happens due to transient networking or server errors and most often resolves itself with no action on your part.
A connector is considered to be broken when during its sync it fails to either extract, process, or load data.
You can see a connector in the Connector List is broken when it has the red Broken label.
In the Connector dashboard, a broken connector also has the red Broken Connection label. Its log in the Logs tab will display the status log event with the red pictogram. Click on the arrow button to expand the log event and see its details.
If we know the breaking issue, we create a corresponding Task and notify you by email with instructions on how to resolve the issue. The Task is displayed on the Alerts page in your Fivetran dashboard. You need to take the actions listed in the Task message to fix the connector. We resend the Task email every seven hours until the Task is completed.
If your connector breaks and the breaking issue is unknown, we do not generate a Task in your dashboard. Instead, we notify you about the broken connector by email after three failed syncs in a row. After a connector is broken without a Task for 48 hours, Fivetran automatically escalates the issue to an engineering team.
For this stage, Fivetran generates the following log events:
| Event name | Description |
|---|---|
| status | FAILURE or FAILURE_WITH_TASK |
11. Connector modifiedlink
You can change the connector credentials, incremental sync frequency, delay notification period and other connector-specific details of a connector. You can modify the connector in your Fivetran dashboard or by using the Modify a Connector endpoint.
For this stage, Fivetran generates the following log events:
| Event name | Description |
|---|---|
| edit_connector | Connector’s credential, sync period or delay notification period is edited |
NOTE: After you have modified and saved your connector, Fivetran automatically runs the setup tests.
12. Connector deletedlink
When you delete a connector, we delete all of its data from Fivetran’s database. You can delete connectors both in your Fivetran dashboard and by using the Delete a Connector endpoint.
For this stage, Fivetran generates the following log events:
| Event name | Description |
|---|---|
| delete_connector | Connector is deleted. Events of this type can be only seen in the logs generated by the Fivetran Log Connector and external logging services because the corresponding connector’s Dashboard becomes unavailable after the connector has been deleted. |