To correct data integrity issues, we will now trigger a re-sync for a table when we detect a reduction in the size of one of its SET columns. Before the re-sync begins, you may notice a
SET type change detected error in your logs.
When we detect a reduction in the size of a table’s ENUM columns, we now create an
ENUM type change detected error in your logs before we trigger a re-sync for that table. Previously, we added
FIVETRAN_ERROR_ENUM_DELETED values to your ENUM columns before we triggered a re-sync.
Data syncs no longer fail without explanation when we encounter prohibited DATETIME values (for example,
29 Feb 2021). How we respond to prohibited values varies by sync type:
Initial syncs: The sync fails when it encounters a prohibited value. We create a Task in your Fivetran dashboard with instructions on how to resolve the issue.
Incremental updates: The sync does not fail when it encounters a prohibited value. We process the prohibited value and create a Warning in your Fivetran dashboard with instructions on how to resolve the issue. The way we process the prohibited value depends on your sync mode:
- Live mode: We ignore the prohibited value.
- Legacy mode: We ignore the prohibited value.
- History mode: If the prohibited value is a primary key, we ignore it. If the prohibited value is not a primary key, we change it to NULL.
Learn more about Tasks and Warnings in our Alerts documentation.
We have fixed a bug where TIME values were sometimes synced to the destination incorrectly. Previously, we added a
.0 to values with no decimals and dropped the hundredths place for values that did. We now sync TIME values to the destination as they appear in the source:
|Source value||Destination value (with bug)||Destination value (fixed)|
If any values were incorrectly synced prior to this change, you will need to initiate a historical re-sync or table re-sync to correct them.
We have increased the fractional second precision of our binary log parser from millisecond to microsecond for DATETIME, TIMESTAMP, and TIME types. To correct existing values of these types in your destination, trigger a table re-sync or a historical re-sync in the connector dashboard.
We now add a fivetran_synced column while syncing empty tables and columns from the source.
We have added Magento MySQL and Magento MySQL RDS to the list of connectors that can be created via our REST API.
We have fixed a problem where unsigned BIGINT values larger than 9223372036854775807 were sometimes parsed incorrectly during incremental updates. If any values were incorrectly synced prior to this change, you will need to initiate a re-sync to correct them.
We now replicate empty tables in a MySQL source database as empty tables in the destination.
To correct data integrity issues, we will now trigger a re-sync for a table when we detect a reduction in the size of one of its ENUM columns. Before the re-sync begins, you may notice
FIVETRAN_ERROR_ENUM_DELETED values in these columns.
We are now rolling out support for BLOB, TINYBLOB, MEDIUMBLOB, LONGBLOB, and YEAR data types in version 4 of our MySQL connector.
We are now rolling out the ability to sync MySQL DATETIME columns to the destination as TIMESTAMP_NTZ columns for version 4 of our MySQL connector.
We have a fixed a bug that caused zeroed DATE and DATETIME values to sync incorrectly. MySQL allows
0000-00-00 00:00:00as values for DATE and DATETIME primary key columns respectively. Previously, we converted these values to
1970-01-01 00:00:00. Therefore, zeroed times and the real values of
1970-01-01 00:00:00were mapped to the same value in the destination. Fivetran now converts these values as follows:
- DATE: from
- DATETIME: from
- DATE: from
If you have a logging service connected to your Fivetran account, the import progress of your tables in the event logs will be visible in your logs as an
import_progressevent. Table names will be in either the
IN_PROGRESSstate, depending on their sync status. Tables that are not reported in the event have not started their import.
We have begun rolling out support for the following spatial types in version 4 of our MySQL connector:
We have changed the MySQL version requirements for Fivetran to store data as JSON in a MySQL warehouse and create tables with primary keys.
We now convert JSON to text in versions below 5.7.8.
We now create primary keys in either of the following scenarios:
MySQL version earlier than 5.7.7:
- If the total number of primary keys provided by the source is less than 16, and the sum of the byte length of String primary keys is less than 191.
MySQL version 5.7.7 and later:
- If the total number of primary keys provided by the source is less than 16, and the sum of the byte length of String primary keys is less than 768.
Fivetran has released version 4 of our MySQL connector, which offers improved sync reliability. We will add a number of upgrades to this version in the coming months:
We will sync DATETIME values as TIMEZONE_NTZ values in your destination.
We will support parsing for TRUNCATE DDL events.
We will support the following new data types:
This month, we have begun rolling out support for BINARY and VARBINARY types.
For more information about data types, see our MySQL documentation.
We have added a test to the MySQL setup forms which checks if you are using RDS. We added the test because if you select the wrong variant in the setup form, your connector may not work properly.
The test fails when there is a discrepancy between your selection and what Fivetran detected. If you are not confident in your selection, you may go back to the Add Connector view and select the connector type that matches your MySQL database variant.
Fivetran no longer supports invalid
Timestamp values. For example,
2019-00-01 00:00:00 is invalid. This change immediately affects all new MySQL connectors, and we will roll it out gradually to existing connectors.
The supported ranges of
Timestamp in MySQL are:
|Date||‘1000-01-01’ to ‘9999-12-31’|
|Datetime||‘1000-01-01 00:00:00’ to ‘9999-12-31 23:59:59’|
|Timestamp||‘1970-01-01 00:00:01’ UTC to ‘2038-01-19 03:14:07’ UTC|
We will load these invalid values as
NULL into the destination and display a warning on your Fivetran dashboard. The warning will indicate the schemas, tables, and columns which contain
Fivetran will create primary keys for a MySQL warehouse in either of the following scenarios:
MySQL version earlier than 5.7 or MariaDB version earlier than 10.2: If the source provides fewer than 16 primary keys and the sum of the byte length of string primary keys is less than 191.
MySQL version 5.7 and later or MariaDB version 10.2 and later: If the source provides fewer than 16 primary keys and the sum of the byte length of string primary keys is less than 768.
We have identified and fixed an extremely rare bug that can cause data to be missed during binary log replication. The bug meant that, in some cases, our system believed that all data had been successfully synced, when the binary log replication process had actually stopped prematurely.
For the following warehouses, Fivetran now drops the
primary_key field from existing tables in your warehouse if there is a
primary_key coming in from your data source:
- SQL Server
The MySQL connector is now able to handle unescaped control characters in textual data as if they had been properly escaped.
The MySQL warehouse now uses the load data infile strategy to write data. It requires the
load_infile system variable to be set to
on. For additional details, refer to our MySQL warehouse documentation.
String values will now be created as
LONGTEXT columns in the MySQL warehouse instead of
VARCHAR fields if they are not the primary key. We made this change to better handle MySQL’s maximum row size limit.
You can now customize the replica ID used to access your database’s binary logs. By default, the replica ID is a random integer greater than 1000.
We now fully support the
SET data type and consistently sync it as a JSON array. Previously, we synced
SET as a string during the initial import and as an integer when processed from the binary logs. If any values were incorrectly synced prior to this change, you will need to initiate a re-sync to correct them.
If your data warehouse uses a MySQL version earlier than 5.7, you can’t use the
timestamp column as a primary key because MySQL incorrectly handles the decimal part of the timestamp. We now detect this scenario, skip tables with this column during our sync, and warn you to upgrade your data warehouse MySQL version. Note that this limitation only applies to MySQL destinations, not MySQL sources.