PostgreSQL Overview

PostgreSQL is an open source database typically used to keep in-house custom data. Fivetran’s integration platform replicates data from your PostgreSQL source database and loads it into your destination.

Supported serviceslink

Fivetran supports six different PostgreSQL database services:

• Generic PostgreSQL
• Amazon Aurora PostgreSQL

  Note: We do not support Amazon Aurora Serverless PostgreSQL.

• Amazon RDS PostgreSQL
• Azure PostgreSQL
• Google Cloud PostgreSQL
• Heroku Postgres

Supported configurationslink

Fivetran supports the following PostgreSQL configurations:

* Maximum throughput is your connector’s end-to-end update speed, measured in megabytes per second (MBps). We calculate the maximum throughput by averaging the number of rows synced per second during your connector’s last 3-4 syncs. To learn more about sync speed, see the Replication speeds section.

Which PostgreSQL instance types we can connect to depend on whether you use WAL (logical replication) with the pgoutput plugin, WAL with the test_decoding plugin, or XMIN as your incremental update mechanism. Read our Updating data documentation for more information.

Featureslink

*XMIN does not capture data that exists for an amount of time smaller than the sync interval.

Setup guidelink

In your master database, you need to do the following:

• Allow access to your PostgreSQL database via Fivetran’s IP
• Create a Fivetran-specific PostgreSQL user with read-level permissions
• (Optional) Allow access to a read-replica of your PostgreSQL database. Using a read-replica can help to avoid unnecessary strain on your master database.
• (WAL Only) Allow access to a logical replication slot

For specific instructions on how to set up your database, see the guide for your PostgreSQL database type:

• Generic PostgreSQL
• Amazon Aurora PostgreSQL
• Amazon RDS PostgreSQL
• Azure PostgreSQL
• Google Cloud PostgreSQL
• Heroku Postgres

Sync overviewlink

Once Fivetran is connected to your PostgreSQL database or read replica, we pull a full dump of all selected data from your database. Using either the WAL or XMIN change data capture process, we pull all your new and changed data at regular intervals.

If data in the source changes (for example, you add new tables or change a data type), Fivetran automatically detects and persists these changes into your destination. For every schema in your PostgreSQL source database, we create a schema in your destination that maps directly to its native schema. This ensures that the data in your destination is in a familiar format to work with.

Syncing empty tables and columnslink

Fivetran will sync empty tables and columns for your PostgreSQL connector. For more information, see our Features documentation.

Replication speedslink

Two major factors can cause disparities between our estimates and the exact replication speed for your Fivetran-connected databases: network latency and discrepancies in the format of the data we receive versus how the data is stored at rest in the data destination.

The ability to sync changes quickly also depends on the sync frequency you configure. The risk of the sync falling behind, or being unable to keep up with data changes, decreases as the sync frequency increases. We recommend a higher sync frequency for data sources with a high rate of data changes.

To measure the rate of new data in your database, check the disk space usage metrics over time for databases hosted on cloud providers. For self-hosted databases, you can run the following query to determine disk space usage:

SELECT SUM(bytes)/1024/1024 AS MB FROM dba_segments;

Schema informationlink

Fivetran tries to replicate the exact schema and tables from your PostgreSQL source database to your destination.

NOTE: We do not sync foreign tables.

Fivetran-generated columnslink

Fivetran adds the following columns to every table that is added to your destination:

• _fivetran_synced (UTC TIMESTAMP) indicates the time when Fivetran last successfully synced the row.
• _fivetran_id (STRING) is the hash of the non-Fivetran values of each row. It’s a unique ID that Fivetran uses to identify rows in tables that do not have a primary key.
• _fivetran_deleted (BOOLEAN) marks rows that were deleted in the source database.

  NOTE: If you use XMIN as your incremental update mechanism, your _fivetran_deleted column will have null values because XMIN does not track deleted rows.

We add these columns to give you insight into the state of your data and the progress of your data syncs.

Type transformation and mappinglink

As we extract your data, we match PostgreSQL data types to types that Fivetran supports. If we don’t support a certain data type, we automatically change that type to the closest supported type or, in some cases, don’t load that data at all. Our system automatically skips columns of data types that we don’t accept or transform.

The following table illustrates how we transform your PostgreSQL data types into Fivetran supported types:

If we are missing an important type that you need, please reach out to support.

In some cases, when loading data into your destination, we may need to convert Fivetran data types into data types that are supported by the destination. For more information, see the individual data destination pages.

PostGIS Geography data typeslink

The following table lists Fivetran-supported PostGIS Geography data types. They are transformed according to the GeoJson specification and stored in the destination as JSON types. The GeoJson specification does not support SRID, so Fivetran ignores this data type.

PostGIS Geometry data typeslink

The following table lists Fivetran supported PostGIS Geometry data types. They are transformed according to the GeoJson specification and stored in the destinations as JSON types.

Domain data typeslink

Fivetran supports domains, which are user-defined data types. When you create a domain, you choose an underlying data type and have the option of specifying additional constraints. Learn more in PostgreSQL’s domains documentation.

We transform and store domain data based on their underlying data types.

Excluding source datalink

If you don’t want to sync all the data from your source database, you can exclude schemas, tables, or columns from your syncs on your Fivetran dashboard. To do so, go to your connector details page and un-check the objects you would like to omit from syncing. For more information, see our Column Blocking documentation.

Alternatively, you can restrict the Fivetran user’s access to certain tables or columns in your source database.

GRANT USAGE ON SCHEMA "some_schema" TO fivetran;

ALTER DEFAULT PRIVILEGES IN SCHEMA "some_schema" REVOKE SELECT ON TABLES FROM fivetran;
REVOKE SELECT ON ALL TABLES IN SCHEMA "some_schema" FROM fivetran;

GRANT SELECT ON "some_schema"."some_table" TO fivetran;
GRANT SELECT ON "other_schema".* TO fivetran; /* all tables in schema */

ALTER DEFAULT PRIVILEGES IN SCHEMA "some_schema" GRANT SELECT ON TABLES TO fivetran;

REVOKE SELECT ON "some_schema"."some_table" FROM fivetran;

GRANT SELECT (xmin, "some_column", "other_column") ON "some_schema"."some_table" TO fivetran;

#!/bin/sh

#Fill in the values without quotes or a space after the equals

host=         # ex: 10.10.135.135
port=         # ex: 30054
user=         # ex: user
password=     # ex: asdf235235asfsdf212

#List all of your SQL commands wrapped in single quotes, for example

sql=(
'REVOKE ALL PRIVILEGES, GRANT OPTION FROM fivetran;'
'GRANT REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO fivetran;'
'GRANT SELECT ON schema1.* TO fivetran;'
'GRANT SELECT ON schema1.* TO fivetran;'
)

#Connecting to Redshift, hence the PGPASSWORD
do
  PGPASSWORD=$password psql --host=$host --port=$port --user=$user $db -c "${sql[$i]}"
done

Initial synclink

When Fivetran connects to a new database, we first copy all rows from every table in every schema for which we have SELECT permissions (except for those that you excluded on your Fivetran dashboard) and add Fivetran-generated columns. We copy rows by performing a SELECT statement on each table. For large tables, we copy a limited number of rows at a time so that we don’t have to start the sync over from the beginning if our connection is lost midway.

Updating datalink

Once the initial sync is complete, Fivetran performs incremental updates of any new or modified data from your source database. We use one of the following mechanisms to perform incremental updates: logical replication with the test_decoding plugin, logical replication with the pgoutput plugin, or the XMIN system column.

NOTE: Logical replication with the pgoutput plugin is in the Beta phase. Learn more in our Release Phases documentation.

Logical replicationlink

Logical replication is based on logical decoding of the PostgreSQL write-ahead log (WAL). We recommend this method for incremental updates because it

• Minimizes processing overhead on your PostgreSQL server
• Replicates row deletions for tables with primary keys

Fivetran supports two plugins for logical replication - test_decoding and pgoutput. Choose one plugin to connect to Fivetran. Your plugin must have its own replication slot.

NOTE: If you use both plugins in your database, you can only connect one of them to Fivetran. Your disk usage may increase in this situation, since your changes register in both plugins’ replication slots.

However, there are reasons why you might not want to or be able to use logical replication:

• Logical replication requires version 9.4.15 or later (prior minor versions have bugs)

• For PostgreSQL versions 10.2+, creating a column with the alter table add column set default command does not trigger an update to the existing table that the logical replication mechanism can recognize. Consequently, those changes are not part of our incremental updates. You need to do a full table re-sync to update the existing table’s data in the destination. Learn more in PostgreSQL’s Modifying Tables documentation.

• In Generic PostgreSQL

  • Only supported on primary instances, so you cannot enable it on a read replica
  • May need significant extra storage space (even if log storage is only temporary)
  • Requires a server reboot to activate logging

• In Amazon RDS PostgreSQL

  • Only supported on RDS primary instances, so you cannot enable it on a read replica
  • May need significant extra storage space (even if log storage is only temporary)
  • Requires a primary instance reboot to activate logging

• In Azure PostgreSQL

  • Only supported on Azure primary instances, so you cannot enable it on a read replica
  • Only supported for Azure PostgreSQL version 10 or later
  • May need significant extra storage space (even if log storage is only temporary)
  • Requires a server reboot to activate logging

• In Heroku Postgres

  • Does not support logical replication

• In Amazon Aurora PostgreSQL

  • Only supported on Aurora primary instances, so you cannot enable it on a read replica
  • Only supported for Aurora PostgreSQL version 10.6 or later

• Google Cloud PostgreSQL

  • Only supported on Google Cloud primary instances, so you cannot enable it on a read replica

Note: Fivetran does not support logical replication if you use the “swap and drop” method of replicating data. In the “swap and drop” method, you create a temporary table and load in data from your current table. You then drop the current table and rename the temporary table so it has the same name as your current table.

If logical replication is not an option for you, Fivetran can fall back on using the XMIN method.

XMIN system columnlink

The hidden PostgreSQL system column xmin can be used to select only the new or changed rows since the last update. The XMIN method has the following disadvantages:

• Cannot replicate row deletions, so there is no way to tell which rows in the destination are no longer present in the source database. XMIN can’t track row deletions because it relies on a hidden system column in PostgreSQL tables that is effectively a last_modified column. When a row is deleted, it doesn’t appear as being recently modified because it no longer exists.
• Requires a full table scan to detect updated rows, which can slow down updates and cause significant processing overhead on your PostgreSQL server

Therefore, we only use the XMIN method if logical replication is not an option for you.

Tables with a primary keylink

We merge changes to tables with primary keys into the corresponding tables in your destination:

• An INSERT in the source table generates a new row in the destination.
• An UPDATE in the source table updates the data in the corresponding row in the destination.
• (both WAL with the test_decoding plugin and WAL with the pgoutput plugin) A DELETE in the source table updates the corresponding row in the destination with _fivetran_deleted = TRUE.

Tables without a primary keylink

When we import tables without primary keys, we page over values of unique key columns, as well as any columns that might belong to unique indexes. Therefore, Fivetran can only import a table without a primary key when at least one column of that table has a unique key constraint and/or is part of a unique index.

We handle changes to tables without a primary key in the following ways:

• An INSERT in the source generates a new row in the destination.
• An UPDATE in the source generates a new row in the destination and leaves the old version of that row in the destination untouched. As a result, one record in your source database may have several corresponding rows in your destination.

Fivetran cannot recognize deleted rows in tables without primary keys. For more information, see the Deleted Rows section.

For logical replication with the pgoutput plugin, only INSERT operations would be passed through unless the table was given Replica Identity.

Identify tables with primary keyslink

To find out which of your tables have (or don’t have) primary keys, run this query:

SELECT
  table_schema,
  table_name,
  (table_schema, table_name) IN (
     SELECT
        _schema.nspname AS table_schema,
        _table.relname  AS table_name
     FROM pg_catalog.pg_constraint c
        LEFT JOIN pg_catalog.pg_class _table ON c.conrelid = _table.oid
        LEFT JOIN pg_catalog.pg_namespace _schema ON _table.relnamespace = _schema.oid
     WHERE c.contype = 'p'
  ) AS has_primary_key
FROM information_schema.tables
WHERE table_type = 'BASE TABLE' AND table_schema NOT IN ('information_schema', 'looker_scratch') AND
     NOT table_schema ~ '^pg_';

The third column in the query results, has_primary_key, is a binary value that shows whether or not each table has a primary key.

Deleted Rowslink

Logical replication does not allow us to recognize deleted rows in tables without primary keys. For more information, see our logical replication documentation.

The XMIN update mechanism does not allow us to recognize deleted rows at all. (For more information, see our XMIN documentation.) Rows that are deleted in the source database are still present in the destination, because there is no way for us to identify them. If you want to recognize deleted rows, use one of these strategies to capture deletions in the source database:

1. Add a trigger to each table to log the primary keys of deleted rows in a separate table. Once this table has been replicated to the destination, you can use it to generate a view that excludes the deleted rows.

2. Instead of removing rows, add an ‘is_deleted’ column to each table and change the business logic to set this column to TRUE when you delete a row. To avoid build up of obsolete rows in the source database, you can delete them after a certain period. Make sure you leave enough time for Fivetran replicate the deleted rows to the destination before you remove them (we recommend seven days).

We do not support deleting data with the TRUNCATE command in incremental updates. However, you can do a full re-sync to capture TRUNCATE deletions.

Incremental sync durationlink

When Fivetran receives no new or changed data from your database, we continue reading the WAL stream for up to 25 minutes before ending the sync. We take this time to search for changes so that we don’t miss any change data before ending the sync.

Historical re-sync scenarioslink

• If you want to migrate to another service provider, we will need to do a full re-sync of your data because the new service provider will not retain the same change tracking data as your original PostgreSQL database.
• If you have two replication slots (one with the test_decoding plugin and one with the pgoutput plugin), your transactions are registered by both replication slots. Consequently, switching your Fivetran connector from one replication slot to the other would sync duplicate transactions. The data in your destination would remain the same, except for updated data in the _fivetran_synced column. To avoid increasing your MAR, you can drop the original replication slot that you connected to Fivetran and re-sync your connector.