DynamoDB Overview – Fivetran Support Portal

Amazon DynamoDB is a fully-managed, proprietary NoSQL database service that is offered as part of Amazon Web Services (AWS).

Featureslink

Feature Name	Supported	Notes
Capture Deletes		All tables and fields
Custom Data		All tables and fields
Data Blocking		Table level and schema level
Column Hashing
Re-sync		Table level
History
API Configurable
Priority-first sync
dbt Package

Setup guidelink

Follow our step-by-step DynamoDB setup guide to connect DynamoDB with your destination using Fivetran connectors.

Sync Overviewlink

Once Fivetran is connected to your database, we pull a full dump of all selected data from your database. We then use DynamoDB Streams to pull all your new and changed data at regular intervals. If data in your master database changes (for example, you add new tables or change a data type), Fivetran automatically detects and persists these changes into your destination.

Sync mode optionslink

Sync modes determine the form in which Fivetran delivers your data. You must choose a sync mode for each table you want Fivetran to sync. There are two sync modes - packed and unpacked. If your table has or will have more than 1000 unique first-level keys, you must use packed mode.

Unpacked mode

If your table has fewer than 1000 first-level attributes, you have the option to have your data delivered unpacked. Fivetran unpacks one layer of nested fields and infer types.

In unpacked mode, the following source table

{
  "foo": 1, <== partion key and/or sort key
  "bar": 2,
  "nested": {
    "baz": 3
  }
}

is delivered to your destination as

foo INTEGER	bar INTEGER	nested JSON
1	2	`{"baz":3}`

Packed mode

All tables with more than 1000 first-level keys must have their data delivered packed, though you can select packed mode for smaller tables too.

In packed mode, the following source table

{
  "foo": 1, <== partion key and/or sort key
  "bar": 2,
  "nested": {
    "baz": 3
  }
}

is delivered to your destination as

foo INTEGER	data JSON
1	`{"foo":1, "bar":2, nested":{"baz":3}}`

Switching sync modes

You can switch sync modes for a table at any time in your Fivetran dashboard. However, you must do a full re-sync for that table when you change sync modes. For example, you may have a table that originally had 700 first-level keys but now has 1100 first-level keys. You’ll need to change the sync mode for that table to packed and do a full-table re-sync.

Schema informationlink

Fivetran tries to replicate the exact schema and tables from your DynamoDB source database to your destination.

Fivetran-generated columnslink

Fivetran adds the following columns to every table in your destination:

_fivetran_deleted (boolean) marks rows that were deleted in the source database.
_fivetran_synced (UTC timestamp) keeps track of when each row was last successfully synced.

We add these columns to give you insight into the state of your data and the progress of your data syncs.

Type transformations and mappinglink

As we extract your data, we match DynamoDB data types to types that Fivetran supports. If we don’t support a data type, we automatically change that type to the closest supported type or, in some cases, don’t load that data at all. Our system automatically skips columns with data types that we don’t accept or transform.

The following table illustrates how we transform your DynamoDB data types into Fivetran supported types:

DynamoDB Type	Fivetran Type	Fivetran Supported
STRING	STRING	True
BINARY	BINARY	True
NUMBER	DECIMAL	True
STRINGSET	JSON	True
NUMBERSET	JSON	True
BINARYSET	JSON	True
MAP	JSON	True
LIST	JSON	True
BOOLEAN	BOOLEAN	True

Excluding source datalink

If you don’t want to sync all the data from your master database, you can exclude schemas or tables from your syncs on your Fivetran dashboard. To do so, go to your connector details page and un-check the objects you would like to omit from syncing. For more information, see our Column Blocking documentation.

Alternatively, you can change the permissions of the Fivetran-specific IAM role you created and restrict its access to certain tables.

Initial synclink

When Fivetran connects to a new DynamoDB database, we scan through each of your selected tables one at a time to fetch your data. If we encounter a ProvisionedThroughputExceededException error message, we retry the sync with an exponential backoff strategy. We recommend that you have a high provisioned throughput read capacity for your tables so that Fivetran doesn’t encounter this error.

Updating datalink

Once the initial sync is complete, Fivetran performs incremental updates of any new or modified data from your source database. We use the DynamoDB Streams Kinesis Adapter to process the change data in your DynamoDB Streams and fetch only the data that has changed since our last sync.

We merge changes to your tables into the corresponding tables in your destination:

Every inserted row in the source generates a new row in the destination with _fivetran_deleted = FALSE.
Every updated row in the source updates the data in the corresponding row in the destination, with _fivetran_deleted = FALSE.
For every deleted row, the _fivetran_deleted column value is set to TRUE for the corresponding row in the destination.

DynamoDB Streams data retention limitlink

DynamoDB Streams have a retention period of 24 hours. Change records exist in the stream for one day and are then deleted. If your syncs fail for more than 24 hours, we issue a warning that you must do a full re-sync to make sure we capture the data changes we missed.

Deleted rowslink

We do not delete rows from your destination. When a row is deleted from the source table, we set the _fivetran_deleted column value of the corresponding row in the destination to TRUE.

Migrating service providerslink

If you want to migrate service providers, we will need to do a full re-sync of your data because the new service provider won’t retain the same change tracking data as your original DynamoDB database.