AWS Lambda Overview – Fivetran Support Portal

AWS Lambda is a serverless computing platform that runs code in response to events and automatically manages the compute resources required by that code.

Featureslink

Feature Name	Supported	Notes
Capture Deletes
Custom Data		All tables and fields
Data Blocking
Column Hashing
Re-sync		Connector level
History
API Configurable
Priority-first sync
dbt Package

Supported languageslink

AWS Lambda supports the following languages:

Node.js (JavaScript)
Python
Java (Java 8 compatible)
C# (.NET Core)
Go

Function request and responselink

Fivetran uses the AWS SDK for the AWS Lambda Functions connector. For more information about how Fivetran syncs data from your cloud function, see our Sync overview documentation.

Request formatlink

Fivetran’s request has a standard format. It is a JSON object with three root nodes:

agent is an informational object.
state is a JSON object that contains cursors from the previous successful function execution. It is key to performing incremental updates. A cursor is a bookmark that marks the data Fivetran has already synced, for example, a timestamp, ID, or index. For the initial sync, state is an empty JSON object {}. Fivetran expects an updated state object in every response.

For more information about the state object, see How should I use the ‘state’ object in a cloud function.
secrets is a JSON object that contains access keys or API keys for the upstream APIs. Secrets allow you to store information (API tokens or database passwords) that you don’t want to maintain in your code. We use encryption at rest to store the secrets. We pass the secrets into your function every time we call the function. You must enter the secrets in the connector setup form. Enter an empty payload {} if there are no secrets to send.

For more information about the secrets object, see How should I use the ‘secrets’ object in a cloud function.

Example request

{
    "agent" : "Fivetran AWS Lambda Connector/<external_id>/<schema>",
    "state": {
        "cursor": "2020-01-01T00:00:00Z"
    },
    "secrets": {
        "apiToken": "abcdefghijklmnopqrstuvwxyz_0123456789"
    }
}

In this example,

external_id is the unique ID tied to your connector. You can find the ID in your connector setup form. You need this to configure your AWS account to connect with Fivetran.
schema is the destination schema name you enter when you first set up your connector.

Response formatlink

The response is a JSON object with five root nodes:

state contains the updated state value(s).
insert specifies the entities and records to be inserted. Fivetran reads the data and infers the data type and the number of columns.
delete (optional) specifies the entities and records to be deleted. Fivetran doesn’t delete the record; instead it marks the record as deleted by setting _fivetran_deleted column value to true. If you specify the delete node, you must also specify the schema node.
schema (optional) specifies primary key columns for each entity. You must be very consistent with the schema node and the primary key columns to avoid any unwanted behavior. If you don’t specify the schema, Fivetran appends the data.
hasMore is an indicator for Fivetran to make a follow-up call for fetching the next set of data. Fivetran keeps making repeated calls until it receives hasMore = false.

For more information about the hasMore node, see How should I use the ‘hasMore’ flag in a cloud function.

Example response

{
    "state": {
        "transaction": "2020-01-02T00:00:00Z",
    },
    "insert": {
        "transaction": [
            {"id":1, "amount": 100},
            {"id":3, "amount": 50}
        ],
    },
    "delete": {
        "transaction": [
            {"id":2},
        ],
    },
    "schema" : {
        "transaction": {
            "primary_key": ["id"]
        },
    },
    "hasMore" : false
}

In this example,

state contains the transaction cursor.
transaction is an entity. Fivetran creates the TRANSACTION table with id and amount columns.
The function inserts records 1 and 3 into the TRANSACTION table.
The function marks record 2 as deleted from the TRANSACTION table.
hasMore is set to false to indicate that there are no more records.

Setup guidelink

Follow our step-by-step AWS Lambda setup guide to connect AWS Lambda with your destination using Fivetran connectors.

Knowledge base articleslink

For more information about cloud functions, see the following knowledge base articles: