AWS Lambda is a serverless computing platform that runs code in response to events and automatically manages the compute resources required by that code.
Featureslink
Feature Name | Supported | Notes |
---|---|---|
Capture deletes | check | |
Custom data | check | All tables and fields |
Data blocking | check | Column level |
Column hashing | check | |
Re-sync | check | Connector level |
History | ||
API configurable | check | |
Priority-first sync | ||
Private networking | check | AWS PrivateLink |
Fivetran data models |
Supported languageslink
AWS Lambda supports the following languages:
- Node.js (JavaScript)
- Python
- Java (Java 8 compatible)
- C# (.NET Core)
- Go
Function request and responselink
Fivetran uses the AWS SDK for the AWS Lambda Functions connector. For more information about how Fivetran syncs data from your cloud function, see our Sync overview documentation.
Request formatlink
Fivetran’s request has a standard format. It is a JSON object with three root nodes:
-
agent
is an informational object. -
state
is a JSON object that contains cursors from the previous successful function execution. It is key to performing incremental updates. A cursor is a bookmark that marks the data Fivetran has already synced, for example, a timestamp, ID, or index. For the initial sync,state
is an empty JSON object {}. Fivetran expects an updatedstate
object in every response.For more information about the
state
object, see How to Use the state Object. -
secrets
(optional) is a JSON object that contains access keys or API keys for the upstream APIs. Secrets allow you to store information (API tokens or database passwords) that you don’t want to maintain in your code. We use encryption at rest to store the secrets. We pass the secrets into your function every time we call the function. Enter your secrets using a JSON format in the connector setup form.For more information about the
secrets
object, see How to Use the secrets Object.
Example request
{
"agent" : "Fivetran AWS Lambda Connector/<external_id>/<schema>",
"state": {
"cursor": "2020-01-01T00:00:00Z"
},
"secrets": {
"apiToken": "abcdefghijklmnopqrstuvwxyz_0123456789"
}
}
In this example,
external_id
is the unique ID tied to your connector. You can find the ID in your connector setup form. You need this to configure your AWS account to connect with Fivetran.schema
is the destination schema name you enter when you first set up your connector.
Response formatlink
The response is a JSON object with five root nodes:
-
state
contains the updated state value(s). -
insert
specifies the entities and records to be inserted. Fivetran reads the data and infers the data type and the number of columns. -
delete
(optional) specifies the entities and records to be deleted. Use this node to mark records as deleted. Fivetran doesn’t delete the record; instead it marks the record as deleted by setting_fivetran_deleted
column value totrue
. If you specify the delete node, you must also specify the schema node.Fivetran creates the
_fivetran_deleted
column in the destination table, only if your function response has thedelete
node. -
schema
(optional) specifies primary key columns for each entity. You must be very consistent with the schema node and the primary key columns to avoid any unwanted behavior. If you don’t specify the schema, Fivetran appends the data. -
hasMore
is an indicator for Fivetran to make a follow-up call for fetching the next set of data. Fivetran keeps making repeated calls until it receiveshasMore = false
.For more information about the
hasMore
node, see How to Use the hasMore Object.
Example response
{
"state": {
"transaction": "2020-01-02T00:00:00Z",
},
"insert": {
"transaction": [
{"id":1, "amount": 100},
{"id":3, "amount": 50}
],
},
"delete": {
"transaction": [
{"id":2},
],
},
"schema" : {
"transaction": {
"primary_key": ["id"]
},
},
"hasMore" : false
}
In this example,
-
state
contains thetransaction
cursor. -
transaction
is an entity. Fivetran creates theTRANSACTION
table withid
andamount
columns. -
The function inserts records
1
and3
into theTRANSACTION
table. -
The function marks record
2
as deleted from theTRANSACTION
table. -
hasMore
is set tofalse
to indicate that there are no more records.
Custom error handlinglink
Cloud functions may fail due to various reasons, including code execution errors, runtime issues, or internal errors. Add an error handling mechanism in your Lambda function response to report an error on your Fivetran dashboard.
Design your Lambda function to report an error:
-
Use the
errorMessage
node in your response to indicate function execution errors. Fivetran creates an Error on the connector dashboard with your custom error message. For example, for the following response, Fivetran creates aThis is an error
Error on your dashboard:{ "errorMessage": "This is an error" }
-
Use the
errorType
andstackTrace
nodes to pass additional information about the error. You must specify theerrorMessage
node to use theerrorType
andstackTrace
nodes. For example, for the following response, Fivetran creates an Error alert on your dashboard with the error type and stack trace details:{ "errorMessage": "name 'response' is not defined", "errorType": "NameError", "stackTrace": [ [ "/var/task/lambda_function.py", 35, "lambda_handler", "response['errorMessage'] = \"This is an error\"" ] ] }
Setup guidelink
Follow our step-by-step AWS Lambda setup guide to connect AWS Lambda with your destination using Fivetran connectors.
Frequently asked questionslink
For more information about cloud functions, see the following: