Connector Improvement: Functionallity for firebase connector to do "updated/inserted" data capture
Answered
Currently firebase connector reads the whole document collection on every sync.
https://fivetran.com/docs/connectors/databases/firebase#replicationspeeds
as noted:
We pull all the data from source in each sync, so there might be some delay to fetch and process this data.
Could you add a functionality to utilize a column ie. inserted/updated at as a point of reference from where the sync should keep reading for the next batch, essentially offset tracking of sorts.
We currently have a set of collections that have upwards of 2-3 milions of rows with the potential for them to exponentially grow.
Fivetran currently puts a lot of load as well as incurs billing / usage on the firebase side because on every sync it has to read trough the whole collection.
Requesting that the functionality of offset tracking gets added in order to avoid overloading and incurring higher costs due to increased usage on googles Firebase.
-
Official comment
Hi Bojan,
Thank you for your suggestion regarding incremental syncing for the Firebase connector. We understand that the current approach of reading entire document collections during each sync can lead to increased load and costs, especially with large and growing datasets.
We’ve identified a solution for improving incremental data sync from Firebase, though it’s not in our immediate development pipeline. I’ll keep you updated and will reach out once it’s been implemented.
Best,
Rean
Please sign in to leave a comment.
Comments
1 comment