Connector Improvement: Salesforce formula models extending across connectors
Answered
We are currently bringing a large amount of SFDC data into our warehouse, and the average runs take around 20 minutes to complete.
Occasionally, for some SFDC changes, some large objects require full synchronization. This can lead to production incidents, where we are unable to complete the data load for a day or two.
To address this issue, we are considering implementing multiple connectors with different sync frequencies, such as every 5 minutes, hourly, and daily. This approach would allow us to keep our critical path in near real-time sync and reduce the impact of incidents caused by a single stuck object.
We also plan to build a staging layer on top of this setup so that users perceive a single serving layer, regardless of the load frequency. Additionally, we hope to implement the formula package on top of this staging layer.
So the catch here is, when we have multiple connectors, the formula models populating in fivetran_formula_model have nulls for formula fields that depends on objects that are not selected in the connector.
We request,
1. Populate formula fields logic for all the objects irrespective of the object selection in the connector. (by bringing the metadata of all the objects)
2. Make database name and schema name as parameters in fivetran_formula_model which can be configured to use objects from different connectors.
-
Official comment
Hi Selva,
Thanks for writing in with this feedback and suggestion. We understand your concerns! You mentioned two issues:
1. Incremental syncs taking up to 20 minutes when you would like your critical path to be able to sync in under 5 minutes.
2. Occasional resyncs delaying the data for days.
Out of these issues, which one is more critical for you?
Best,
Sadie
Please sign in to leave a comment.
Comments
1 comment