Connector Improvement: Salesforce Formula + History Mode`
Currently, the `fivetran_formula_model` adjusts when you turn on history mode to filter to `fivetran_active` records automatically. This is great for avoiding fanout issues that would be caused by not filtering. BUT it has the side effect of making the "history" part of history mode not work for formula fields.
There are 2 possible ways I could see results being calculated:
1. Formulas from today, applied to historical data
In my experience this is actually the most common use case we come across, because it lets us look at formula values that change due to the underlying data rather than because the formula itself was changed.
E.g.
Jan 1: Formula for X is set to A+B
Jan 2: Value for A changes to from A1 to A2
Jan 3: Formula for X changes to A+B+C
What we're looking for in this use case is to see:
- A Jan 2 history row with X being reported as A1 + B + C
- A Jan 3 history row with X being reported as A2 + B + C
(note: this can be complicated if C is a new field, where the Jan 2 record would have a null value for C, but as a rule it probably makes sense to coalesce to 0)
2. Formulas history being applied to the relevant records
In the example above, this would produce
- A Jan 2 history row with X being A1 + B
- A Jan 3 row with X being A2 + B + C
This has different challenges of course, because it fundamentally changes the grain of the current `fivetran_formula_model`
Note: If formulas are changed in a way that's backwards compatible then both approaches are equally useful, but if it's some sort of breaking change my experience tells me that 1 is more challenging but also likely more aligned with what people expect to see.
A note on how we built something custom for our business needs
For ease of joining multiple salesforce history tables together, we were already working to convert the Type II SCD into a "daily_history" where each record gets one row each day.
We built a modified version of fivetran's `sfdc_formula_view` macro that points to these `daily_history` models and applies the formulas to all historical records, not just the active ones.
This approach works for us. The main downside is that it took a fair amount of expertise that I imagine is not super common, and also that our modeling is not officially supported by Fivetran.
-
This would also be of interest to us. I agree with you in that we'll want to apply current formula field definitions 90+% of the time.
That said there are two cases where it would be useful to apply formula fields as of a given record date.
1. Debugging business processes. We, like many, have a lot tied to SF, and use formula fields to drive work queue prioritization, calculate important dollar amounts, etc. We also integrate with other internal systems. It'd be awesome to use fivetran as research tool to be able to tell if an errant value came from a process error, a formula error, or an integration error.
2. Auditors.
Please sign in to leave a comment.
Comments
1 comment