Transformations: Fivetran Quickstart
Answered
Fivetran replication of salesforce replicates data into snowflake target.
When i make use of fivetran_quickstart. The ordinal values of columns are way of.
For instance for Case object. The ID is the column or ordinal 1 in intitial fivetran replication.
When we get to fivetran_quikcstart it goes to ordinal value of 468.
-
Official comment
Hi Ajay,
Thanks for raising this request. Can you explain a little bit more about why it's important that the columns in the Quickstart models are in the same order as the columns in the initial Fivetran replication? What probles is this causing? A detailed explanation will help, since we haven't heard about this issue from other customers before.
Best,
Sadie
-
There are a few items of consideration.
- Upon doing a select * the order is important as the ID and several of the other fields are in a logical order. For those querying regularly consistency is a big time saver.
- It's hard to find things when browsing large objects with hundreads/thousands of columns - as is common with this dataset as the tables are overloaded.
- For the larger tables in Salesforce, there are hundreads or thousands of columns. They are designed (other than new/changes) with the most relevent columns to the left.
- To avoid having to explicitly handle queries when doing dataframe joins or other work, ie Select ID, CAMPAIGN_ID, .... order is very helpful when working with languages that generalize the dataframe with NoSQL techniques-in our case Python.
- Loading and unloading of data is much easier when using inferred schemas - having to reorder or maintain a ensure_columns function adds effort and compute
- we're working with a VERY wide and have a LOT of junk at the end
Please sign in to leave a comment.
Comments
2 comments