Other: HVR: Snowflake replication enhancement request
At present when a batch refresh job is triggered Integrate is suspended which is aborting existing sessions, instead of aborting running sessions if the refresh job can wait till running transactions are completed it would avoid data issues, reduce latency, also no need to rerun transactions again.
Here is the issue and use case for Snowflake:
- Batch refresh is triggered and Integrate is suspended aborting integrate sessions on Snowflake target
- these sessions have been running for long time, around ~30mins and consumed significant SF resources incurring WH costs
- Latency is already 30mins+ and data availability is delayed for business
- Batch refresh job takes 10 - 15 mins to complete
- Integrate wakes up after batch refresh job is complete
- Has to rerun long merge statement again, taking 30-45 minutes incurring SF costs and also latency is over 90+minutes now!
---------
This enhancement is going to significantly improve and benefit data availability on SF side and reduce SF cost.
Happy to get into a call to explain our architecture (Oracle => SF replication) which includes CDC plus batch refresh jobs that syncs exchange partition data on daily basis.
-
Hi Jithendra,
We are working on a feature to no longer block integration for an ongoing refresh in the same channel (for the same integration job). That will take care of your concern. The feature will be released in the second half of this calendar year.
Thanks,
Mark.
Please sign in to leave a comment.
Comments
1 comment