Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: Snowflake Snowpipe Streaming api

Ali Habib User

Raising a feature request for HVR to support the snowflake streaming API this is to support low latency row based ingestion. Our use-case means we ingest in low latency with 10's millions of rows potentially every day which has a low latency requirement. This is to bypass any additional paths such as Kafka  to utilise the low latency streaming sdk. 

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Mark Van de Wiel User

    Hi Ali,

    We look into this. However, the requirement to implement an interface in Java is foreign to HVR. You may know that almost all components in HVR have no dependency on Java. The only exception is the Haddop (HDFS/Hive) components. We believe there are disadvantages when introducing Java.

    Have you considered looking into hybrid tables to support low latency? Another customer told us they get low latency with hybrid tables and continuous (rather than burst) integration. At present HVR does not create hybrid tables, but I am considering an option to add this to the product. I'd be curious if you run a test with hybrid tables and let me know your experiences.

    Thanks,
    Mark.

  • Ali Habib User

    There are a number of limitations when it comes to hybrid tables, fundementally for us being that dynamic tables are not supported which our multi layer transformations use as a fundemental base. As such and with some of the other limitations we cannot go that direction.