HVR Databricks: Support Databricks volumes as staging backend.
Summary
Request support for using Databricks Unity Catalog Volumes as the staging area for HVR Databricks targets, in addition to the currently supported staging backends such as S3, ADLS, and GCS.
Background
Today, HVR documentation for Databricks staging lists cloud-object-storage-based staging options, but does not list Databricks Volumes as a supported staging backend.
However, Databricks documentation indicates the ODBC/SQL path supports PUT operations to Volumes, which suggests this may be technically feasible at the driver/platform level.
Problem
Customers using Databricks—especially those standardized on Unity Catalog and Databricks-native governance patterns—may prefer or require Volumes instead of external object storage for staging.
Currently, the apparent gap is that HVR supports external cloud storage for staging, but not Databricks Volumes.
Requested enhancement
Add support for Databricks Volumes as an HVR staging backend for Databricks targets, including:
- configuration in the Databricks location setup
- validation of required permissions
- support for
Burst Integrate/Bulk Refresh - documentation for supported runtime / driver / privilege requirements
- clear behavior for cleanup, retry, and recovery
Expected benefit
- Better alignment with Unity Catalog-native Databricks deployments
- Reduced dependency on separate staging buckets/containers
- Simpler architecture for customers who want a more Databricks-native staging pattern
- Potentially improved governance and security consistency
Please sign in to leave a comment.
Comments
0 comments