Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Other: Custom S3 endpoint for databricks staging directory

Completed
HVR Platform User

As per versionm 6.3 of hvr, whenever you set the staging directory databricks, there is no option to specify the exact bucket endpoint, only bucket name is given. That is not sufficient enough in a case like mine where i have complex firewall system.

Please sign in to leave a comment.

Comments

6 comments

Date Votes
  • Official comment
    Edwin Fivetranner

    Hi,

    in discussing your case with engineering, it came up that we already have an environmental variable, HVR_S3_ENDPOINT_URL,   which should support your use case:

    It was added in the context of two similar cases:

    1:   f~ [HVR-13981,integ] Support S3 private link through VPC and AWS Direct Connect instead of public s3
    2:   f~ [HVR-15636,integ] Add support for local S3 MinIO storage

    You should be able to use it to let it point to your directory. Otherwise, take this up with Fivetran support.

    Best regards,

    Edwin

  • Edwin Fivetranner

    Hi,

    can you give an example of the naming convention you would like to use?

    Best regards,

    Edwin

  • HVR Platform User

    Hi, I am not sure what naming convention i can provide you. Just give us a field where rather than entering bucket name, we give a url. In my case, the url (masked) i want is vpce-xxxx-cbf0ymxe.s3.eu-central-1.vpce.amazonaws.com

    If you need to setup a call, we would be happy to explain you this

  • Edwin Fivetranner
    • Edited

    I've fed your requirements into engineering. We will check whether we need additional specifications for this and come back here when necessary.

    Best regards,

    Edwin

  • HVR Platform User

    Hi Edwin, Can we have any ETA on this?

  • HVR Platform User

    If i set this variable, this will force hvr to use this endpoint only. But in our case, we have different s3 targets which will have different endpoints. This might only solve for one target. Not the others.

    Let me know if you can find me a way where this will be handled.