Skip to main content

Community

Transformations: Source dbt files from GCS Bucket

Please sign in to leave a comment.

Comments

2 comments

    Hi Christoph Gmeiner, can you tell me more about how these files are stored in your GCS bucket? What structure is applied to the model and package and project files?

    Hi Anna,

    Christoph and I are in the same team, and this request is related: https://support.fivetran.com/hc/en-us/community/posts/4409011892119-Transformations-Manage-simple-SQL-transformations-from-API

    We can focus our discussion here. 

    Currently we are: 

    - Using simple SQL transformations from Fivetran UI
    - Applying dbt code within our own structure (totally bypassing your UI) 

    We would like to:

    - Be able to define and trigger simple SQL transformations from our system through your API, while still seeing them in the UI
    - Be able to define dbt transformations by linking to our dbt files in GCS, because the connection with our Gitlab is problematic. Hence to start using the dbt features provided by your UI

    As for your question about the structure. We have a GCS bucket which stores our Airflow files. This bucket has a dbt folder. Every project has its own directory, for example `dbt/projects/fivetran_log`. The fivetran package is recorded in the `packages.yml` file in that folder and installed automatically through `dbt deps`, therefore it is found in `dbt/projects/fivetran_log/dbt_modules/fivetran_log`. Does this answer your question? 

    Thanks, 

    Vlad

Didn’t find what you need?

Contact support