Skip to main content
📢 Syncs times may take longer than usual during major retail events, such as Black Friday and Cyber Monday due to an increase in activity and larger data sets. For any other issues, please visit our Status Page or contact our Support Team.

Community

Transformations: Source dbt files from GCS Bucket

Answered

Please sign in to leave a comment.

Comments

3 comments

    Hi Christoph Gmeiner, can you tell me more about how these files are stored in your GCS bucket? What structure is applied to the model and package and project files?

    Hi Anna,

    Christoph and I are in the same team, and this request is related: https://support.fivetran.com/hc/en-us/community/posts/4409011892119-Transformations-Manage-simple-SQL-transformations-from-API

    We can focus our discussion here. 

    Currently we are: 

    - Using simple SQL transformations from Fivetran UI
    - Applying dbt code within our own structure (totally bypassing your UI) 

    We would like to:

    - Be able to define and trigger simple SQL transformations from our system through your API, while still seeing them in the UI
    - Be able to define dbt transformations by linking to our dbt files in GCS, because the connection with our Gitlab is problematic. Hence to start using the dbt features provided by your UI

    As for your question about the structure. We have a GCS bucket which stores our Airflow files. This bucket has a dbt folder. Every project has its own directory, for example `dbt/projects/fivetran_log`. The fivetran package is recorded in the `packages.yml` file in that folder and installed automatically through `dbt deps`, therefore it is found in `dbt/projects/fivetran_log/dbt_modules/fivetran_log`. Does this answer your question? 

    Thanks, 

    Vlad

    Thanks for elaborating Vlad Gheorghe. We do not have support for this type of setup currently planned.

    I'd like to understand the issues you're having with Gitlab better– do they extend from the connection to Gitlab from Fivetran Transformations? Or is there complexity in using Airflow to orchestrate dbt through Gitlab?

    In terms of how Fivetran Transformations might be helpful here– is there a layer of processing that happens between Fivetran Connectors and dbt Transformations that must be accounted for?

Didn’t find what you need?

Contact support