Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: SharePoint (and others) Magic Folder sync only with matching regex

Answered
Patrick Negri User

Include subfolder feature was a great addition, but to improve a bit more, we want to sync only files that are marked with a certain pattern like [sync].

Data/Finance/File A.xslx
Data/Finance/Balance Sheet [sync].xlsx
Data/Enginerring/Engineering Salaries [sync].xlsx

That way would vastly improve how we import data that can be analyzed into the warehouse

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Official comment
    Parmeet Kohli Fivetranner

    Hi Patrick,

    Merge Mode may already cover what you’re looking for here. It’s designed for cases where each destination table corresponds to a particular file pattern, so you can selectively sync only the files that match something like [sync] rather than everything in a folder. You can see details here: https://fivetran.com/docs/connectors/files#mergemode.

    If Merge Mode doesn’t work for your setup, could you share what specifically prevents it?

    Thank you,
    Parmeet

  • Patrick Negri User

    The main problem with merge mode is that I must create a pattern for each subset of files that I want to map to a particular table. Isn't what we are looking for.

    Magic Folder works great to get files to their particular table. Teams can just save files in the folders and the data is "magically" available to the warehouse. The main problem is that there are some files that we dont want to go to the warehouse, if we could specify a pattern to only sync files with this pattern, teams could work in our data directory directly at the sharepoint folder, without the need to move files to a sync folder.

    I show this in the original feature request:
    Data/Finance/File A.xslx
    Data/Finance/Balance Sheet [sync].xlsx
    Data/Enginerring/Engineering Salaries [sync].xlsx

    That means that File A don't get sync, but Balance Sheet and Engineering Salaries go automatically to the warehouse. Now think a folder like that with a hundred of files, I dont want to create a Merge Mode for each type of information, I want to consume it like the Magic Folder feature does, but ignore some files.