Connector Improvement: SFTP Connector
AnsweredWe are loading market data from vendors which are time sensitive. For most feeds, they are delivered daily. This combination forces us to set fairly aggressive FTP frequencies so that when the data is available we get it quickly, even though most of the time we aren't expecting anything.
I would like to request an option for SFTP (and maybe other similar connectors) where I can specify a time range for aggressive checks and then less aggressive checks the rest of the day just in case there are updates.
For example, we expect a file at 5am. I would like to check every 60 seconds between 5am and 5:30am, and after 5:30a if a certain number of rows haven't been loaded, or a certain number of files found, put the status as "delayed" and send a notification. After the file has arrived, I would like to go back to every 4 hours since I am not expecting anything.
This would allow me to be less aggressive for 23 hours of the day when I am not expecting files and more aggressive during the period when I am.
-
Official comment
Hi Tim,
We are working on CRON style schedules for connectors and will be looking to release this functionality in the next few months.
I'm curious, does the aggressive checking have a real business impact on your system(s), or are you just trying to avoid unnecessary work?
Thanks,
Eric
Please sign in to leave a comment.
Comments
1 comment