According to Gartner, By 2018 spark processing will dominate over hadoop based processing.
We are a big fan of Apache Spark and started building our framework using Spark as the data processing layer. Along the way we contributed to the spark community by releasing Salesforce connector.
Now we are excited to announce our next package SFTP connector for Spark. This is a simple package that allows to connect to a SFTP location to create dataframes directly.
While working with enterprises, getting access to data is a big hurdle. Using the SFTP package we can now access to a secure FTP locations. This package can be deployed on spark instance and easily configurable. More information on setup can be found here.
For any questions on Apache Spark, setup or data processing use cases, feel free to reach out to us at firstname.lastname@example.org