Machine Learning with Spark

Fast, flexible, and developer-friendly platform for large-scale SQL. 

The most recent hot topic for Machine Learning Technology for Scala and Hadoop developers is Apache Spark. Before I begin explaining the roots of this technology, let me tell you why it is a hot topic in recent times?. 

Enthusiastic about exploring the skill set of Apache Spark? Then, have a look at the Spark Training together additional knowledge. 

What is Apache Spark?

Apache Spark is an open-source, general data processing framework in the Apache Hadoop ecosystem. It helps to develop an end-to-end Big Data application fast and accurate. You can even cover live streaming combining batch and responsive interactive analytics to your big data. It is a domain-specific language. It is implemented in Scala as host language in the form of embedded scripts. 

Why and when Apache Spark is in news?

On 28, February 2018 Gartner, Inc a technology research company. They released a research report on most recent technological game changers in the world of Machine learning and Big Data. They listed Databricks ( a company founded by the creators of Apache Spark) as one among the list prepared. It was published in the report named as “ Magic Quadrant for machine learning and data science platforms.“

When such a big technological research company upon research named another company as Winner then there must be something great right!. 

Now, let’s get deeper into the point.

What’s There In Gartner’s Report?

Gartner Inc debuted Databricks for powerful technical bona files which are unquestionable. It is one of the big data distributed processing frameworks of the world. It’s technological innovations in streaming, deep learning, and IoT areas,extensible support to multiple languages like R, Scala, and Python with its notebook-based development environment.

Oho! I talked so much technical right!. You must be wondering what is Apache spark?. After-all, that’s what our topic is all about.  Well!. I am coming to that point next.

What are the companies using Apache Spark?

Amazon EMR, Celtra, Bizo, Dinoping.com, Quantifinda etc., are top technological companies using Apache Spark.

• Amazon supports Apache Spark on the AWS platform through their EMR (Elastic MapReduce) product/service.
• Bizo is a marketing platform for B2B Companies. They use Spark to make their platform more developer friendly. 
• Quantified is a data platform company that drives data of key performance metrics for a business success. Spark helps them in predictive analytics.
• Meituan-Dianping is a Chinese mobile internet company. They operate the information of locals and in-country trading platform.

Are there any non-companies using Apache Spark?

Yes. They are. Some of the companies, using Spark for market analysis and ranking on their On Fair-Recommendation system . Let me tell you who?

1 - UberEats - By the name, you must have understood that it is an  online meal ordering and delivery platform by Uber Technologies. They used Spark’s conventional Machine Language for predictive analysis based on the multi-objective ranking given by On fair-recommendation system. Previously, they have the single-objective ranking which leads to having fewer options for the customers to recommend.
2 - Netflix builds their pipeline tasks for their recommendation engine like label generation, data retrieval, feature generation, training, validations etc., using Spark ML PipleStage framework.
3 - BMW use the available data from cars and workshops to train models that are able to predict the right part to switch or to take action.

Check Out Apache Spark Tutorials

What is making Apache Spark to stand out?

It has got some outstanding features like:

1. Language integrations with, R- programming, Scala etc.,
2. Hardware deployment is compatible with HADOOP. That means companies can deploy Spark using HADOOP Hardware and can use it’s management layer called YARN. 
3. Ease to deployment spark at infrastructure level. They can experiment on elastic infrastructure with real-time data without disturbing the database.
4. Image deduplication, arrangement of photos as per the customer Id, Image categorization.
5. Spark extensive library has GraphX for graph processing
6. MLlib for Machine learning,, Datasets, Dataframes and Streaming. 
7. Spark has APIs for languages like Scala, Java and Python.

Why you should use Spark for Machine Learning?

When you create a Machine learning model, the most important aspect for preparing a model is accuracy in data processing and to save computer memory. If the given dataset does not fit the memory, then you use distributing computing for computing a cluster with many machines. The kind of distributed computing model available till now is HADOOP. But, with SPARK, you are now able to process the data from standalone local machines and build data models with larger input datasets. Usually, these input data sets are larger than the amount of memory your computer has. That is the kind of elasticity Apache Spark provides. Hence we called Apache Spark is featured with Elastic Infrastructure. This enables data scientists to iterate the data problems 100 times more faster than HADOOP. It has got 8000 nodes forming the world’s largest cluster known. Spark’s largest cluster enabling faster deployments

Frequency Asked Apache Spark Interview Questions

What is the Spark Ecosystem? 

• Spark Core Engine supports on Java, R, Python & Scala. It is responsible for basic i/o functionalities, scheduling and monitoring tasks on cluster. 
• Spark SQL  runs SQL queries
• Spark Streaming allows the data processing and streaming
• MLib deploys and develops the Machine learning pipelines.
• GraphX deals with graphs and non graph sources to achieve elasticity in graph construction and transformation.
• Spark R connects R program into the cluster.

Related Page: Machine Learning with Python

How to get started with Apache Spark? 

There are simple 7 steps involved from downloading an Apache Spark to running the application like:

1. Install Java 
2. Install Spark
3. Configure Spark
4. Deploy Spark cluster manually
5. Link with Spark
6. IInstruct Apache Spark to access a cluster
7. Run your Application

You can download Spark. And remember, If you are using Spark in scala programming, you need to install Scala first. Likewise, install the programming language in which you want to use Spark before you actually install Spark.

Once you download the Spark files, just extract the files of  .tar.gz  and save the files in your local system folder.

$ tar xvf spark-1.6.1-bin-hadoop2.6.tgz

Now setup the environment variable for Apache Spark by providing path of your local folder in which you save the files.

To verify the Spark being installed in your system use 

$spark-shell

Now, it’s time to create a cluster manually.
You need to explicitly import the Spark clases into Spark program. This can be done by using the following:

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark._

Spark Classes requires some objects to tell Spark on how to access a cluster.

val sc = new SparkContext( "parameter denotes the master URL to connect the spark application to.", "the name of the application that you want to run", "the home directory of Apache Spark", Nil, Map( ) indicates {the  environment}, Map()-indicates { the variables to work nodes})  

Create a fundamental data structure called as Spark RDD (Resilient Distributed Datasets) using the input file you want to run the application in Spark.

scala> val inputfile = sc.textFile ("input.txt")

Give the output command as 

scala> val outfile = sc.textFile( “output.txt”)

What is the market rate of Apache Spark in the business of Big Data and Machine Learning?

It is expected to grow at a CAGR of 67% between 2017 and 2020 that is up to $4.2 billion by 2020.  As per Apache Spark Market. 

What is the Job Market for Apache Spark?

Oreilly surveyed the job market of Apache Spark and there are approx. 2,579 permanent jobs on Apache Spark till 2018 March in UK, USA and is increasing month by month drastically.

What are the prerequisites to learn Spark?

Spark is a very powerful language, not though to learn of you know the basics of programming languages like  C, C++, core java, php, python and scala. Java, Scala and Python acts major for Spark as the functions and libraries are similar to Python.  Java for coding purposes. C, C++ and PHP are good for OOPs concept, Web and logical integrations. Minimum 3 years real time working experience in these language is recommended to take up Spark as your career.  Scala is use full to build streaming and some data science concepts like numpy, scikit-learn, pandas are recommended.  If you know SQL or if you belong to Database family like SQL/ PLSQL, Spark SQL is the best option for you to explore.