UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
Upload
Sign in
AJ
Uploaded byAsim Jalis
PDF, PPTX13,848 views

Neural Networks, Spark MLlib, Deep Learning

The document discusses neural networks and deep learning, covering their definitions, functionalities, and differences, as well as practical applications in Apache Spark. It outlines the processes of training neural networks, including concepts like feedforward, backpropagation, and gradient descent. Additionally, it highlights various architectures within deep learning, such as auto-encoders and convolutional networks, along with major deep learning platforms.

Data & Analyticsā—¦
Related topics:
Neural Networksā€¢ Deep Learningā€¢ Apache Sparkā€¢
Download as PDF, PPTX
1 / 87
2 / 87
3 / 87
4 / 87
5 / 87
6 / 87
7 / 87
8 / 87
9 / 87
10 / 87
11 / 87
12 / 87
13 / 87
14 / 87
15 / 87
16 / 87
17 / 87
18 / 87
19 / 87
20 / 87
NEURAL NETWORKS AND DEEP LEARNING BY ASIM JALIS GALVANIZE
WHO AM I?
ASIM JALIS Galvanize/Zipfian, Data Engineering Cloudera, Microso!, Salesforce MS in Computer Science from University of Virginia https://www.linkedin.com/in/asimjalis
WHAT IS GALVANIZEā€™S DATA ENGINEERING PROGRAM?
DO YOU WANT TO . . . Play with terabytes of data Build data applications using Spark, Hadoop, Hive, Kafka, Storm, HBase Use Data Science algorithms at scale
WHAT IS INVOLVED? Learn concepts in interactive lectures Develop skills in hands-on labs Design and build your Capstone Project Show project to SF tech companies at Hiring Day
FOR MORE INFORMATION Check out Talk to me http://galvanize.com asim.jalis@galvanize.com
INTRO
WHAT IS THIS TALK ABOUT? What are Neural Networks and how do they work? What is Deep Learning? What is the diļ¬€erence? How can we build neural networks in Apache Spark?
HOW MANY PEOPLE HERE ARE FAMILIAR WITH NEURAL NETWORKS?
HOW MANY PEOPLE HERE ARE FAMILIAR WITH CONVOLUTION NEURAL NETWORKS?
HOW MANY PEOPLE HERE ARE FAMILIAR WITH DEEP LEARNING?
HOW MANY PEOPLE HERE ARE FAMILIAR WITH APACHE SPARK AND MLLIB?
NEURAL NETWORKS
WHAT IS A NEURON?
Receives signal on synapse When trigger sends signal on axon
Mathematical abstraction Inspired by biological neuron Either on or oļ¬€ based on sum of input
Neuron is a mathematical function Adds up (weighted) inputs Applies the sigmoid function This determines if it fires or not
WHAT ARE NEURAL NETWORKS? Biologically inspired machine learning algorithm Mathematical neurons arranged in layers Accumulate signals from the previous layer Fire when signal reaches threshold
HOW MANY NEURONS SHOULD I HAVE IN MY NETWORK?
HOW MANY INPUT LAYER NEURONS SHOULD WE HAVE?
The number of inputs or features
HOW MANY OUTPUT LAYER NEURONS SHOULD WE HAVE?
The number of classes we are classifying the input into.
HOW MANY HIDDEN LAYER NEURONS SHOULD WE HAVE?
SIMPLEST OPTION IS TO USE 0.
SINGLE LAYER PERCEPTRON
WHAT ARE THE DOWNSIDES OF NO HIDDEN LAYERS? Only works if data is linearly separable. Identical to logistic regression.
MULTILAYER PERCEPTRON For most realistic classification tasks you will need a hidden layer. Rule of thumb: Number of hidden layers equals one Number of neurons in hidden layer is mean of size of input and output layers.
HOW DO WE USE THIS THING?
NEURAL NETWORK WORKFLOW Split labeled data into train and test sets Train with labeled data Test and compare prediction with actual labels
HOW DO WE TRAIN IT?
FEED FORWARD Also called forward propagation or forward prop Initialize inputs Weigh inputs into hidden layer, sum, apply sigmoid Calculate activation of hidden layer Weight inputs into output layer, sum, apply sigmoid Calculate activation of output layer
BACK PROPAGATION Use forward prop to calculate the error Error is function of all network weights Adjust weights using gradient descent Repeat with next record Keep going over training set until convergence
WHAT IS GRADIENT DESCENT?
HOW DO YOU FIND THE MINIMUM IN AN N-DIMENSIONAL SPACE? Take a step in the steepest direction. Steepest direction is vector sum of all derivatives.
PUTTING ALL THIS TOGETHER
Use forward prop to activate Use back prop to train Then use forward prop to test
WHY NOT HAVE MULTIPLE LAYERS?
DOWNSIDE OF MULTIPLE LAYERS Number of weights is a product of the layer sizes The mathematics quickly becomes intractable Particularly when your input is an image with tens of thousands of pixels
APACHE SPARK MLLIB
WHAT IS SPARK
Framework for processing data across a cluster By sending the code to the data And executing the code where the data lives
WHAT IS MLLIB? Library for Machine Learning. Builds on top of Spark RDDs. Provides RDDs for Machine Learning. Implements common Machine Learning algorithms.
DEMO USING APACHE TOREE
WHAT IS APACHE TOREE? Like IPython Notebook but for Spark/Scala. Jupyter kernel for Spark/Scala.
HOW CAN I INSTALL TOREE? Use pip to install IPython or Jupyter. Install Apache Spark by downloading tgz file and expanding. SPARK_HOME=$HOME/spark-1.6.0 pip install toree jupyter toree install --spark_home=$SPARK_HOME
HOW CAN I RUN A TOREE NOTEBOOK jupyter notebook Visit Create new notebook. Set kernel to Toree. sc in notebook should print Spark Context. http://localhost:8888
NEURAL NETWORK CONSTRUCTION
HOW CAN I FIGURE OUT HOW MANY LAYERS? To figure out how many layers to use and what topology to use you have to rely on standard machine learning techniques. Use cross-validation. In general k-fold cross validation. 10-fold cross validation is popular.
WHAT IS 10-FOLD CROSS VALIDATION OR K-FOLD CROSS VALIDATION?
Split your data into 10 (or in general k) equal-sized subsets. Train model on 9 of them, set one aside for cross- validation. Validate model on 10th and remember your error rate. Repeat by setting aside each one of the 10. Average the 10 error rates. Then repeat for the next model. Choose the model with the lowest error rate.
HOW DO I DEPLOY MY NEURAL NETWORK INTO PRODUCTION? There are two phases. The training phase can be run on the back-end servers. Cross-validate your model and its hyper-parameters on the back-end. Then deploy the model to the front-end servers, browsers, devices. The front-end only uses forward prop and is always fast.
DEEP LEARNING
WHAT IS DEEP LEARNING? Deep Learning is a learning method that can train the system with more than 2 or 3 non-linear hidden layers.
WHAT IS DEEP LEARNING? Machine learning techniques which enable unsupervised feature learning and pattern analysis/classification. The essence of deep learning is to compute representations of the data. Higher-level features are defined from lower-level ones.
HOW IS DEEP LEARNING DIFFERENT FROM REGULAR NEURAL NETWORKS? Training neural networks requires applying gradient descent on millions of dimensions. This is intractable for large networks. Deep learning places constraints on neural networks. This allows them to be solvable iteratively. The constraints are generic.
WHAT IS THE BIG DEAL ABOUT IT? AlexNet submitted to the ImageNet ILSVRC challenge in 2012 is partly responsible for the renaissance. Alex Krizhevsky, Ilya Sutskever, and Geoļ¬€rey Hinton used Deep Learning techniques. They combined this with GPUs, some other techniques. The result was a neural network that could classify images of cats and dogs. It had an error 16% compared to 26% for the runner up.
ILYA SUTSKEVER, ALEX KRIZHEVSKY, GEOFFREY HINTON
WHAT ARE THE DIFFERENT KINDS OF DEEP ARCHITECTURES? Generative Discriminative Hybrid
WHAT ARE GENERATIVE ARCHITECTURES Extract features from data Find common features in unlabelled data Like Principal Component Analysis Unsupervised: no labels required
WHAT ARE DISCRIMINATIVE ARCHITECTURES Classify inputs into classes Require labels Require supervised training
WHAT ARE HYBRID ARCHITECTURES? STEP 1 Combination of generative and discriminative Extract features using generative network Use unsupervised learning STEP 2 Train discriminative network on extracted features Use supervised learning
WHAT ARE AUTO-ENCODERS? An auto-encoder is a learning algorithm. It applies backpropagation and sets the target values to be equal to its inputs. In other words it trains itself to do the identity transformation.
WHY DOES IT DO THIS? By placing constraints on it, like restricting the number of hidden neurons, it can find a good representation of the data.
IS THE AUTO-ENCODER SUPERVISED OR UNSUPERVISED?
It is unsupervised. The data is unlabeled. Auto-encoders are similar to PCA (Principal Component Analysis). PCA is a technique for reducing the dimensions of data.
WHAT ARE CONVOLUTION NEURAL NETWORKS? Feedforward neural networks. Connection pattern inspired by visual cortex.
CONVOLUTION NEURAL NETWORKS The convolution layerā€™s parameters are a set of learnable filters. Every filter is small along width and height. During the forward pass, each filter slides across the width and height of the input, producing a 2-dimensional activation map. As we slide across the input we compute the dot product between the filter and the input.
CONVOLUTION NEURAL NETWORKS Intuitively, the network learns filters that activate when they see a specific type of feature anywhere. In this way it creates translation invariance.
WHAT IS A POOLING LAYER? The pooling layer reduces the resolution of the image further. It tiles the output area with 2x2 mask and takes the maximum activation value of the area.
DOES SPARK SUPPORT DEEP LEARNING? Not directly yet https://issues.apache.org/jira/browse/SPARK-2352
WHAT ARE SOME MAJOR DEEP LEARNING PLATFORMS?
Theano: Low-level GPU-enabled tensor library. Lasagne, Blocks: NN libraries that make Theano easier to use. Torch7: NN library. Uses Lua for binding. Used by Facebook and Google. Caļ¬€e: NN library by Berkeley AMPLab. Pylearn2: ML library based on Theano by University of Toronto. Google DeepMind. cuDNN: NN library by Nvidia based on CUDA. Can be used with Torch7, Caļ¬€e. Chainer: NN library that uses CUDA. TensorFlow: NN library from Google.
WHAT LANGUAGE ARE THESE IN? All the frameworks support Python. Except Torch7 which uses Lua for its binding language.
WHAT CAN I DO ON SPARK? SparkNet: Integrates running Caļ¬€e with Spark. Sparkling Water: Integrates H2O with Spark. DeepLearning4J: Built on top of Spark. TensorFlow on Spark (experimental)
QUESTIONS
GALVANIZE DATA ENGINEERING