Image credit: Nvidia

Deep Learning Explained

The future of Smart Networks

Melanie Swan
Philosophy & Economic Theory
Purdue University
Boulder Futurists: Solid State Depot Hackspace melanie@[Link]
Boulder CO, August 12, 2017
Slides: [Link]
 Melanie Swan, Technology Theorist
 Philosophy and Economic Theory, Purdue
University, Indiana, USA
 Founder, Institute for Blockchain Studies
 Singularity University Instructor; Institute for Ethics and
Emerging Technology Affiliate Scholar; EDGE
Essayist; FQXi Advisor
Economics and Financial
Traditional Markets Background Theory Leadership

New Economies research group

[Link]

Source: [Link] [Link] [Link]

12 Aug 2017
2
Deep Learning
 Agenda
 Deep Learning
 Definition
 Technical details
 Applications
 Deep Qualia: Deep Learning and the Brain
 Smart Network Convergence Theory
 Conclusion

Image Source: [Link]

12 Aug 2017
3
Deep Learning
 Deep Learning vocabulary
What do these terms mean?

 Deep Learning, Machine Learning, Artificial Intelligence

 Perceptron, Artificial Neuron, Logit
 Deep Belief Net, Artificial Neural Net, Boltzmann Machine
 Google DeepDream, Google Brain, Google DeepMind
 Supervised and Unsupervised Learning
 Convolutional Neural Nets
 Recurrent NN & LSTM (Long Short Term Memory)
 Activation Function ReLU (Rectified Linear Unit)
 Deep Learning libraries and frameworks
 TensorFlow, Caffe, Theano, Torch, DL4J
 Backpropagation, gradient descent, loss function

12 Aug 2017
4
Deep Learning
 Conceptual Definition:
Deep learning is a computer program that can
identify what something is

Technical Definition:
Deep learning is a class of machine learning
algorithms in the form of a neural network that
uses a cascade of layers (tiers) of processing
units to extract features from data and make
predictive guesses about new data

Source: Extending Jann LeCun, [Link]

12 Aug 2017 on-deep-learning
5
Deep Learning
 Deep Learning Theory
 System is “dumb” (i.e. mechanical)
 “Learns” with big data (lots of input examples) and trial-and-error
guesses to adjust weights and bias to establish key features
 Creates a predictive system to identity new examples
 Same AI argument: big enough data is what makes a
difference (“simple” algorithms run over large data sets)

Input: Big Data (e.g.; Method: Trial-and-error Output: system identifies

many examples) guesses to adjust node new examples
weights
12 Aug 2017
6
Deep Learning
 Sample task: is that a Car?
 Create an image recognition system that determines
which features are relevant (at increasingly higher levels
of abstraction) and correctly identifies new examples

Source: Jann LeCun, [Link]

12 Aug 2017
7
Deep Learning
 Broader Computer Science Context
 Within the Computer Science discipline, in the field of
Artificial Intelligence, Deep Learning is a class of
Machine Learning algorithms, that are in the form of a
Neural Network

Source: Machine Learning Guide, 9. Deep Learning

12 Aug 2017
8
Deep Learning
 Statistical Mechanics
Deep Learning is inspired by Physics
 Sigmoid function suggested as a model for neurons,
per statistical mechanical behavior (Jack Cowan)
 Stationary solutions for dynamic models (asymmetric
weights create an oscillator to model neuron signaling)
 Hopfield Neural Network: content-addressable
memory system with binary threshold nodes,
converges to a local minimum (John Hopfield)
 Can use an Ising model (of ferromagnetism) for neurons
 Restricted Boltzmann Machine (Geoffrey Hinton)
 Studied in theoretical physics, condensed matter field
theory; Statistical Mechanics concepts: Renormalization,
Boltzmann Distribution, Free Energies, Gibbs Sampling

Source: [Link]
12 Aug 2017
9
Deep Learning
 What is a Neural Net?
 Motivation: create an Artificial Neural Network to solve
problems the same way a human brain would

12 Aug 2017
10
Deep Learning
 What is a Neural Net?
 Structure: input-processing-output
 Mimic neuronal signal firing structure of brain with
computational processing units

Source: [Link]
12 Aug 2017 [Link]
11
Deep Learning
 What is an Artificial Neural Network?
 Collection of connected units called artificial
neurons (analogous to axons in a biological brain)
 Organized in layers of signaling cascades
 Each neuron transmits a signal to another neuron
 Neurons may have state
 Represented by a number between 0 and 1
 Variable parameters
 Neurons may have a weight that varies as learning
proceeds, which can increase or decrease the strength of
the signal that it sends downstream
 Neurons may have a threshold (bias) such that only if the
aggregate signal is below (or above) that level is the
downstream signal sent

12 Aug 2017
12
Deep Learning
 Why is it called Deep Learning?
 Deep: Hidden layers (cascading tiers) of processing
 “Deep” networks (5+ layers) versus “shallow” (1-2 layers)
 Learning: Algorithms “learn” from data by modeling
features and updating probability weights assigned to
feature nodes in testing how relevant specific features
are in determining the general type of item

Deep: Hidden processing layers Learning: Updating probability

weights re: feature importance

12 Aug 2017
13
Deep Learning
 Supervised and Unsupervised Learning
 Supervised (classify
labeled data)

 Unsupervised (find
patterns in unlabeled
data)

Source: [Link]
12 Aug 2017
14
Deep Learning
 Early success in Supervised Learning (2011)
 YouTube: user-classified data
perfect for Supervised Learning

Source: Google Brain: Le, QV, Dean, Jeff, Ng, Andrew, et al. 2012. Building high-level features using large scale unsupervised
12 Aug 2017 learning. [Link]
15
Deep Learning
 2 main kinds of Deep Learning neural nets
 Convolutional Neural Nets
 Image recognition
 Convolve: roll up to higher
levels of abstraction in feature
sets
 Recurrent Neural Nets
 Speech, text, audio
recognition
 Recur: iterate over sequential
inputs with a memory function
 LSTM (Long Short-Term
Memory) remembers
sequences and avoids
gradient vanishing
Source: Yann LeCun, CVPR 2015 keynote (Computer Vision ), "What's wrong with Deep Learning" [Link]
12 Aug 2017
16
Deep Learning
 Image Recognition and Computer Vision
History

Marv Minsky, 1966 Jeff Hawkins, 2004, Hierarchical Quoc Le, 2011, Google
“summer project” Temporal Memory (HTM) Brain cat recognition

Current state of
the art - 2017

Convolutional net for autonomous driving, [Link]

Source: Quoc Le, [Link] Yann LeCun, NIPS 2016,

12 Aug 2017 [Link]
17
Deep Learning
 Progression in AI Deep Learning machines

Deep Blue, 1997 Watson, 2011 AlphaGo, 2016

Hard-coded AI machine Deep Learning prototype Deep Learning machine

Single-purpose AI: Question-answering AI: Multi-purpose AI:

Hard-coded rules Natural-language processing Algorithm detects rules,
reusable template

12 Aug 2017
18
Deep Learning
 Why do we need Deep Learning?
 A contemporary data science method to keep up
with the growth in data, older learning algorithms no
longer performing

Source: [Link]
12 Aug 2017
19
Deep Learning
 Agenda
 Deep Learning
 Definition
 Technical details
 Applications
 Deep Qualia: Deep Learning and the Brain
 Smart Network Convergence Theory
 Conclusion

Image Source: [Link]

12 Aug 2017
20
Deep Learning
 3 Key Technical Principles of Deep Learning
Sigmoid Function Perceptron Structure Loss Function

Squash values into Core processing unit Reduce combinatoric

Sigmoidal S-curve (input-processing-output) dimensionality
What -Binary values (Y/N, 0/1) Levers: weights and bias
-Probability values (0 to 1)
-Tanh values 9(-1) to 1)

Non-linear formulation “Dumb” system learns by Loss function

Why as a logistic regression adjusting parameters and optimizes efficiency
problem means checking against outcome of solution
greater mathematical
manipulation

12 Aug 2017
21
Deep Learning
 Linear Regression

House price vs. Size (square feet)

House price

y=mx+b Size (square feet)

Source: [Link]
12 Aug 2017
22
Deep Learning
 Logistic Regression

Source: [Link]
12 Aug 2017
23
Deep Learning
 Logistic Regression
 Higher-order mathematical
formulation Sigmoid Function
 Sigmoid function
 S-shaped and bounded
 Maps the whole real axis into a finite
interval (0-1)
 Non-linear
 Can fit probability
 Can apply optimization techniques

 Deep Learning classification

predictions are in the form of a
probability value
Unit Step Function

Source: [Link]
12 Aug 2017
24
Deep Learning
 Sigmoid function: Taleb
 Thesis: if can map a phenomenon onto
a sigmoid curve (“convexify” it), then
can control its risk
 Antifragility = convexity = risk-manageable
 Fragility = concavity
 Non-linearity of dose-response in
medicine, and therefore suggested
treatment optimality

Source: [Link]
12 Aug 2017
25
Deep Learning
 Regression

Linear Regression Logistic Regression (Sigmoid function)

(0-1) or Tanh ((-1)-1)

 Linear regression  Logistic regression

 Predict continuous set  Predict binary outcomes:
of values (house prices)  Perceptron (0 or 1)
 Predict probabilities:
 Sigmoid Neuron (values 0-1)
 Tanh Hyperbolic Tangent
Neuron (values (-1)-1)

12 Aug 2017
26
Deep Learning
 Deep Learning Architecture

Source: Michael A. Nielsen, Neural Networks and Deep Learning

12 Aug 2017
27
Deep Learning
 Processing Unit, Perceptron, Neuron
 Unit (processing unit, logistic regression
unit), perceptron (“multilayer perceptron”),
artificial neuron
1. Input 2. Hidden layers 3. Output

X X X X X
X X X X X
X X X X X
X X X X X
X X X X X
X X X X X
X X X X X
Source: [Link]
12 Aug 2017
28
Deep Learning
 Example: Image recognition
1. Obtain training data set

2. Digitize pixels (convert images to numbers)

 Divide image into 28x28 grid, assign a value (0-255) to each
square based on brightness

3. Read into vector (array; list of numbers)

 28x28 = 784 elements per image)

Source: Quoc V. Le, A Tutorial on Deep Learning, Part 1: Nonlinear Classifiers and The Backpropagation Algorithm, 2015, Google
12 Aug 2017 Brain, [Link]
29
Deep Learning
 Deep Learning Architecture
4. Load spreadsheet of vectors into deep learning system
 Each row of spreadsheet (784-element array) is an input

1. Input 2. Hidden layers 3. Output

Vector data

784-element array
X X X X X
X X X X X
X X X X X
X X X X X
X X X X X
X X X X X
X X X X X
Source: [Link] MNIST dataset: [Link]
12 Aug 2017
30
Deep Learning
 What happens in the Hidden Layers?
 First layer learns primitive features (line, edge, tiniest
unit of sound) by finding combinations of the input vector
data that occur more frequently than by chance
 Logistic regression performed and encoded at each processing
node (Y/N (0,1)), does this example have this feature?
 Feeds these basic features to next layer, which trains
itself to recognize slightly more complicated features
(corner, combination of speech sounds)
 Feeds features to new layers until recognizes full objects

Source: Michael A. Nielsen, Neural Networks and Deep Learning

12 Aug 2017
31
Deep Learning
 Image Recognition
Higher Abstractions of Feature Recognition

Edges Object Parts Object Models

(combinations of edges)

Source: Jann LeCun, [Link]

12 Aug 2017
32
Deep Learning
 Speech, Text, Audio Recognition
Sequence-to-sequence Recognition + LSTM

Source: Andrew Ng
12 Aug 2017
33
Deep Learning
 Example: NVIDIA Facial Recognition
 First hidden layer extracts all possible low-level features
from data (lines, edges, contours); next layers abstract
into more complex features of possible relevance

Source: Nvidia
12 Aug 2017
34
Deep Learning
 Deep Learning

Source: Quoc V. Le et al, Building high-level features using large scale unsupervised learning, 2011, [Link]
12 Aug 2017
35
Deep Learning
 Deep Learning Architecture

1. Input 2. Hidden layers 3. Output

(0,1)

Source: Michael A. Nielsen, Neural Networks and Deep Learning

12 Aug 2017
36
Deep Learning
 Mathematical methods update weights
 Linear algebra: matrix multiplications of input vectors
 Statistics: logistic regression units (Y/N (0,1)), probability
weighting and updating, inference for outcome prediction
 Calculus: optimization (minimization), gradient descent in
back-propagation to avoid local minima with saddle points
Inference Actual
1. Input 2. Hidden layers 3. Output Guess

Feed-forward pass (0,1)

X .5 X .5 X .5 X
.75
X 0 0
Backward pass to update probabilities

X .5
X .5
X .5
X X 1 0
.25

X X X X X
Source: [Link] MNIST dataset: [Link]
12 Aug 2017
37
Deep Learning
 More complicated in actual use
 Convolutional neural net scale-up for
number recognition
 Example data: MNIST dataset
 [Link]

Source: [Link]
12 Aug 2017
38
Deep Learning
 Structure of a Node: Computation Graph

Edge
(input value) 3

Node
(operation) Add
Edge ??
(output value)
Edge
4
(input value)

Architecture Example 1

Mult
??

Example 2
12 Aug 2017
39
Deep Learning
 Neural net unit: perceptron, neuron, node
 Sigmoid function means all inputs and outputs in the
system are (0,1)

(0,1)

Oper-
ation
(0,1) (0,1)

(0,1)

Source: [Link]
12 Aug 2017
40
Deep Learning
 Other parameters: weights and bias
 Weight and bias are variable parameters that
get adjusted as the system iterates and learns
Values have Operation node Values have Operation node
Weights has Bias Weights has Bias
B=3 =0 B=3
W1 = (-2) W1 = (-2)

Output
=0
=0
W2 = (-2) W2 = (-2)

Input (0,1) X1, X2 W1*X1 + W2*X2 + Bias = n Output (0,1)

0,0 (-2)*0 + (-2)*0 + 3 = 3 1
Weight and Bias are
“randomly” assigned at the 0,1 (-2)*0 + (-2)*1 + 3 = 1 1
beginning: (here (-2) and 3) 1,0 (-2)*1 + (-2)*0 + 3 = 1 1
Mimics NAND gate 1,1 (-2)*1 + (-2)*1 + 3 = (-1) 0

Source: [Link]
12 Aug 2017
41
Deep Learning
 Actual: same structure, more complicated

12 Aug 2017
42
Deep Learning
 Neural net: massive scale-up of nodes

Source: [Link]
12 Aug 2017
43
Deep Learning
 Same Structure

12 Aug 2017
44
Deep Learning
 How does the neural net actually learn?
 Structural system based on cascading layers of
neurons with variable parameters: weight and bias
 Vary the weights and
biases to see if a
better outcome is
obtained
 Repeat until the net
correctly classifies
the data

Source: [Link]
12 Aug 2017
45
Deep Learning
 Backpropagation
 Problem: Inefficient to test the combinatorial
explosion of all possible parameter variations

 Solution: Backpropagation (1986 Nature paper)

 Backpropagation is an optimization method used to
calculate the error contribution of each neuron after
a batch of data is processed

Source: [Link]
12 Aug 2017
46
Deep Learning
 Backpropagation of error
 Calculate the total error
 Calculate the contribution to the error at each step
going backwards
 Variety of Error Calculation methods: Mean Square Error
(MSE), sum of squared errors of prediction (SSE), Cross-
Entropy (Softmax), Softplus

12 Aug 2017
47
Deep Learning
 Backpropagation
 Heart of Deep Learning
 Backpropagation: algorithm dynamically calculates
the gradient (derivative) of the loss function with
respect to the weights in a network to find the
minimum and optimize the function from there
 Algorithms optimize the performance of the network by
adjusting the weights, e.g.; in the gradient descent algorithm
 Error and gradient are computed for each node
 Intermediate errors transmitted backwards through the
network (backpropagation)
 Objective: optimize the weights so that the neural
network can learn how to correctly map arbitrary
inputs to outputs
Source: [Link]
12 Aug 2017 [Link]
48
Deep Learning
 Gradient Descent
 Gradient: derivative to find the minimum of a function
 Gradient descent: optimization algorithm to find the
biggest errors (minima) most quickly
 Error = MSE, log loss, cross-entropy; e.g.; least correct
predictions to correctly identify data

Source: [Link]
12 Aug 2017
49
Deep Learning
 Loss Function
 Optimization Technique
 Mathematical tool used in statistics, finance, decision theory,
Laplace
biological modeling, computational neuroscience
 State as non-linear equation to optimize
 Minimize loss or cost
 Maximize reward, utility, profit, or fitness
 Loss function links instance of an event to its cost
 Accident (event) means $1,000 damage on average (cost)
 5 cm height (event) confers 5% fitness advantage (reward)
 Deep learning: system feedback loop
 Use penalty cost for incorrect classifications to train system
 CNN (classification): cross-entropy; RNN (regression): MSE

12 Aug 2017
50
Deep Learning
 Overfitting
 Regularization
 Introduce additional information
such as a lambda parameter in the
cost function (to update the theta
parameters in the gradient descent
algorithm)
 Dropout: prevent complex
adaptations on training data by
dropping out units (both hidden and
visible)
 Test new datasets

12 Aug 2017
51
Deep Learning
 Research Topics
 Layer depth vs. height (1x9, 3x3, etc.); L1/2 slow-downs

 Backpropagation, gradient descent, loss function

 Saddle-free optimization, vanishing gradients
 Composition of non-linearities
 Non-parametric manifold learning, auto-encoders
 Activation maximization
 Synthesizing preferred inputs for neurons

Source: [Link] [Link]

12 Aug 2017 [Link]
52
Deep Learning
 Advanced
Deep Learning Architectures
 Deep Belief Network
 Connections between layers not units
 Establish weighting guesses for
processing units before run deep Deep
Belief
learning system Network
 Used to pre-train systems to assign
initial probability weights (more efficient)
 Deep Boltzmann Machine
 Stochastic recurrent neural network
 Runs learning on internal
representations
 Represent and solve combinatoric Deep
problems Boltzmann
Machine

Source: [Link]
12 Aug 2017
53
Deep Learning
 Convolutional net: Image Enhancement
 Google DeepDream: Convolutional neural network
enhances (potential) patterns in images; deliberately
over-processing images

Source: Georges Seurat, Un dimanche après-midi à l'Île de la Grande Jatte, 1884-1886;

12 Aug 2017 [Link] Google DeepDream uses algorithmic pareidolia (seeing an image
54
Deep Learning when none is present) to create a dream-like hallucinogenic appearance
 Hardware and Software Tools

12 Aug 2017
55
Deep Learning
 Deep Learning frameworks and libraries

Source: [Link]
12 Aug 2017 [Link]#[Link]-ifwsb
56
Deep Learning
 What is TensorFlow?
 Google’s open-source machine learning library
 “Tensor” = multidimensional arrays used in NN operations

Computation graph Design in TensorFlow

TensorBoard (TensorFlow) visualization Python code invoking TensorFlow

Source: [Link]
12 Aug 2017
57
Deep Learning
 Hardware
 Advances in chip design
 GPU chips (graphics processing unit):
3D graphics cards designed to do fast
matrix multiplication
 Google TPU chip (tensor processing
unit): custom ASICs for machine
learning, used in AlphaGo
 TPUs process matrix Google TPU chip (Tensor
multiplications without storing Processing Unit), 2016

intermediate values in memory

 NVIDIA DGX-1 integrated deep
learning system NVIDIA DGX-1
 Eight Tesla P100 GPU accelerators Deep Learning System

Source: [Link]
12 Aug 2017 the-future-of-computing-looks-like-1326915
58
Deep Learning
 USB and Browser-based Machine Learning
 Intel: Movidius Visual Processing
Unit (VPU): USB ML for IOT
 Security cameras, industrial
equipment, robots, drones
 Apple: ML acquisition Turi (Dato)
 Browser-based Deep Learning
 ConvNetJS; TensorFire
 Javascript library to run Deep
Learning (Neural Networks) in a
browser
 Smart Network in a browser
 JavaScript Deep Learning
 Blockchain EtherWallets

Source: [Link] [Link]

12 Aug 2017 [Link]
59
Deep Learning
 How big are Deep Learning neural nets?
 Google Deep Brain cat recognition, 2011
 1 billion connections, 10 million images (200x200
pixel), 1,000 machines (16,000 cores), 3 days, each
instantiation of the network spanned 170 servers, and
20,000 object categories
 State of the art, 2016-2017
 NVIDIA facial recognition, 100 million images, 10
layers, 1 bn parameters, 30 exaflops, 30 GPU days
 Google, 11.2-billion parameter system
 Lawrence Livermore Lab, 15-billion parameter system
 Digital Reasoning, cognitive computing (Nashville TN),
160 billion parameters, trained on three multi-core
computers overnight

Source: [Link] Digital Reasoning paper:

12 Aug 2017 [Link]
60
Deep Learning
 Agenda
 Deep Learning
 Definition
 Technical details
 Applications
 Deep Qualia: Deep Learning and the Brain
 Smart Network Convergence Theory
 Conclusion

Image Source: [Link]

12 Aug 2017
61
Deep Learning
 Applications: Cats to Cancer to Cognition

Computational imaging: Machine learning for 3D microscopy

[Link]

Source: Yann LeCun, CVPR 2015 keynote (Computer Vision ), "What's wrong with Deep Learning" [Link]
12 Aug 2017
62
Deep Learning
 Tumor Image Recognition
 Computer-Aided
Diagnosis with
Deep Learning
Architecture
 Breast tissue
lesions in images
and pulmonary
nodules in CT
Scans

Source: [Link]
12 Aug 2017
63
Deep Learning
 Melanoma Image Recognition

Source: [Link]
12 Aug 2017
64
Deep Learning
 DIY Image Recognition: use Contrast
Apple or Orange? Melanoma risk or healthy skin?

Degree of contrast in photo colors?

How many orange pixels?

Source: [Link]
12 Aug 2017
65
Deep Learning
 Deep Learning and Genomics
 Large classes of hypothesized but unknown correlations
 Genotype-phenotype disease linkage unknown
 Computer-identifiable patterns in genomic data
 CNN: genome symmetries; RNN: textual analysis

Source: [Link]
12 Aug 2017
66
Deep Learning
 Deep Learning and the Brain

12 Aug 2017
67
Deep Learning
 Deep Qualia machine? General purpose AI
Mutual inspiration of neurological and computing research

 Deep learning neural networks are inspired by the

structure of the cerebral cortex
 The processing unit, perceptron, artificial neuron is the
mathematical representation of a biological neuron
 In the cerebral cortex, there can be several layers of
interconnected perceptrons

12 Aug 2017
68
Deep Learning
 Deep Qualia machine?
 Visual cortex is hierarchical with intermediate layers
 The ventral (recognition) pathway in the visual cortex has multiple
stages: Retina - LGN - V1 - V2 - V4 - PIT – AIT
 Human brain simulation projects
 Swiss Blue Brain project, European Human Brain Project

Source: Jann LeCun, [Link]

12 Aug 2017
69
Deep Learning
 Social Impact of Deep Learning
 WHO estimates 400 million people without
access to essential health services
 6% in extreme poverty due to healthcare costs
 Next leapfrog technology: Deep Learning
 Last-mile build out of brick-and-mortar clinics
does not make sense in era of digital medicine
 Medical diagnosis via image recognition, natural
language processing symptoms description
 Convergence Solution: Digital Health Wallet
 Deep Learning medical diagnosis + Blockchain-
based EMRs (electronic medical records)
 Empowerment Effect: Deep learning = “tool I Digital Health Wallet:
Deep Learning diagnosis
use,” not hierarchically “doctor-administered” Blockchain-based EMRs

Source: [Link]
12 Aug 2017
70
Deep Learning
 Agenda
 Deep Learning
 Definition
 Technical details
 Applications
 Deep Qualia: Deep Learning and the Brain
 Smart Network Convergence Theory
 Conclusion

Image Source: [Link]

12 Aug 2017
71
Deep Learning
 New Technology

Better horse AND new car

12 Aug 2017
72
Deep Learning
 Smart Network Convergence Theory

Smart networks are computing networks with

intelligence built in such that identification
and transfer is performed by the network
itself through protocols that automatically
identify (deep learning), and validate,
confirm, and route transactions (blockchain)
within the network

12 Aug 2017
73
Deep Learning
 Smart Network Convergence Theory
Two Fundamental Eras of Network Computing

 Network intelligence “baked in” to smart networks

 Deep Learning algorithms for predictive identification
 Blockchains to transfer value, confirm authenticity

Source: Expanded from Mark Sigal, [Link]

12 Aug 2017
74
Deep Learning
 Blockchain is the tamper-resistant
distributed ledger software underlying
cryptocurrencies such as Bitcoin, for the
secure transfer of money, assets, and
information via the Internet without a third-
party intermediary

Source: [Link]
12 Aug 2017
75
Deep Learning
 Blockchain Deep Learning nets
 Provide increasingly sophisticated automated network
computational infrastructure
 Make predictive guesses of reality states of the world
 Predictive inference (deep learning) and cryptographic nonce-
guesses (blockchain)
 Instantiate decentralization
 Hierarchical models do not scale

12 Aug 2017
76
Deep Learning
 Next Phase

 Put Deep Learning systems on the Internet

 Deep Learning Blockchain Networks
 Combine Deep Learning and Blockchain Technology
 Blockchain offers secure audit ledger of activity
 Advanced computational infrastructure to tackle
larger-scale problems
 Genomic disease, protein modeling, energy storage,
global financial risk assessment, voting, astronomical
data

12 Aug 2017
77
Deep Learning
 Example: Autonomous Driving
 Requires the smart network functionality
of deep learning and blockchain

 Deep Learning: identify what things are

 Convolutional neural nets core element of
machine vision system
 Blockchain: secure automation
technology
 Track arbitrarily-many fleet units
 Legal accountability
 Software upgrades
 Remuneration

12 Aug 2017
78
Deep Learning
 The Very Small
Blockchain Deep Learning nets in Cells
 Medical nanorobotics for cell repair
 Deep Learning: identify what things are
(diagnosis)
 Blockchain: secure automation technology
 Bio-cryptoeconomics: secure automation
of medical nanorobotics for cell repair
 Medical nanorobotics as coming-onboard
repair platform for the human body
 High number of agents and “transactions”
 Identification and automation is obvious

Sources: Swan, M. Blockchain Thinking: The Brain as a DAC (Decentralized Autonomous Corporation). Technology and Society Magazine,
12 Aug 2017 IEEE 2015; 34(4): 41-52 , [Link]
79
Deep Learning
 The Very Large
Blockchain Deep Learning nets in Space
 Automated space
construction bots/agents
 Deep Learning: identify
what things are
(classification)
 Blockchain: secure
automation technology
 Applications: asteroid
mining, terraforming,
radiation-monitoring,
space-based solar power,
debris tracking net

12 Aug 2017
80
Deep Learning
 Agenda
 Deep Learning
 Definition
 Technical details
 Applications
 Deep Qualia: Deep Learning and the Brain
 Smart Network Convergence Theory
 Conclusion

Image Source: [Link]

12 Aug 2017
81
Deep Learning
 Our human future
 Are we doomed?

12 Aug 2017
82
Deep Learning
 Human-machine collaboration
 Team-members excel at different things
 Differently-abled agents in society

12 Aug 2017
83
Deep Learning
 Conceptual Definition:
Deep learning is a computer program that can
identify what something is

Technical Definition:
Deep learning is a class of machine learning
algorithms in the form of a neural network that
uses a cascade of layers (tiers) of processing
units to extract features from data and make
predictive guesses about new data

Source: Extending Jann LeCun, [Link]

12 Aug 2017 on-deep-learning
84
Deep Learning
 Deep Learning Theory
 System is “dumb” (i.e. mechanical)
 “Learns” with big data (lots of input examples) and trial-and-error
guesses to adjust weights and bias to establish key features
 Creates a predictive system to identity new examples
 Same AI argument: big enough data is what makes a
difference (“simple” algorithms run over large data sets)

Input: Big Data (e.g.; Method: Trial-and-error Output: system identifies

many examples) guesses to adjust node new examples
weights
12 Aug 2017
85
Deep Learning
 3 Key Technical Principles of Deep Learning
Sigmoid Function Perceptron Structure Loss Function

Squash values into Core processing unit Reduce combinatoric

probability function (input-processing-output) dimensionality
What
(Sigmoid (0-1); Levers: weights and bias
Tanh ((-1)-1))

Formulate as a logistic “Dumb” system learns by Loss function

Why regression problem for adjusting parameters and optimizes efficiency
greater mathematical checking against outcome of solution
manipulation

12 Aug 2017
86
Deep Learning
 Conclusion
 Next-generation global infrastructure:
Deep Learning Blockchain Networks
merging deep learning systems and
blockchain technology

 Smart Network Convergence Theory:

pushing more complexity and
automation through Internet pipes
 Blockchain Deep Learning nets: Ability to
identify what something is (machine
learning) and securely verify and transact it
(blockchain)

12 Aug 2017
87
Deep Learning
 Resources
 Neural Networks and Deep Learning, Michael Nielsen,
[Link]
 Deep Learning, Ian Goodfellow, Yoshua Bengio, Aaron
Courville, [Link] learning and deep neural nets
 Machine Learning Guide podcast, Tyler Renelle,
[Link]
 notMNIST dataset [Link]
 Metacademy; [Link]; [Link]

[Link]

Distill (visual ML journal)

12 Aug 2017
Source: [Link] [Link]
88
Deep Learning
 Image credit: Nvidia

Deep Learning Explained

The future of Smart Networks

Thank You! Questions?

Melanie Swan
Philosophy & Economic Theory
Purdue University
Boulder Futurists: Solid State Depot Hackspace melanie@[Link]
Boulder CO, August 12, 2017
Slides: [Link]
 Deep Learning Taxonomy
AI (artificial intelligence)

Machine learning Other methods

Supervised learning Unsupervised learning Reinforcement learning

(labeled data: (unlabeled data: pattern
classification) recognition)

Deep learning Shallow learning

(5-20 layers) (1-2 layers)

Neural Nets (NN) Other methods

Bayesian inference
Convolutional nets (images) Support Vector Machines
Recurrent nets (text, speech) Decision trees
K-means clustering
K-nearest neighbor

Source: Machine Learning Guide, 9. Deep Learning;

12 Aug 2017
90
Deep Learning
 Kinds of Deep Learning Systems
What Deep Learning net to choose?

 Supervised algorithms (classify labeled data)

 Image (object) recognition
 Convolutional net (image processing), deep belief
network, recursive neural tensor network
 Text analysis (name recognition, sentiment
analysis)
 Recurrent net (iteration; character level text),
recursive neural tensor network
 Speech recognition
 Recurrent net
 Unsupervised algorithms (find patterns in
unlabeled data)
 Boltzmann machine or autoencoder

Source: Yann LeCun, CVPR 2015 keynote (Computer Vision ), "What's wrong with Deep Learning" [Link]
12 Aug 2017
91
Deep Learning