Deeplearning

FromWikipedia,thefreeencyclopedia

Deeplearning(alsoknownasdeepstructuredlearning,hierarchicallearningordeepmachinelearning)isa
classofmachinelearningalgorithmsthat:[1](pp199200)

[Link]
[Link]
unsupervisedandapplicationsincludepatternanalysis(unsupervised)andclassification(supervised).
arebasedonthe(unsupervised)[Link]
levelfeaturesarederivedfromlowerlevelfeaturestoformahierarchicalrepresentation.
arepartofthebroadermachinelearningfieldoflearningrepresentationsofdata.
learnmultiplelevelsofrepresentationsthatcorrespondtodifferentlevelsofabstractionthelevelsforma
hierarchyofconcepts.

Inasimplecase,theremightbetwosetsofneurons:onesetthatreceivesaninputsignalandonethatsendsan
[Link].
Inadeepnetwork,therearemanylayersbetweentheinputandtheoutput(andthelayersarenotmadeofneurons
butitcanhelptothinkofitthatway),allowingthealgorithmtousemultipleprocessinglayers,composedof
multiplelinearandnonlineartransformations.[2][1][3][4][5][6][7][8][9]

Deeplearningispartofabroaderfamilyofmachinelearningmethodsbasedonlearningrepresentationsofdata.
Anobservation(e.g.,animage)canberepresentedinmanywayssuchasavectorofintensityvaluesperpixel,or
inamoreabstractwayasasetofedges,regionsofparticularshape,[Link]
othersatsimplifyingthelearningtask(e.g.,facerecognitionorfacialexpressionrecognition[10]).Oneofthe
promisesofdeeplearningisreplacinghandcraftedfeatureswithefficientalgorithmsforunsupervisedorsemi
supervisedfeaturelearningandhierarchicalfeatureextraction.[11]

Researchinthisareaattemptstomakebetterrepresentationsandcreatemodelstolearntheserepresentationsfrom
[Link]
basedoninterpretationofinformationprocessingandcommunicationpatternsinanervoussystem,suchasneural
codingwhichattemptstodefinearelationshipbetweenvariousstimuliandassociatedneuronalresponsesinthe
brain.[12]

Variousdeeplearningarchitecturessuchasdeepneuralnetworks,convolutionaldeepneuralnetworks,deepbelief
networksandrecurrentneuralnetworkshavebeenappliedtofieldslikecomputervision,automaticspeech
recognition,naturallanguageprocessing,audiorecognitionandbioinformaticswheretheyhavebeenshownto
producestateoftheartresultsonvarioustasks.

Deeplearninghasbeencharacterizedasabuzzword,orarebrandingofneuralnetworks.[13][14]

Contents
1 Introduction
1.1 Definitions
1.2 Fundamentalconcepts
2 Interpretations
2.1 Universalapproximationtheoreminterpretation
2.2 Probabilisticinterpretation
 3 History
4 Artificialneuralnetworks
5 Deepneuralnetworkarchitectures
5.1 Briefdiscussionofdeepneuralnetworks
5.1.1 Backpropagation
5.1.2 Problemswithdeepneuralnetworks
5.2 Firstdeeplearningnetworksof1965:GMDH
5.3 Convolutionalneuralnetworks
5.4 Neuralhistorycompressor
5.5 Recursiveneuralnetworks
5.6 Longshorttermmemory
5.7 Deepbeliefnetworks
5.8 Convolutionaldeepbeliefnetworks
5.9 Largememorystorageandretrievalneuralnetworks
5.10 DeepBoltzmannmachines
5.11 Stacked(denoising)autoencoders
5.12 Deepstackingnetworks
5.13 Tensordeepstackingnetworks
5.14 SpikeandslabRBMs
5.15 Compoundhierarchicaldeepmodels
5.16 Deepcodingnetworks
5.17 DeepQnetworks
5.18 Networkswithseparatememorystructures
5.18.1 LSTMrelateddifferentiablememorystructures
5.18.2 Semantichashing
5.18.3 NeuralTuringmachines
5.18.4 Memorynetworks
5.18.5 Pointernetworks
5.18.6 Encoderdecodernetworks
6 Otherarchitectures
6.1 Multilayerkernelmachine
7 Applications
7.1 Automaticspeechrecognition
7.2 Imagerecognition
7.3 Naturallanguageprocessing
7.4 Drugdiscoveryandtoxicology
7.5 Customerrelationshipmanagement
7.6 Recommendationsystems
7.7 Biomedicalinformatics
8 Theoriesofthehumanbrain
9 Commercialactivities
10 Criticismandcomment
11 Softwarelibraries
12 Seealso
13 References
14 Externallinks

Introduction
Definitions
Deeplearningisaclassofmachinelearningalgorithmsthat[1](pp199200)

[Link]
[Link]
unsupervisedandapplicationsincludepatternanalysis(unsupervised)andclassification(supervised).
arebasedonthe(unsupervised)[Link]
levelfeaturesarederivedfromlowerlevelfeaturestoformahierarchicalrepresentation.
arepartofthebroadermachinelearningfieldoflearningrepresentationsofdata.
learnmultiplelevelsofrepresentationsthatcorrespondtodifferentlevelsofabstractionthelevelsforma
hierarchyofconcepts.

Thesedefinitionshaveincommon(1)multiplelayersofnonlinearprocessingunitsand(2)thesupervisedor
unsupervisedlearningoffeaturerepresentationsineachlayer,withthelayersformingahierarchyfromlowlevel
tohighlevelfeatures.[1](p200)Thecompositionofalayerofnonlinearprocessingunitsusedinadeeplearning
[Link]
ofanartificialneuralnetworkandsetsofcomplicatedpropositionalformulas.[3]Theymayalsoincludelatent
variablesorganizedlayerwiseindeepgenerativemodelssuchasthenodesinDeepBeliefNetworksandDeep
BoltzmannMachines.

[Link]
layer,thesignalistransformedbyaprocessingunit,likeanartificialneuron,whoseparametersare'learned'
throughtraining.[5](p6)Achainoftransformationsfrominputtooutputisacreditassignmentpath(CAP).CAPs
describepotentiallycausalconnectionsbetweeninputandoutputandmayvaryinlengthforafeedforward
neuralnetwork,thedepthoftheCAPs(thusofthenetwork)isthenumberofhiddenlayersplusone(astheoutput
layerisalsoparameterized),butforrecurrentneuralnetworks,inwhichasignalmaypropagatethroughalayer
morethanonce,[Link]
dividingshallowlearningfromdeeplearning,butmostresearchersinthefieldagreethatdeeplearninghas
multiplenonlinearlayers(CAP>2)andJuergenSchmidhuberconsidersCAP>10tobeverydeeplearning.[5](p7)

Fundamentalconcepts

[Link]
[Link]
[Link]
numbersoflayersandlayersizescanbeusedtoprovidedifferentamountsofabstraction.[4]

Deeplearningexploitsthisideaofhierarchicalexplanatoryfactorswherehigherlevel,moreabstractconceptsare
[Link].
Deeplearninghelpstodisentangletheseabstractionsandpickoutwhichfeaturesareusefulforlearning.[4]

Forsupervisedlearningtasks,deeplearningmethodsobviatefeatureengineering,bytranslatingthedatainto
compactintermediaterepresentationsakintoprincipalcomponents,andderivelayeredstructureswhichremove
redundancyinrepresentation.[1]

[Link]
[Link]
unsupervisedmannerareneuralhistorycompressors[15]anddeepbeliefnetworks.[4][16]

Interpretations
Deepneuralnetworksaregenerallyinterpretedintermsof:Universalapproximationtheorem[17][18][19][20][21]or
Probabilisticinference.[1][3][4][5][16][22]

Universalapproximationtheoreminterpretation

Theuniversalapproximationtheoremconcernsthecapacityoffeedforwardneuralnetworkswithasinglehidden
layeroffinitesizetoapproximatecontinuousfunctions.[17][18][19][20][21]

In1989,thefirstproofwaspublishedbyGeorgeCybenkoforsigmoidactivationfunctions[18]andwasgeneralised
tofeedforwardmultilayerarchitecturesin1991byKurtHornik.[19]

Probabilisticinterpretation

Theprobabilisticinterpretation[22][Link],[1][3][4][5][16][22]
aswellastheoptimizationconceptsoftrainingandtesting,[Link]
specifically,theprobabilisticinterpretationconsiderstheactivationnonlinearityasacumulativedistribution
function.[22][Link]
regularizerinneuralnetworks.[23]

TheprobabilisticinterpretationwasintroducedandpopularizedbyGeoffHinton,YoshuaBengio,YannLeCunand
JuergenSchmidhuber.

History
Thefirstgeneral,workinglearningalgorithmforsuperviseddeepfeedforwardmultilayerperceptronswas
publishedbyIvakhnenkoandLapain1965.[24]A1971paper[25]describedadeepnetworkwith8layerstrainedby
[Link]
implementedinacomputeridentificationsystem"Alpha",[Link]
Learningworkingarchitectures,specificallythosebuiltfromartificialneuralnetworks(ANN),datebacktothe
NeocognitronintroducedbyKunihikoFukushimain1980.[26][Link]
challengewashowtotrainnetworkswithmultiplelayers.In1989,[Link]
standardbackpropagationalgorithm,whichhadbeenaroundasthereversemodeofautomaticdifferentiationsince
1970,[27][28][29][30]toadeepneuralnetworkwiththepurposeofrecognizinghandwrittenZIPcodesonmail.
Despitethesuccessofapplyingthealgorithm,thetimetotrainthenetworkonthisdatasetwasapproximately3
days,makingitimpracticalforgeneraluse.[31]

In1993,JrgenSchmidhuber'sneuralhistorycompressor[15]implementedasanunsupervisedstackofrecurrent
neuralnetworks(RNNs)solveda"VeryDeepLearning"task[5]thatrequiresmorethan1,000subsequentlayersin
anRNNunfoldedintime.[32]In1994,[Link],[Link]
Bisset,publishedaworkwithexperimentalresultsofaseverallayersbooleanneuralnetwork,alsoknownas
weightlessneuralnetwork,composedbytwomodules,aselforganisingfeatureextractionneuralnetworkmodule
followedbyaclassificationneuralnetworkmodule,whichwereindependentlyandsequentiallytrained.[33]In
1995,BrendanFreydemonstratedthatitwaspossibletotrainanetworkcontainingsixfullyconnectedlayersand
severalhundredhiddenunitsusingthewakesleepalgorithm,whichwascodevelopedwithPeterDayanand
GeoffreyHinton.[34]However,[Link],onebeingthe
vanishinggradientproblemanalyzedin1991bySeppHochreiter.[35][36]
Whileby1991suchneuralnetworkswereusedforrecognizingisolated2Dhandwrittendigits,recognizing3D
[Link]
ahumanbraindoesnotuseamonolithic3Dobjectmodel,andin1992theypublishedCresceptron,[37][38][39]a
[Link]
[Link],
Cresceptronautomaticallylearnedanopennumberofunsupervisedfeaturesineachlayer,whereeachfeatureis
[Link]
[Link],nowoftenadoptedbydeepneuralnetworks(e.g.
ImageNettests),wasfirstusedinCresceptrontoreducethepositionresolutionbyafactorof(2x2)to1throughthe
[Link],simplermodelsthatusetaskspecifichandcrafted
featuressuchasGaborfiltersandsupportvectormachines(SVMs)wereapopularchoiceinthe1990sand2000s,
becauseofthecomputationalcostofANNsatthetime,andagreatlackofunderstandingofhowthebrain
autonomouslywiresitsbiologicalnetworks.

Inthelonghistoryofspeechrecognition,bothshallowanddeeplearning(e.g.,recurrentnets)ofartificialneural
networkshavebeenexploredformanyyears.[40][41][42]Butthesemethodsneverwonoverthenonuniform
internalhandcraftingGaussianmixturemodel/HiddenMarkovmodel(GMMHMM)technologybasedon
generativemodelsofspeechtraineddiscriminatively.[43]Anumberofkeydifficultieshavebeenmethodologically
analyzed,includinggradientdiminishing[35]andweaktemporalcorrelationstructureintheneuralpredictive
models.[44][45]Additionaldifficultieswerethelackofbigtrainingdataandweakercomputingpowerintheseearly
[Link],mostspeechrecognitionresearcherswhounderstoodsuchbarriersmovedawayfromneuralnetsto
[Link]
government'sNSAandDARPA,SRIconductedresearchondeepneuralnetworksinspeechandspeaker
[Link],ledbyLarryHeck([Link]
thefirstsignificantsuccesswithdeepneuralnetworksinspeechprocessingasdemonstratedinthe1998NIST
(NationalInstituteofStandardsandTechnology)SpeakerRecognitionevaluation([Link]
[Link])andlaterpublishedinthejournalofSpeechCommunication.[46]WhileSRIestablishedsuccesswithdeep
neuralnetworksinspeakerrecognition,theywereunsuccessfulindemonstratingsimilarsuccessinspeech
[Link]
eachotherandthenwithcolleaguesacrossfourgroups(UniversityofToronto,Microsoft,Google,andIBM)
ignitedarenaissanceofdeepfeedforwardneuralnetworksinspeechrecognition.[47][48][49][50]

Today,however,manyaspectsofspeechrecognitionhavebeentakenoverbyadeeplearningmethodcalledLong
shorttermmemory(LSTM),arecurrentneuralnetworkpublishedbySeppHochreiter&JrgenSchmidhuberin
1997.[51]LSTMRNNsavoidthevanishinggradientproblemandcanlearn"VeryDeepLearning"tasks[5]that
requirememoriesofeventsthathappenedthousandsofdiscretetimestepsago,[Link]
2003,LSTMstartedtobecomecompetitivewithtraditionalspeechrecognizersoncertaintasks.[52]Lateritwas
combinedwithCTC[53]instacksofLSTMRNNs.[54]In2015,Google'sspeechrecognitionreportedlyexperienced
adramaticperformancejumpof49%throughCTCtrainedLSTM,whichisnowavailablethroughGoogleVoice
Searchtoallsmartphoneusers,[55]andhasbecomeashowcaseofdeeplearning.

Theuseoftheexpression"DeepLearning"inthecontextofArtificialNeuralNetworkswasintroducedbyIgor
Aizenbergandcolleaguesin2000.[56]AGoogleNgramchartshowsthattheusageofthetermhasgainedtraction
(actuallyhastakenoff)since2000.[57]In2006,apublicationbyGeoffreyHintonandRuslanSalakhutdinovdrew
additionalattentionbyshowinghowmanylayeredfeedforwardneuralnetworkcouldbeeffectivelypretrained
onelayeratatime,treatingeachlayerinturnasanunsupervisedrestrictedBoltzmannmachine,thenfinetuningit
usingsupervisedbackpropagation.[58]In1992,Schmidhuberhadalreadyimplementedaverysimilarideaforthe
moregeneralcaseofunsuperviseddeephierarchiesofrecurrentneuralnetworks,andalsoexperimentallyshown
itsbenefitsforspeedingupsupervisedlearning.[15][59]

Sinceitsresurgence,deeplearninghasbecomepartofmanystateoftheartsystemsinvariousdisciplines,
particularlycomputervisionandautomaticspeechrecognition(ASR).Resultsoncommonlyusedevaluationsets
suchasTIMIT(ASR)andMNIST(imageclassification),aswellasarangeoflargevocabularyspeechrecognition
tasksareconstantlybeingimprovedwithnewapplicationsofdeeplearning.[47][60][61]Recently,itwasshownthat
deeplearningarchitecturesintheformofconvolutionalneuralnetworkshavebeennearlybestperforming[62][63]
however,thesearemorewidelyusedincomputervisionthaninASR,andmodernlargescalespeechrecognitionis
typicallybasedonCTC[53]forLSTM.[51][55][64][65][66]

Therealimpactofdeeplearninginindustryapparentlybeganintheearly2000s,whenCNNsalreadyprocessedan
estimated10%to20%ofallthecheckswrittenintheUSintheearly2000s,accordingtoYannLeCun.[67]
Industrialapplicationsofdeeplearningtolargescalespeechrecognitionstartedaround2010.Inlate2009,Li
DenginvitedGeoffreyHintontoworkwithhimandcolleaguesatMicrosoftResearchinRedmond,Washingtonto
applydeeplearningtospeechrecognition.Theycoorganizedthe2009NIPSWorkshoponDeepLearningfor
[Link],andthe
possibilitythatthebigcompute,bigdataerawarrantedaserioustryofdeepneuralnets(DNN).Itwasbelieved
thatpretrainingDNNsusinggenerativemodelsofdeepbeliefnets(DBN)wouldovercomethemaindifficulties
ofneuralnetsencounteredinthe1990s.[49]However,earlyintothisresearchatMicrosoft,itwasdiscoveredthat
withoutpretraining,butusinglargeamountsoftrainingdata,andespeciallyDNNsdesignedwithcorresponding
large,contextdependentoutputlayers,producederrorratesdramaticallylowerthanthenstateoftheartGMM
[Link]
byseveralothermajorspeechrecognitionresearchgroups.[47][68]Further,thenatureofrecognitionerrorsproduced
bythetwotypesofsystemswasfoundtobecharacteristicallydifferent,[48][69]offeringtechnicalinsightsintohow
tointegratedeeplearningintotheexistinghighlyefficient,runtimespeechdecodingsystemdeployedbyallmajor
[Link]
describedandanalyzedinrecentbooksandarticles.[1][70][71]

Advancesinhardwarehavealsobeenimportantinenablingtherenewedinterestindeeplearning.In2009,Nvidia
wasinvolvedinwhatwascalledthebigbangofdeeplearning,asdeeplearningneuralnetworkswere
combinedwithNvidiagraphicsprocessingunits(GPUs).[72]Thatyear,theGoogleBrainusedNvidiaGPUsto
createDeepNeuralNetworkscapableofmachinelearning,whereAndrewNgdeterminedthatGPUscould
increasethespeedofdeeplearningsystemsbyabout100times.[73]Inparticular,powerfulgraphicsprocessing
units(GPUs)arewellsuitedforthekindofnumbercrunching,matrix/vectormathinvolvedinmachine
learning.[74][75]GPUshavebeenshowntospeeduptrainingalgorithmsbyordersofmagnitude,bringingrunning
timesofweeksbacktodays.[76][77]Specializedhardwareandalgorithmoptimizationscanalsobeusedforefficient
processingofDNNs.[78]

Artificialneuralnetworks
[Link]
[Link]&TorstenWiesel,who
foundtwotypesofcellsintheprimaryvisualcortex:[Link]
networkscanbeviewedascascadingmodels[37][38][39][79]ofcelltypesinspiredbythesebiologicalobservations.
Fukushima'sNeocognitronintroducedconvolutionalneuralnetworkspartiallytrainedbyunsupervisedlearning
[Link].(1989)appliedsupervisedbackpropagationto
sucharchitectures.[80]Wengetal.(1992)publishedconvolutionalneuralnetworksCresceptron[37][38][39]for3D
objectrecognitionfromimagesofclutteredscenesandsegmentationofsuchobjectsfromimages.

[Link]
poolingappearedtobefirstproposedbyCresceptron[37][38]toenablethenetworktotoleratesmalltolarge
deformationinahierarchicalway,[Link],butdoesnotguarantee,shift
invarianceatthepixellevel.[39]

Withtheadventofthebackpropagationalgorithmbasedonautomatic
differentiation,[27][29][30][81][82][83][84][85][86][87]manyresearcherstriedtotrainsuperviseddeepartificialneural
networksfromscratch,[Link]'sdiplomathesisof1991[35][36]formally
identifiedthereasonforthisfailureasthevanishinggradientproblem,whichaffectsmanylayeredfeedforward
[Link]
feedforwardnetworks,whereanewlayeriscreatedforeachtimestepofaninputsequenceprocessedbythe
[Link],theyshrinkexponentiallywiththenumberoflayers,impeding
thetuningofneuronweightswhichisbasedonthoseerrors.

Toovercomethisproblem,[Link]'smultilevelhierarchyof
networks(1992)pretrainedonelevelatatimebyunsupervisedlearning,finetunedbybackpropagation.[15]Here
eachlevellearnsacompressedrepresentationoftheobservationsthatisfedtothenextlevel.

Anothermethodisthelongshorttermmemory(LSTM)networkofHochreiter&Schmidhuber(1997).[51]In
2009,deepmultidimensionalLSTMnetworkswonthreeICDAR2009competitionsinconnectedhandwriting
recognition,withoutanypriorknowledgeaboutthethreelanguagestobelearned.[88][89]

SvenBehnkein2003reliedonlyonthesignofthegradient(Rprop)whentraininghisNeuralAbstraction
Pyramid[90]tosolveproblemslikeimagereconstructionandfacelocalization.

Othermethodsalsouseunsupervisedpretrainingtostructureaneuralnetwork,makingitfirstlearngenerally
[Link]
[Link].(2006)involveslearningthedistributionofahighlevelrepresentationusing
[Link](Smolensky,
1986[91])[Link]
boundoftheloglikelihoodofthedata,thusimprovingthemodel,[Link]
layershavebeenlearned,thedeeparchitecturemaybeusedasagenerativemodelbyreproducingthedatawhen
samplingdownthemodel(an"ancestralpass")fromthetoplevelfeatureactivations.[92]Hintonreportsthathis
modelsareeffectivefeatureextractorsoverhighdimensional,structureddata.[93]

In2012,theGoogleBrainteamledbyAndrewNgandJeffDeancreatedaneuralnetworkthatlearnedto
recognizehigherlevelconcepts,suchascats,onlyfromwatchingunlabeledimagestakenfromYouTube
videos.[94][95]

Othermethodsrelyonthesheerprocessingpowerofmoderncomputers,inparticular,GPUs.In2010,DanCiresan
andcolleagues[76]inJrgenSchmidhuber'sgroupattheSwissAILabIDSIAshowedthatdespitetheabove
mentioned"vanishinggradientproblem,"thesuperiorprocessingpowerofGPUsmakesplainbackpropagation
[Link]
learningtechniquesontheold,famousMNISThandwrittendigitsproblemofYannLeCunandcolleaguesatNYU.
Ataboutthesametime,inlate2009,deeplearningfeedforwardnetworksmadeinroadsintospeechrecognition,as
[Link]
MicrosoftResearchandUniversityofTorontoresearchersdemonstratedbymid2010inRedmondthatdeepneural
networksinterfacedwithahiddenMarkovmodelwithcontextdependentstatesthatdefinetheneuralnetwork
[Link]
samedeepneuralnetmodelwasshowntoscaleuptoSwitchboardtasksaboutoneyearlateratMicrosoftResearch
[Link],in2007,LSTM[51]trainedbyCTC[53]startedtogetexcellentresultsincertainapplications.[54]
Thismethodisnowwidelyused,forexample,inGoogle'sgreatlyimprovedspeechrecognitionforallsmartphone
users.[55]

Asof2011,thestateoftheartindeeplearningfeedforwardnetworksalternatesconvolutionallayersandmax
poolinglayers,[96][97]toppedbyseveralfullyconnectedorsparselyconnectedlayerfollowedbyafinal
[Link].Since2011,GPUbased
implementations[96]ofthisapproachwonmanypatternrecognitioncontests,includingtheIJCNN2011Traffic
SignRecognitionCompetition,[98]theISBI2012SegmentationofneuronalstructuresinEMstackschallenge,[99]
theImageNetCompetition,[100]andothers.

Suchsuperviseddeeplearningmethodsalsowerethefirstartificialpatternrecognizerstoachievehuman
competitiveperformanceoncertaintasks.[101]

ToovercomethebarriersofweakAIrepresentedbydeeplearning,itisnecessarytogobeyonddeeplearning
architectures,becausebiologicalbrainsusebothshallowanddeepcircuitsasreportedbybrainanatomy[102]
[Link][103]arguedthatthebrainselfwireslargelyaccordingtosignal
statisticsand,therefore,[Link]
guaranteeshiftinvariancetodealwithsmallandlargenaturalobjectsinlargeclutteredscenes,onlywhen
invarianceextendedbeyondshift,toallANNlearnedconcepts,suchaslocation,type(objectclasslabel),scale,
[Link](DNs)[104]whoseembodimentsareWhereWhat
Networks,WWN1(2008)[105]throughWWN7(2013).[106]

Deepneuralnetworkarchitectures
[Link]
[Link],because
[Link],andnewarchitectures,
variants,oralgorithmsappeareveryfewweeks.

Briefdiscussionofdeepneuralnetworks

Adeepneuralnetwork(DNN)isanartificialneuralnetwork(ANN)withmultiplehiddenlayersofunitsbetween
theinputandoutputlayers.[3][5]SimilartoshallowANNs,DNNscanmodelcomplexnonlinearrelationships.
DNNarchitectures,e.g.,forobjectdetectionandparsing,generatecompositionalmodelswheretheobjectis
expressedasalayeredcompositionofimageprimitives.[107]Theextralayersenablecompositionoffeaturesfrom
lowerlayers,givingthepotentialofmodelingcomplexdatawithfewerunitsthanasimilarlyperformingshallow
network.[3]

DNNsaretypicallydesignedasfeedforwardnetworks,butresearchhasverysuccessfullyappliedrecurrentneural
networks,especiallyLSTM,[51][108]forapplicationssuchaslanguagemodeling.[109][110][111][112][113]Convolutional
deepneuralnetworks(CNNs)areusedincomputervisionwheretheirsuccessiswelldocumented.[114]CNNsalso
havebeenappliedtoacousticmodelingforautomaticspeechrecognition(ASR),wheretheyhaveshownsuccess
overpreviousmodels.[63]Forsimplicity,alookattrainingDNNsisgivenhere.

Backpropagation

[Link]
sources,[5][8][87][115]basicsofcontinuousbackpropagationwerederivedinthecontextofcontroltheorybyHenryJ.
Kelley[82]in1960andbyArthurE.Brysonin1961,[83]usingprinciplesofdynamicprogramming.In1962,Stuart
Dreyfuspublishedasimplerderivationbasedonlyonthechainrule.[84]Vapnikcitesreference[116]inhisbookon
[Link]
optimizationmethodin1969.[117][118]In1970,SeppoLinnainmaafinallypublishedthegeneralmethodfor
automaticdifferentiation(AD)ofdiscreteconnectednetworksofnesteddifferentiablefunctions.[27][119]This
correspondstothemodernversionofbackpropagationwhichisefficientevenwhenthenetworksare
sparse.[5][8][28][81]In1973,StuartDreyfususedbackpropagationtoadaptparametersofcontrollersinproportionto
errorgradients.[85]In1974,PaulWerbosmentionedthepossibilityofapplyingthisprincipletoartificialneural
networks,[120]andin1982,heappliedLinnainmaa'sADmethodtoneuralnetworksinthewaythatiswidelyused
today.[8][30]In1986,[Link],[Link]
experimentsthatthismethodcangenerateusefulinternalrepresentationsofincomingdatainhiddenlayersof
neuralnetworks.[86]In1993,[Link][5]towinaninternationalpatternrecognitioncontest
throughbackpropagation.[121]

Theweightupdatesofbackpropagationcanbedoneviastochasticgradientdescentusingthefollowingequation:

Here, isthelearningrate, isthecostfunctionand [Link]

dependsonfactorssuchasthelearningtype(supervised,unsupervised,reinforcement,etc.)andtheactivation
[Link],whenperformingsupervisedlearningonamulticlassclassificationproblem,common
choicesfortheactivationfunctionandcostfunctionarethesoftmaxfunctionandcrossentropyfunction,

[Link] where representstheclassprobability

(outputoftheunit )and and representthetotalinputtounits and [Link]

entropyisdefinedas where representsthetargetprobabilityforoutputunit and is

theprobabilityoutputfor afterapplyingtheactivationfunction.[122]

[Link]
[Link]
[Link]
[Link]
multiplelayers,[Link]
convolutional,[Link]
networkistrainedtominimizeL2errorforpredictingthemaskrangingovertheentiretrainingsetcontaining
boundingboxesrepresentedasmasks.

Problemswithdeepneuralnetworks
AswithANNs,[Link]
andcomputationtime.

DNNsarepronetooverfittingbecauseoftheaddedlayersofabstraction,whichallowthemtomodelrare
[Link]'sunitpruning[25]orweightdecay(
regularization)orsparsity( regularization)canbeappliedduringtrainingtohelpcombatoverfitting.[123]A
[Link],somenumberofunits
[Link]
occurinthetrainingdata.[124]

Thedominantmethodfortrainingthesestructureshasbeenerrorcorrectiontraining(suchasbackpropagationwith
gradientdescent)duetoitseaseofimplementationanditstendencytoconvergetobetterlocaloptimathanother
[Link],thesemethodscanbecomputationallyexpensive,[Link]
manytrainingparameterstobeconsideredwithaDNN,suchasthesize(numberoflayersandnumberofunitsper
layer),[Link]
[Link]'tricks'suchasusingminibatching
(computingthegradientonseveraltrainingexamplesatonceratherthanindividualexamples)[125]havebeen
[Link]
training,duetothematrixandvectorcomputationsrequiredbeingwellsuitedforGPUs.[5]Radicalalternativesto
backpropsuchasExtremeLearningMachines,[126]"Noprop"networks,[127]trainingwithoutbacktracking,[128]
"weightless"networks,[129]andnonconnectionistneuralnetworksaregainingattention.

Firstdeeplearningnetworksof1965:GMDH

Accordingtoahistoricsurvey,[5]thefirstfunctionalDeepLearningnetworkswithmanylayerswerepublishedby
AlexeyGrigorevichIvakhnenkoandV.G.Lapain1965.[24][130]ThelearningalgorithmwascalledtheGroup
MethodofDataHandlingorGMDH.[131]GMDHfeaturesfullyautomaticstructuralandparametricoptimization
[Link]
[Link]'s1971paper[25]describesthelearningofadeepfeedforwardmultilayer
perceptronwitheightlayers,[Link]
grownlayerbylayer,[Link]
detectedusingavalidationset,[Link]
[Link].[132]

Convolutionalneuralnetworks

CNNshavebecomethemethodofchoiceforprocessingvisualandothertwodimensionaldata.[31][67]ACNNis
composedofoneormoreconvolutionallayerswithfullyconnectedlayers(matchingthoseintypicalartificial
neuralnetworks)[Link],maxpooling[38]isoftenusedin
Fukushima'sconvolutionalarchitecture.[26]ThisarchitectureallowsCNNstotakeadvantageofthe2Dstructureof
[Link],convolutionalneuralnetworkshaveshownsuperiorresults
[Link]
trainthanotherregular,deep,feedforwardneuralnetworksandhavemanyfewerparameterstoestimate,making
themahighlyattractivearchitecturetouse.[133]ExamplesofapplicationsinComputerVisioninclude
DeepDream.[134]SeethemainarticleonConvolutionalneuralnetworksfornumerousadditionalreferences.

Neuralhistorycompressor
Thevanishinggradientproblem[35]ofautomaticdifferentiationorbackpropagationinneuralnetworkswas
partiallyovercomein1992byanearlygenerativemodelcalledtheneuralhistorycompressor,implementedasan
unsupervisedstackofrecurrentneuralnetworks(RNNs).[15]TheRNNattheinputlevellearnstopredictitsnext
[Link]
[Link]
[Link]
[Link]
minimisesthedescriptionlengthorthenegativelogarithmoftheprobabilityofthedata.[8]Ifthereisalotof
learnablepredictabilityintheincomingdatasequence,thenthehighestlevelRNNcanusesupervisedlearningto
easilyclassifyevendeepsequenceswithverylongtimeintervalsbetweenimportantevents.In1993,suchasystem
alreadysolveda"VeryDeepLearning"taskthatrequiresmorethan1000subsequentlayersinanRNNunfoldedin
time.[32]

ItisalsopossibletodistilltheentireRNNhierarchyintoonlytwoRNNscalledthe"conscious"chunker(higher
level)andthe"subconscious"automatizer(lowerlevel).[15]Oncethechunkerhaslearnedtopredictandcompress
inputsthatarestillunpredictablebytheautomatizer,theautomatizerisforcedinthenextlearningphasetopredict
[Link]
easyfortheautomatizertolearnappropriate,[Link]
turnhelpstheautomatizertomakemanyofitsonceunpredictableinputspredictable,suchthatthechunkercan
focusontheremainingstillunpredictableevents,tocompressthedataevenfurther.[15]

Recursiveneuralnetworks

Arecursiveneuralnetwork[135]iscreatedbyapplyingthesamesetofweightsrecursivelyoveradifferentiable
graphlikestructure,[Link]
thereversemodeofautomaticdifferentiation.[27][81]Theywereintroducedtolearndistributedrepresentationsof
structure,[Link]
[Link].[136]
TheRecursiveNeuralTensorNetworkusesatensorbasedcompositionfunctionforallnodesinthetree.[137]

Longshorttermmemory

NumerousresearchersnowusevariantsofadeeplearningRNNcalledtheLongshorttermmemory(LSTM)
networkpublishedbyHochreiter&Schmidhuberin1997.[51]ItisasystemthatunliketraditionalRNNsdoesn't
[Link].[108]
LSTMRNNspreventbackpropagatederrorsfromvanishingorexploding.[35]Insteaderrorscanflowbackwards
[Link],LSTMcanlearn"Very
DeepLearning"tasks[5]thatrequirememoriesofeventsthathappenedthousandsorevenmillionsofdiscretetime
[Link].[138]LSTMworksevenwhentherearelong
delays,anditcanhandlesignalsthathaveamixoflowandhighfrequencycomponents.

Today,manyapplicationsusestacksofLSTMRNNs[139]andtrainthembyConnectionistTemporalClassification
(CTC)[53]tofindanRNNweightmatrixthatmaximizestheprobabilityofthelabelsequencesinatrainingset,
[Link].In2009,CTCtrained
LSTMwasthefirstRNNtowinpatternrecognitioncontests,whenitwonseveralcompetitionsinconnected
handwritingrecognition.[5][88]Alreadyin2003,LSTMstartedtobecomecompetitivewithtraditionalspeech
recognizersoncertaintasks.[52]In2007,thecombinationwithCTCachievedfirstgoodresultsonspeechdata.[54]
Sincethen,thisapproachhasrevolutionisedspeechrecognition.In2014,theChinesesearchgiantBaiduused
CTCtrainedRNNstobreaktheSwitchboardHub5'00speechrecognitionbenchmark,withoutusingany
traditionalspeechprocessingmethods.[140]LSTMalsoimprovedlargevocabularyspeechrecognition,[64][65]text
tospeechsynthesis,[141]alsoforGoogleAndroid,[8][66]andphotorealtalkingheads.[142]In2015,Google'sspeech
recognitionreportedlyexperiencedadramaticperformancejumpof49%throughCTCtrainedLSTM,whichis
nowavailablethroughGoogleVoicetobillionsofsmartphoneusers.[55]

[Link]
onHMMsandsimilarconcepts,LSTMcanlearntorecognisecontextsensitivelanguages.[109]LSTMimproved
machinetranslation,[110]Languagemodeling[111]andMultilingualLanguageProcessing.[112]LSTMcombined
withConvolutionalNeuralNetworks(CNNs)alsoimprovedautomaticimagecaptioning[143]andaplethoraof
otherapplications.

Deepbeliefnetworks

Adeepbeliefnetwork(DBN)isaprobabilistic,generativemodelmadeup
[Link]
simplelearningmodulesthatmakeupeachlayer.[16]

ADBNcanbeusedtogenerativelypretrainaDNNbyusingthelearned
[Link]
discriminativealgorithmscanthenbeappliedforfinetuningofthese
[Link]
available,becausepoorlyinitializedweightscansignificantlyhinderthe
learnedmodel'[Link]
theweightspacethatisclosertotheoptimalweightsthanarerandomly
[Link]
convergenceofthefinetuningphase.[144]
ArestrictedBoltzmannmachine
ADBNcanbeefficientlytrainedinanunsupervised,layerbylayer (RBM)withfullyconnectedvisible
manner,wherethelayersaretypicallymadeofrestrictedBoltzmann [Link]
machines(RBM).AnRBMisanundirected,generativeenergybased hiddenhiddenorvisiblevisible
modelwitha"visible"inputlayerandahiddenlayer,andconnections connections.
[Link]
proposedbyGeoffreyHintonforusewithtraining"ProductofExpert"
modelsiscalledcontrastivedivergence(CD).[145]CDprovidesanapproximationtothemaximumlikelihood
methodthatwouldideallybeappliedforlearningtheweightsoftheRBM.[125][146]IntrainingasingleRBM,
weightupdatesareperformedwithgradientascentviathefollowingequation:

.Here, istheprobabilityofavisiblevector,whichisgivenby

. isthepartitionfunction(usedfornormalizing)and istheenergyfunction

[Link]"desirable"configuration.

Thegradient hasthesimpleform where representaverageswith

respecttodistribution .Theissuearisesinsampling becausethisrequiresrunningalternatingGibbs

[Link] steps(valuesof
haveempiricallybeenshowntoperformwell).After steps,thedataaresampledandthatsampleisusedinplace
of .TheCDprocedureworksasfollows:[125]

[Link].
[Link]: . isthe

sigmoidfunctionand isthebiasof .
[Link]: . isthe

biasof .Thisiscalledthe"reconstruction"step.
[Link]
2.
[Link]: .

OnceanRBMistrained,anotherRBMis"stacked"atopit,takingitsinputfromthefinalalreadytrainedlayer.
Thenewvisiblelayerisinitializedtoatrainingvector,andvaluesfortheunitsinthealreadytrainedlayersare
[Link]
processisrepeateduntilsomedesiredstoppingcriterionismet.[3]

AlthoughtheapproximationofCDtomaximumlikelihoodisverycrude(hasbeenshowntonotfollowthe
gradientofanyfunction),ithasbeenempiricallyshowntobeeffectiveintrainingdeeparchitectures.[125]

Convolutionaldeepbeliefnetworks

Arecentachievementindeeplearningistheuseofconvolutionaldeepbeliefnetworks(CDBN).CDBNshave
structureverysimilartoaconvolutionalneuralnetworksandaretrainedsimilartodeepbeliefnetworks.
Therefore,theyexploitthe2Dstructureofimages,likeCNNsdo,andmakeuseofpretraininglikedeepbelief
[Link].
Recently,manybenchmarkresultsonstandardimagedatasetslikeCIFAR[147]havebeenobtainedusing
CDBNs.[148]

Largememorystorageandretrievalneuralnetworks

Largememorystorageandretrievalneuralnetworks(LAMSTAR)[149][150]arefastdeeplearningneuralnetworks
[Link],stochastic,logic,non
stationary,[Link].

[Link]
isprovidedbyHebbianlinkweights([Link],2013[151]),whichservetointegratethevarious
andusuallydifferentfilters(preprocessingfunctions)intoitsmanylayersandtodynamicallyrankthesignificance
[Link]
learningwhichintegratesoutputsvariouspreprocessors(cochlea,retina,etc.)andcortexes(auditory,visual,etc.)
[Link],correlationandbyits
abilitytocopewithincompletedata,or"lost"[Link],itisfully
[Link]
redundancy,andfacilitatetherankingoflayers,offiltersorofindividualneuronsrelativetoatask.
LAMSTARhasbeenappliedtomanymedical[152][153][154]andfinancialpredictions(seeGraupe,2013[155]Section
9C),adaptivefilteringofnoisyspeechinunknownnoise,[156]stillimagerecognition[157](Graupe,2013[158]
Section9D),videoimagerecognition,[159]softwaresecurity,[160]adaptivecontrolofnonlinearsystems,[161]and
[Link]
networkbasedonReLUfunctionfiltersandmaxpooling,in20comparativestudies.[162]

Theseapplicationsdemonstratedelvingintoaspectsofthedatathatarehiddenfromshallowlearningnetworksor
evenfromthehumansenses(eye,ear),suchasinthecasesofpredictingonsetofsleepapneaevents,[153]ofan
electrocardiogramofafetusasrecordedfromskinsurfaceelectrodesplacedonthemother'sabdomenearlyin
pregnancy,[154]offinancialprediction(Section9CinGraupe,2013),[149]orinblindfilteringofnoisyspeech.[156]

LAMSTARwasproposedin1996(AU.S.Patent5,920,852A([Link]
wasfurtherdevelopedbyDGraupeandHKordylewski19972002.[163][164][165]Amodifiedversion,knownas
LAMSTAR2,wasdevelopedbyNCSchneiderandDGraupein2008.[166][167]

DeepBoltzmannmachines

AdeepBoltzmannmachine(DBM)isatypeofbinarypairwiseMarkovrandomfield(undirectedprobabilistic
graphicalmodel)[Link]
[Link] ,andaseriesoflayersofhiddenunits
.Thereisnoconnectionbetweenunitsofthesamelayer
(likeRBM).FortheDBM,theprobabilityassignedtovectoris

where arethesetofhiddenunits,and arethemodel

parameters,[Link] and thenetwork
isthewellknownrestrictedBoltzmannmachine. [168][Link]
contrast,inadeepbeliefnetwork(DBN)onlythetoptwolayersformarestrictedBoltzmannmachine(whichisan
undirectedgraphicalmodel),butlowerlayersformadirectedgenerativemodel.

LikeDBNs,DBMscanlearncomplexandabstractinternalrepresentationsoftheinputintaskssuchasobjector
speechrecognition,usinglimitedlabeleddatatofinetunetherepresentationsbuiltusingalargesupplyof
[Link],unlikeDBNsanddeepconvolutionalneuralnetworks,theyadoptthe
inferenceandtrainingprocedureinbothdirections,bottomupandtopdownpass,whichallowtheDBMstobetter
unveiltherepresentationsoftheambiguousandcomplexinputstructures.[169][170]

However,[Link]
learningisintractableforDBMs,[Link]
usemeanfieldinferencetoestimatedatadependentexpectations,andapproximationtheexpectedsufficient
statisticsofthemodelbyusingMarkovchainMonteCarlo(MCMC).[168]Thisapproximateinference,whichmust
bedoneforeachtestinput,[Link]
jointoptimizationimpracticalforlargedatasets,andseriouslyrestrictstheuseofDBMsfortaskssuchasfeature
representation.[171]

Stacked(denoising)autoencoders
[Link],foraclassifier,agood
representationcanbedefinedasonethatwillyieldabetterperformingclassifier.

Anencoderisadeterministicmapping thattransformsaninputvectorxintohiddenrepresentationy,where
, istheweightmatrixandbisanoffsetvector(bias).Adecodermapsbackthehidden
representationytothereconstructedinputzvia .Thewholeprocessofautoencodingistocomparethis
reconstructedinputtotheoriginalandtrytominimizethiserrortomakethereconstructedvalueascloseas
possibletotheoriginal.

Instackeddenoisingautoencoders,thepartiallycorruptedoutputiscleaned(denoised).Thisideawasintroduced
in2010byVincentetal.[172]withaspecificapproachtogoodrepresentation,agoodrepresentationisonethatcan
beobtainedrobustlyfromacorruptedinputandthatwillbeusefulforrecoveringthecorrespondingcleaninput.
Implicitinthisdefinitionarethefollowingideas:

Thehigherlevelrepresentationsarerelativelystableandrobusttoinputcorruption
Itisnecessarytoextractfeaturesthatareusefulforrepresentationoftheinputdistribution.

Thealgorithmconsistsofmultiplestepsstartsbyastochasticmappingof to through ,thisisthe

[Link] passesthroughabasicautoencoderprocessandismappedtoahidden
representation .Fromthishiddenrepresentation,wecanreconstruct .Inthe
laststage,aminimizationalgorithmrunsinordertohavezascloseaspossibletouncorruptedinput .The
reconstructionerror mightbeeitherthecrossentropylosswithanaffinesigmoiddecoder,orthe
squarederrorlosswithanaffinedecoder.[172]

Inordertomakeadeeparchitecture,autoencodersstackoneontopofanother.[173]Oncetheencodingfunction
ofthefirstdenoisingautoencoderislearnedandusedtouncorrupttheinput(corruptedinput),wecantrainthe
secondlevel.[172]

Oncethestackedautoencoderistrained,itsoutputcanbeusedastheinputtoasupervisedlearningalgorithmsuch
assupportvectormachineclassifieroramulticlasslogisticregression.[172]

Deepstackingnetworks

Onedeeparchitecturebasedonahierarchyofblocksofsimplifiedneuralnetworkmodulesisadeepconvex
network,introducedin2011.[174]Here,theweightslearningproblemisformulatedasaconvexoptimization
[Link](DSN),[175]
emphasizingthemechanism'ssimilaritytostackedgeneralization.[176]EachDSNblockisasimplemodulethatis
easytotrainbyitselfinasupervisedfashionwithoutbackpropagationfortheentireblocks.[177]

AsdesignedbyDengandDong,[174]eachblockconsistsofasimplifiedmultilayerperceptron(MLP)witha
[Link],andtheoutputlayerhaslinearunits.
ConnectionsbetweentheselayersarerepresentedbyweightmatrixUinputtohiddenlayerconnectionshave
[Link],andtheinputdatavectorsxformthecolumnsof
[Link] .Modulesaretrainedinorder,solowerlayerweightsW
[Link]
thesamefinallabelclassy,anditsestimateisconcatenatedwithoriginalinputXtoformtheexpandedinputfor
[Link],theinputtothefirstblockcontainstheoriginaldataonly,whiledownstreamblocks'input
[Link]
networkcanbeformulatedasaconvexoptimizationproblem:
whichhasaclosedformsolution.

Unlikeotherdeeparchitectures,suchasDBNs,thegoalisnottodiscoverthetransformedfeaturerepresentation.
Thestructureofthehierarchyofthiskindofarchitecturemakesparallellearningstraightforward,asabatchmode
[Link],DSNsperformbetterthanconventionalDBN.[175]

Tensordeepstackingnetworks

Thisarchitectureisanextensionofdeepstackingnetworks(DSN).ItimprovesonDSNintwoimportantways:it
useshigherorderinformationfromcovariancestatistics,andittransformsthenonconvexproblemofalower
layertoaconvexsubproblemofanupperlayer.[178]TDSNsusecovariancestatisticsofthedatabyusinga
bilinearmappingfromeachoftwodistinctsetsofhiddenunitsinthesamelayertopredictions,viaathirdorder
tensor.

WhileparallelizationandscalabilityarenotconsideredseriouslyinconventionalDNNs,[179][180][181]alllearning
forDSNsandTDSNsisdoneinbatchmode,toallowparallelizationonaclusterofCPUorGPUnodes.[174][175]
Parallelizationallowsscalingthedesigntolarger(deeper)architecturesanddatasets.

Thebasicarchitectureissuitablefordiversetaskssuchasclassificationandregression.

SpikeandslabRBMs

Theneedfordeeplearningwithrealvaluedinputs,asinGaussianrestrictedBoltzmannmachines,motivatesthe
spikeandslabRBM(ssRBMs),whichmodelscontinuousvaluedinputswithstrictlybinarylatentvariables.[182]
SimilartobasicRBMsanditsvariants,aspikeandslabRBMisabipartitegraph,whilelikeGRBMs,thevisible
units(input)[Link],whereeachhiddenunithasabinaryspike
[Link],whileaslabisadensityover
continuousdomain[183][183][Link].[184]

AnextensionofssRBMcalledssRBMprovidesextramodelingcapacityusingadditionaltermsintheenergy
[Link]
marginalizingouttheslabvariablesgivenanobservation.

Compoundhierarchicaldeepmodels

[Link]
belearnedusingdeeparchitecturessuchasDBNs,[92]DBMs,[169]deepautoencoders,[185]convolutional
variants,[186][187]ssRBMs,[183]deepcodingnetworks,[188]DBNswithsparsefeaturelearning,[189]recursiveneural
networks,[190]conditionalDBNs,[191]denoisingautoencoders.[192]Thisprovidesabetterrepresentation,allowing
[Link],thesearchitecturesarepoor
atlearningnovelclasseswithfewexamples,becauseallnetworkunitsareinvolvedinrepresentingtheinput(a
distributedrepresentation)andmustbeadjustedtogether(highdegreeoffreedom).Limitingthedegreeof
freedomreducesthenumberofparameterstolearn,facilitatinglearningofnewclassesfromfewexamples.
HierarchicalBayesian(HB)modelsallowlearningfromfewexamples,forexample[193][194][195][196][197]for
computervision,statistics,andcognitivescience.
[Link]
DBMarchitecture,ahierarchicalDirichletprocess(HDP)asahierarchicalmodel,incorporatedwithDBM
[Link],generalizedfromabstractconceptsflowingthroughthelayersofthe
model,[Link]
learnedjointlybymaximizingajointlogprobabilityscore.[198]

InaDBMwiththreehiddenlayers,theprobabilityofavisibleinputis:

where isthesetofhiddenunits,and arethemodelparameters,

representingvisiblehiddenandhiddenhiddensymmetricinteractionterms.

AfteraDBMmodelislearned,wehaveanundirectedmodelthatdefinesthejointdistribution .
Onewaytoexpresswhathasbeenlearnedistheconditionalmodel andapriorterm .

Here representsaconditionalDBMmodel,whichcanbeviewedasatwolayerDBMbutwith
biastermsgivenbythestatesof :

Deepcodingnetworks

[Link]
(DPCN)isapredictivecodingschemewheretopdowninformationisusedtoempiricallyadjustthepriorsneeded
[Link]
[Link],apoolingstrategy
[Link],andaretrained
[Link]
anylayeronlydependontheprecedingandsucceedinglayers.

Deeppredictivecodingnetwork(DPCN)[199]predictstherepresentationofthelayer,byusingatopdown
approachusingtheinformationinupperlayerandtemporaldependenciesfromthepreviousstates.

DPCNscanbeextendedtoformaconvolutionalnetwork.[199]

DeepQnetworks

AdeepQnetwork(DQN)isatypeofdeeplearningmodeldevelopedatGoogleDeepMindwhichcombinesa
deepconvolutionalneuralnetworkwithQlearning,[Link]
learningagents,[Link]
in2014,withapaperpublishedinFebruary2015inNature[200]Theapplicationdiscussedinthispaperislimitedto
Atari2600gaming,[Link],muchbeforethiswork,therehad
beenanumberofreinforcementlearningmodelsthatapplydeeplearningapproaches(e.g.,[201]).

Networkswithseparatememorystructures
Integratingexternalmemorywithartificialneuralnetworksdatestoearlyresearchindistributed
representations[202]andTeuvoKohonen'[Link],insparsedistributedmemoryor
hierarchicaltemporalmemory,thepatternsencodedbyneuralnetworksareusedasaddressesforcontent
addressablememory,with"neurons"[Link],theearly
controllersofsuchmemorieswerenotdifferentiable.

LSTMrelateddifferentiablememorystructures

Apartfromlongshorttermmemory(LSTM),otherapproachesofthe1990sand2000salsoaddeddifferentiable
[Link]:

Differentiablepushandpopactionsforalternativememorynetworkscalledneuralstackmachines[203][204]
Memorynetworkswherethecontrolnetwork'sexternaldifferentiablestorageisinthefastweightsof
anothernetwork[205]
LSTM"forgetgates"[206]
Selfreferentialrecurrentneuralnetworks(RNNs)withspecialoutputunitsforaddressingandrapidly
manipulatingeachoftheRNN'sownweightsindifferentiablefashion(internalstorage)[207][208]
Learningtotransducewithunboundedmemory[209]

Semantichashing

Approacheswhichrepresentpreviousexperiencesdirectlyanduseasimilarexperiencetoformalocalmodelare
oftencallednearestneighbourorknearestneighborsmethods.[210]Morerecently,deeplearningwasshowntobe
usefulinsemantichashing[211]whereadeepgraphicalmodelthewordcountvectors[212]obtainedfromalargeset
[Link]
[Link]
[Link]
whichoperateson1000bitaddresses,semantichashingworkson32or64bitaddressesfoundinaconventional
computerarchitecture.

NeuralTuringmachines

NeuralTuringmachines,[213]developedbyGoogleDeepMind,coupleLSTMnetworkstoexternalmemory
resources,[Link]
machinebutisdifferentiableendtoend,[Link]
resultsdemonstratethatneuralTuringmachinescaninfersimplealgorithmssuchascopying,sorting,and
associativerecallfrominputandoutputexamples.

Memorynetworks

Memorynetworks[214][215]areanotherextensiontoneuralnetworksincorporatinglongtermmemory,whichwas
[Link],withthegoalof
[Link](QA)wherethelong
termmemoryeffectivelyactsasa(dynamic)knowledgebase,andtheoutputisatextualresponse.[216]

Pointernetworks
Deepneuralnetworkscanbepotentiallyimprovediftheygetdeeperandhavefewerparameters,whilemaintaining
[Link](e.g.1millionlayerdeep)neuralnetworksmightnotbepractically
feasible,CPUlikearchitecturessuchaspointernetworks[217]andneuralrandomaccessmachines[218]developed
byGoogleBrainresearchersovercomethislimitationbyusingexternalrandomaccessmemoryaswellasadding
othercomponentsthattypicallybelongtoacomputerarchitecturesuchasregisters,[Link]
[Link],themodelisfully
[Link],thesizeoftheir
shorttermmemory,andthenumberofparameterscanbealteredindependentlyunlikemodelslikeLongshort
termmemory,whosenumberofparametersgrowsquadraticallywithmemorysize.

Encoderdecodernetworks

Anencoderdecoderframeworkisaframeworkbasedonneuralnetworksthataimstomaphighlystructuredinput
[Link],[219][220][221]wherethe
[Link],anLSTMrecurrentneuralnetwork
(RNN)orconvolutionalneuralnetwork(CNN)wasusedasanencodertosummarizeasourcesentence,andthe
summarywasdecodedusingaconditionalrecurrentneuralnetworklanguagemodeltoproducethetranslation.[222]
Allthesesystemshavethesamebuildingblocks:gatedRNNsandCNNs,andtrainedattentionmechanisms.

Otherarchitectures
Multilayerkernelmachine

Multilayerkernelmachines(MKM)asintroducedin[223]areawayoflearninghighlynonlinearfunctionsby
[Link](KPCA),
in[224]asmethodforunsupervisedgreedylayerwisepretrainingstepofthedeeplearningarchitecture.

Layer thlearnstherepresentationofthepreviouslayer ,extractingthe principalcomponent(PC)ofthe

projectionlayer [Link]
theupdatedrepresentationineachlayer,asupervisedstrategyisproposedtoselectthebestinformativefeatures
[Link]:

rankthe featuresaccordingtotheirmutualinformationwiththeclasslabels
fordifferentvaluesofKand ,computetheclassificationerrorrateofaKnearestneighbor
(KNN)classifierusingonlythe mostinformativefeaturesonavalidationset
thevalueof withwhichtheclassifierhasreachedthelowesterrorratedeterminesthenumberoffeatures
toretain.

TherearesomedrawbacksinusingtheKPCAmethodasthebuildingcellsofanMKM.

AmorestraightforwardwaytousekernelmachinesfordeeplearningwasdevelopedbyMicrosoftresearchersfor
spokenlanguageunderstanding.[225]Themainideaistouseakernelmachinetoapproximateashallowneuralnet
withaninfinitenumberofhiddenunits,thenusestackingtosplicetheoutputofthekernelmachineandtheraw
inputinbuildingthenext,[Link]
ahyperparameteroftheoverallsystem,tobedeterminedbycrossvalidation.

Applications
Automaticspeechrecognition
Speechrecognitionhasbeenrevolutionisedbydeeplearning,especiallybyLongshorttermmemory(LSTM),a
recurrentneuralnetworkpublishedbySeppHochreiter&JrgenSchmidhuberin1997.[51]LSTMRNNs
circumventthevanishinggradientproblemandcanlearn"VeryDeepLearning"tasks[5]thatinvolvespeechevents
separatedbythousandsofdiscretetimesteps,whereonetimestepcorrespondstoabout10ms.In2003,LSTM
withforgetgates[108]becamecompetitivewithtraditionalspeechrecognizersoncertaintasks.[52]In2007,LSTM
trainedbyConnectionistTemporalClassification(CTC)[53]achievedexcellentresultsincertainapplications,[54]
althoughcomputersweremuchslowerthantoday.In2015,Google'slargescalespeechrecognitionsuddenly
almostdoubleditsperformancethroughCTCtrainedLSTM,nowavailabletoallsmartphoneusers.[55]

Theinitialsuccessofdeeplearninginspeechrecognition,however,[Link]
[Link]
commondatasetusedforinitialevaluationsofdeeplearningarchitectures.Theentiresetcontains630speakers
fromeightmajordialectsofAmericanEnglish,whereeachspeakerreads10sentences.[226]Itssmallsizeallows
[Link],theTIMITtaskconcernsphonesequence
recognition,which,unlikewordsequencerecognition,allowsveryweak"languagemodels"andthusthe
[Link]
TIMITbyLiDengandcollaboratorsaround20092010,contrastingtheGMM(andothergenerativemodelsof
speech)[Link],stimulatedearlyindustrialinvestmentindeeplearningforspeechrecognitionfromsmall
tolargescales,[48][69][Link]
withcomparableperformance(lessthan1.5%inerrorrate)betweendiscriminativeDNNsandgenerativemodels.
Theerrorrateslistedbelow,includingtheseearlyresultsandmeasuredaspercentphoneerrorrates(PER),have
beensummarizedoveratimespanofthepast20years:

Method PER(%)
RandomlyInitializedRNN 26.1
BayesianTriphoneGMMHMM 25.6
HiddenTrajectory(Generative)Model 24.8
MonophoneRandomlyInitializedDNN 23.4
MonophoneDBNDNN 22.4
TriphoneGMMHMMwithBMMITraining 21.7
MonophoneDBNDNNonfbank 20.7
ConvolutionalDNN[227] 20.0
[Link] 18.7
EnsembleDNN/CNN/RNN[228] 18.2
BidirectionalLSTM 17.9

In2010,industrialresearchersextendeddeeplearningfromTIMITtolargevocabularyspeechrecognition,by
adoptinglargeoutputlayersoftheDNNbasedoncontextdependentHMMstatesconstructedbydecision
trees.[229][230][231]ComprehensivereviewsofthisdevelopmentandofthestateoftheartasofOctober2014are
providedintherecentSpringerbookfromMicrosoftResearch.[70]Anearlierarticle[232]reviewedthebackground
ofautomaticspeechrecognitionandtheimpactofvariousmachinelearningparadigms,includingdeeplearning.

Onefundamentalprincipleofdeeplearningistodoawaywithhandcraftedfeatureengineeringandtouseraw
[Link]"raw"
spectrogramorlinearfilterbankfeaturesatSRIinthelate1990s,[46]andlateratMicrosoft,[233]showingits
superiorityovertheMelCepstralfeatureswhichcontainafewstagesoffixedtransformationfromspectrograms.
Thetrue"raw"featuresofspeech,waveforms,havemorerecentlybeenshowntoproduceexcellentlargerscale
speechrecognitionresults.[234]

SincetheinitialsuccessfuldebutofDNNsforspeakerrecognitioninthelate1990sandspeechrecognitionaround
20092011andofLSTMaround20032007,[Link](andfuture
directions)canbesummarizedintoeightmajorareas:[1][50][70]

Scalingup/outandspeedupDNNtraininganddecoding
SequencediscriminativetrainingofDNNs
Featureprocessingbydeepmodelswithsolidunderstandingoftheunderlyingmechanisms
AdaptationofDNNsandofrelateddeepmodels
MultitaskandtransferlearningbyDNNsandrelateddeepmodels
Convolutionneuralnetworksandhowtodesignthemtobestexploitdomainknowledgeofspeech
RecurrentneuralnetworkanditsrichLSTMvariants
Othertypesofdeepmodelsincludingtensorbasedmodelsandintegrateddeepgenerative/discriminative
models.

Largescaleautomaticspeechrecognitionisthefirstandmostconvincingsuccessfulcaseofdeeplearninginthe
recenthistory,embracedbybothindustryandacademiaacrosstheboard.Between2010and2014,thetwomajor
conferencesonsignalprocessingandspeechrecognition,IEEEICASSPandInterspeech,haveseenalarge
increaseinthenumbersofacceptedpapersintheirrespectiveannualconferencepapersonthetopicofdeep
[Link],allmajorcommercialspeechrecognitionsystems(e.g.,
MicrosoftCortana,Xbox,SkypeTranslator,AmazonAlexa,GoogleNow,AppleSiri,BaiduandiFlyTekvoice
search,andarangeofNuancespeechproducts,etc.)areallbasedondeeplearningmethods.[1][235][236][237]See
alsotherecentmediainterviewwiththeCTOofNuanceCommunications.[238]

Imagerecognition

[Link]
handwrittendigitsandincludes60,000trainingexamplesand10,[Link],itssmallsize
[Link].[239]The
currentbestresultonMNISTisanerrorrateof0.23%,achievedbyCiresanetal.in2012.[240]

AccordingtoLeCun,[67]intheearly2000s,inanindustrialapplicationCNNsalreadyprocessedanestimated10%
to20%[Link]
[Link]
aroundfordecades,[31]andGPUimplementationsofNNsforyears,[74]includingCNNs,[75]fastimplementations
ofCNNswithmaxpoolingonGPUsinthestyleofDanCiresanandcolleagues[96]wereneededtomakeadentin
computervision.[5]In2011,thisapproachachievedforthefirsttimesuperhumanperformanceinavisualpattern
recognitioncontest.[98]Alsoin2011,itwontheICDARChinesehandwritingcontest,andinMay2012,itwonthe
ISBIimagesegmentationcontest.[99]Until2011,CNNsdidnotplayamajorroleatcomputervisionconferences,
butinJune2012,[Link][101]showedhowmaxpooling
CNNsonGPUcandramaticallyimprovemanyvisionbenchmarkrecords,sometimeswithhumancompetitiveor
evensuperhumanperformance.InOctober2012,asimilarsystembyAlexKrizhevskyintheteamofGeoff
Hinton[100]wonthelargescaleImageNetcompetitionbyasignificantmarginovershallowmachinelearning
methods.InNovember2012,Ciresanetal.'ssystemalsowontheICPRcontestonanalysisoflargemedicalimages
forcancerdetection,andinthefollowingyearalsotheMICCAIGrandChallengeonthesametopic.[241]In2013
and2014,theerrorrateontheImageNettaskusingdeeplearningwasfurtherreducedquickly,followingasimilar
[Link]
improvementsinthetechnologytothepubliceye.[242]

Asintheambitiousmovesfromautomaticspeechrecognitiontowardautomaticspeechtranslationand
understanding,imageclassificationhasrecentlybeenextendedtothemorechallengingtaskofautomaticimage
captioning,inwhichdeeplearning(oftenasacombinationofCNNsandLSTMs)istheessentialunderlying
technology[243][244][245][246]

Oneexampleapplicationisacarcomputersaidtobetrainedwithdeeplearning,whichmayenablecarsto
interpret360cameraviews.[247]AnotherexampleisthetechnologyknownasFacialDysmorphologyNovel
Analysis(FDNA)usedtoanalyzecasesofhumanmalformationconnectedtoalargedatabaseofgenetic
syndromes.

Naturallanguageprocessing

Neuralnetworkshavebeenusedforimplementinglanguagemodelssincetheearly2000s.[109][248]Recurrent
neuralnetworks,especiallyLSTM,[51][Link]
improvemachinetranslation[110]andLanguageModeling.[111][112]LSTMcombinedwithCNNsalsoimproved
automaticimagecaptioning[143]andaplethoraofotherapplications.[5]

Otherkeytechniquesinthisfieldarenegativesampling[249][Link],suchas
word2vec,canbethoughtofasarepresentationallayerinadeeplearningarchitecture,thattransformsanatomic
wordintoapositionalrepresentationofthewordrelativetootherwordsinthedatasetthepositionisrepresented
[Link](RNN)allows
thetrainingofthenetworktoparsesentencesandphrasesusinganeffectivecompositionalvectorgrammar.A
compositionalvectorgrammarcanbethoughtofasprobabilisticcontextfreegrammar(PCFG)implementedbya
recursiveneuralnetwork.[250]Recursiveautoencodersbuiltatopwordembeddingshavebeentrainedtoassess
sentencesimilarityanddetectparaphrasing.[250]Deepneuralarchitectureshaveachievedstateoftheartresultsin
manynaturallanguageprocessingtaskssuchasconstituencyparsing,[251]sentimentanalysis,[252]information
retrieval,[253][254]spokenlanguageunderstanding,[255]machinetranslation,[110][256]contextualentitylinking,[257]
writingstylerecognition[258]andothers.[259]

Drugdiscoveryandtoxicology

Thepharmaceuticalindustryfacestheproblemthatalargepercentageofcandidatedrugsfailtoreachthemarket.
Thesefailuresofchemicalcompoundsarecausedbyinsufficientefficacyonthebiomoleculartarget(ontarget
effect),undetectedandundesiredinteractionswithotherbiomolecules(offtargeteffects),orunanticipatedtoxic
effects.[260][261]In2012,ateamledbyGeorgeDahlwonthe"MerckMolecularActivityChallenge"usingmulti
taskdeepneuralnetworkstopredictthebiomoleculartargetofacompound.[262][263]In2014,SeppHochreiter's
groupusedDeepLearningtodetectofftargetandtoxiceffectsofenvironmentalchemicalsinnutrients,household
productsanddrugsandwonthe"Tox21DataChallenge"ofNIH,FDAandNCATS.[264][265]Theseimpressive
successesshowthatdeeplearningmaybesuperiortoothervirtualscreeningmethods.[266][267]Researchersfrom
GoogleandStanfordenhanceddeeplearningfordrugdiscoverybycombiningdatafromavarietyofsources.[268]
In2015,AtomwiseintroducedAtomNet,thefirstdeeplearningneuralnetworksforstructurebasedrationaldrug
design.[269]Subsequently,AtomNetwasusedtopredictnovelcandidatebiomoleculesforseveraldiseasetargets,
mostnotablytreatmentsfortheEbolavirus[270]andmultiplesclerosis.[271][272]
Customerrelationshipmanagement

Recentlysuccesshasbeenreportedwithapplicationofdeepreinforcementlearningindirectmarketingsettings,
[Link]
possibledirectmarketingactionsoverthecustomerstatespace,[Link]
valuefunctionwasshowntohaveanaturalinterpretationascustomerlifetimevalue.[273]

Recommendationsystems

Recommendationsystemshaveuseddeeplearningtoextractmeaningfuldeepfeaturesforlatentfactormodelfor
contentbasedrecommendationformusic.[274]Recently,amoregeneralapproachforlearninguserpreferences
frommultipledomainsusingmultiviewdeeplearninghasbeenintroduced.[275]Themodelusesahybrid
collaborativeandcontentbasedapproachandenhancesrecommendationsinmultipletasks.

Biomedicalinformatics

Recently,adeeplearningapproachbasedonanautoencoderartificialneuralnetworkhasbeenusedin
bioinformatics,topredictGeneOntologyannotationsandgenefunctionrelationships.[276]

Inmedicalinformatics,deeplearninghasalsobeenusedinthehealthdomain,includingthepredictionofsleep
qualitybasedonwearabledata[277]andpredictionsofhealthcomplicationsfromElectronicHealthRecord
data.[278]

Theoriesofthehumanbrain
Computationaldeeplearningiscloselyrelatedtoaclassoftheoriesofbraindevelopment(specifically,neocortical
development)proposedbycognitiveneuroscientistsintheearly1990s.[279]Anapproachablesummaryofthis
workisElman,etal.'s1996book"RethinkingInnateness"[280](seealso:ShragerandJohnson[281]Quartzand
Sejnowski[282]).Asthesedevelopmentaltheorieswerealsoinstantiatedincomputationalmodels,theyare
[Link]
sharetheinterestingpropertythatvariousproposedlearningdynamicsinthebrain(e.g.,awaveofnervegrowth
factor)conspiretosupporttheselforganizationofjustthesortofinterrelatedneuralnetworksutilizedinthelater,
purelycomputationaldeeplearningmodelsandsuchcomputationalneuralnetworksseemanalogoustoaviewof
thebrain'sneocortexasahierarchyoffiltersinwhicheachlayercapturessomeoftheinformationintheoperating
environment,andthenpassestheremainder,aswellasmodifiedbasesignal,tootherlayersfurtherupthe
[Link],[Link]
describedinTheNewYorkTimesin1995:"...theinfant'sbrainseemstoorganizeitselfundertheinfluenceof
wavesofsocalledtrophicfactors...differentregionsofthebrainbecomeconnectedsequentially,withonelayer
oftissuematuringbeforeanotherandsoonuntilthewholebrainismature."[283]

Theimportanceofdeeplearningwithrespecttotheevolutionanddevelopmentofhumancognitiondidnotescape
[Link]
neighborsmaybechangesinthetimingofdevelopment.[284]Amongprimates,thehumanbrainremainsrelatively
plasticuntillateinthepostnatalperiod,whereasthebrainsofourclosestrelativesaremorecompletelyformedby
[Link],humanshavegreateraccesstothecomplexexperiencesaffordedbybeingoutintheworldduringthe
[Link]"tunein"torapidlychangingfeaturesofthe
environmentthatotheranimals,moreconstrainedbyevolutionarystructuringoftheirbrains,areunabletotake
[Link]
corticaldevelopment,theymayalsoleadtochangesintheextractionofinformationfromthestimulus
[Link],alongwiththisflexibilitycomesan
extendedperiodofimmaturity,duringwhichwearedependentuponourcaretakersandourcommunityforboth
[Link]
fundamentalconditionofhumanevolution.[285]

Commercialactivities
DeeplearningisoftenpresentedasasteptowardsrealisingstrongAI[286]andthusmanyorganizationshave
becomeinterestedinitsuseforparticularapplications.InDecember2013,FacebookhiredYannLeCuntoheadits
newartificialintelligence(AI)labthatwastohaveoperationsinCalifornia,London,[Link]
willdevelopdeeplearningtechniquestohelpFacebookdotaskssuchasautomaticallytagginguploadedpictures
withthenamesofthepeopleinthem.[287]Latein2014,FacebookalsohiredVladimirVapnik,amaindeveloperof
theVapnikChervonenkistheoryofstatisticallearning,andcoinventorofthesupportvectormachinemethod.[288]

In2014,GooglealsoboughtDeepMindTechnologies,aBritishstartupthatdevelopedasystemcapableof
learninghowtoplayAtarivideogamesusingonlyrawpixelsasdatainput.In2015theydemonstratedAlphaGo
systemwhichachievedoneofthelongstanding"grandchallenges"ofAIbylearningthegameofGowellenough
tobeatahumanprofessionalGoplayer.[289][290][291]

In2015,Blippardemonstratedanewmobileaugmentedrealityapplicationthatmakesuseofdeeplearningto
recognizeobjectsinrealtime.[292]

Criticismandcomment
Giventhefarreachingimplicationsofartificialintelligencecoupledwiththerealizationthatdeeplearningis
emergingasoneofitsmostpowerfultechniques,thesubjectisunderstandablyattractingbothcriticismand
comment,andinsomecasesfromoutsidethefieldofcomputerscienceitself.

[Link]
mostcommondeeparchitecturesisimplementedusinggradientdescentwhilegradientdescenthasbeen
understoodforawhilenow,thetheorysurroundingotheralgorithms,suchascontrastivedivergenceislessclear.
(i.e.,Doesitconverge?Ifso,howfast?Whatisitapproximating?)Deeplearningmethodsareoftenlookedatasa
blackbox,withmostconfirmationsdoneempirically,ratherthantheoretically.

OtherspointoutthatdeeplearningshouldbelookedatasasteptowardsrealizingstrongAI,notasanall
[Link],theystilllackmuchofthefunctionality
[Link]:

"Realistically,[Link]
techniqueslackwaysofrepresentingcausalrelationships(...)havenoobviouswaysofperforming
logicalinferences,andtheyarealsostillalongwayfromintegratingabstractknowledge,suchas
informationaboutwhatobjectsare,whattheyarefor,[Link]
[Link],likeWatson(...)usetechniqueslikedeeplearningasjustoneelementinavery
complicatedensembleoftechniques,rangingfromthestatisticaltechniqueofBayesianinferenceto
deductivereasoning."[293]
Totheextentthatsuchaviewpointimplies,withoutintendingto,thatdeeplearningwillultimatelyconstitute
nothingmorethantheprimitivediscriminatorylevelsofacomprehensivefuturemachineintelligence,arecentpair
ofspeculationsregardingartandartificialintelligence[294][Link]
firstsuchspeculationisthatitmightbepossibletotrainamachinevisionstacktoperformthesophisticatedtaskof
discriminatingbetween"oldmaster"andamateurfiguredrawingsandthesecondisthatsuchasensitivitymight
[Link],moreover,thatsuchaneventuality
wouldbeinlinewithanthropology,whichidentifiesaconcernwithaestheticsasakeyelementofbehavioral
modernity.[295]

Infurtherreferencetotheideathatasignificantdegreeofartisticsensitivitymightinherewithinrelativelylow
levels,whetherbiologicalordigital,ofthecognitivehierarchy,apublishedseriesofgraphicrepresentationsofthe
internalstatesofdeep(2030layers)neuralnetworksattemptingtodiscernwithinessentiallyrandomdatathe
imagesonwhichtheyweretrained[296]seemtodemonstrateastrikingvisualappealinlightoftheremarkablelevel
ofpublicattentionwhichthisworkcaptured:theoriginalresearchnoticereceivedwellover1,000comments,and
thecoveragebyTheGuardian[297]wasforatimethemostfrequentlyaccessedarticleonthatnewspaper'swebsite.

Somecurrentlypopularandsuccessfuldeeplearningarchitecturesdisplaycertainproblematicbehaviors,[298]such
asconfidentlyclassifyingunrecognizableimagesasbelongingtoafamiliarcategoryofordinaryimages[299]and
misclassifyingminusculeperturbationsofcorrectlyclassifiedimages.[300]ThecreatorofOpenCog,BenGoertzel,
hypothesized[298]thatthesebehaviorsareduetolimitationsintheinternalrepresentationslearnedbythese
architectures,andthattheselimitationswouldinhibitintegrationofthesearchitecturesintoheterogeneousmulti
[Link]
architecturesthatinternallyformstateshomologoustoimagegrammar[301]decompositionsofobservedentities
andevents.[298]Learningagrammar(visualorlinguistic)fromtrainingdatawouldbeequivalenttorestrictingthe
systemtocommonsensereasoningwhichoperatesonconceptsintermsofproductionrulesofthegrammar,andis
abasicgoalofbothhumanlanguageacquisition[302]andAI.(SeealsoGrammarinduction.[303])

Softwarelibraries
Deeplearning4jAnopensourcedeeplearninglibrarywrittenforJava/C++withLSTMsand
[Link].
GensimAtoolkitfornaturallanguageprocessingimplementedinthePythonprogramminglanguage.
KerasAnopensourcedeeplearningframeworkforthePythonprogramminglanguage.
MicrosoftCNTK(ComputationalNetworkToolkit)Microsoft'sopensourcedeeplearningtoolkitfor
[Link].
MXNetAnopensourcedeeplearningframeworkthatallowsyoutodefine,train,anddeploydeepneural
networks.
OpenNNAnopensourceC++librarywhichimplementsdeepneuralnetworksandprovides
parallelizationwithCPUs.
PaddlePaddle([Link]
scalabledeeplearningplatformwithCPUsandGPUs,originallydevelopedbyBaidu.
TensorFlowGoogle'sopensourcemachinelearninglibraryinC++[Link]
providesparallelizationwithCPUsandGPUs.
TheanoAnopensourcemachinelearninglibraryforPythonsupportedbytheUniversityofMontrealand
YoshuaBengio'steam.
TorchAnopensourcesoftwarelibraryformachinelearningbasedontheLuaprogramminglanguageand
usedbyFacebook.
Caffe([Link]
[Link](BVLC)andby
communitycontributors.
 DIANNE([Link]
developedatGhentUniversity,[Link]
servers.

Seealso
Applicationsofartificialintelligence
Artificialneuralnetworks
Boltzmannmachine
CompressedSensing
Connectionism
Echostatenetwork
Listofartificialintelligenceprojects
Liquidstatemachine
Listofdatasetsformachinelearningresearch
Reservoircomputing
Sparsecoding

References
[Link],[Link],D.(2014)."DeepLearning:MethodsandApplications"(PDF).FoundationsandTrendsinSignal
Processing.7(34):[Link].1561/2000000039.
[Link],YoshuaBengio,andAaronCourville(2016).[Link]([Link]
[Link])
[Link],Yoshua(2009)."LearningDeepArchitecturesforAI"(PDF).FoundationsandTrendsinMachineLearning.2
(1):[Link].1561/2200000006.
[Link],[Link],[Link],P.(2013)."RepresentationLearning:AReviewandNewPerspectives".IEEE
TransactionsonPatternAnalysisandMachineIntelligence.35(8):[Link].5538 .
doi:10.1109/tpami.2013.50.
[Link],J.(2015)."DeepLearninginNeuralNetworks:AnOverview".NeuralNetworks.61:85117.
arXiv:1404.7828 .doi:10.1016/[Link].2014.09.003.
[Link],YoshuaLeCun,YannHinton,Geoffrey(2015)."DeepLearning".Nature.521:436444.
doi:10.1038/nature14539.PMID26017442.
[Link],DerekC.
Rose,[Link],2013
[Link],Jrgen(2015)."DeepLearning".Scholarpedia.10(11):[Link].4249/scholarpedia.32832.
[Link]."APatternLanguageforDeepLearning".
[Link],P.(2015).DeepConvolutionalNeuralNetworksforSmileRecognition(MScThesis).ImperialCollegeLondon,
[Link].06535 .
[Link],[Link],S.Y.(2013)."HierarchicalRepresentationUsingNMF".[Link]
[Link].1007/9783642420542_58.
ISBN9783642420535.
[Link],B.A.(1996)."Emergenceofsimplecellreceptivefieldpropertiesbylearningasparsecodefornatural
images".Nature.381(6583):[Link].1038/381607a0.PMID8637596.
[Link],R.(April2011).[Link].Eventoccursat7min
45s.
[Link],L.(20October2014)."MachineLearningMaestroMichaelJordanontheDelusionsofBigDataandOtherHuge
EngineeringEfforts".IEEESpectrum.
[Link].,"Learningcomplex,extendedsequencesusingtheprincipleofhistorycompression,"Neural
Computation,4,pp.234242,1992.
[Link],G.E."Deepbeliefnetworks".Scholarpedia.4(5):[Link].4249/scholarpedia.5947.
[Link],
Hungary
[Link](1989)."Approximationsbysuperpositionsofsigmoidalfunctions"(PDF).MathematicsofControl,Signals,
andSystems.2(4):[Link].1007/bf02551274.
[Link],Kurt(1991)."ApproximationCapabilitiesofMultilayerFeedforwardNetworks".NeuralNetworks.4(2):251
[Link].1016/08936080(91)90009t.
[Link],Simon(1998).NeuralNetworks:AComprehensiveFoundation,Volume2,PrenticeHall.ISBN0132733501.
[Link],M.(1995)FundamentalsofArtificialNeuralNetworksMITPress,p.48
[Link],K.P.(2012)Machinelearning:aprobabilisticperspectiveMITPress
[Link],[Link],[Link],[Link],[Link],R.R.(2012)."Improvingneuralnetworksby
preventingcoadaptationoffeaturedetectors".arXiv:1207.0580 [[Link]].
[Link],Alexey(1965).[Link]:NaukovaDumka.
[Link],Alexey(1971)."Polynomialtheoryofcomplexsystems".IEEETransactionsonSystems,Manand
Cybernetics(4):364378.
[Link],K.(1980)."Neocognitron:Aselforganizingneuralnetworkmodelforamechanismofpatternrecognition
unaffectedbyshiftinposition".[Link].36:[Link].1007/bf00344251.PMID7370364.
[Link](1970).TherepresentationofthecumulativeroundingerrorofanalgorithmasaTaylorexpansionof
[Link]'sThesis(inFinnish),[Link],67.
[Link],Andreas(2012).WhoInventedtheReverseModeofDifferentiation?.OptimizationStories,Documenta
Matematica,ExtraVolumeISMP(2012),389400.
[Link].,"BeyondRegression:NewToolsforPredictionandAnalysisintheBehavioralSciences,"PhDthesis,
HarvardUniversity,1974.
[Link](1982).[Link]
(pp.762770).[Link]([Link]
[Link].,"BackpropagationAppliedtoHandwrittenZipCodeRecognition,"NeuralComputation,1,pp.541551,
1989.
[Link](1993).Habilitationthesis,TUM,1993.Page150ffdemonstratescreditassignmentacrossthe
equivalentof1,[Link]([Link]
[Link],[Link],[Link],David(19940808)."AnintegratedBooleanneuralnetworkfor
patternclassification".PatternRecognitionLetters.15(8):[Link].1016/01678655(94)900094.
[Link],[Link],PeterFrey,[Link],Radford(19950526)."Thewakesleepalgorithmfor
unsupervisedneuralnetworks".Science.268(5214):[Link].1126/science.7761831.
[Link].,"UntersuchungenzudynamischenneuronalenNetzen([Link]
[Link]),"[Link],[Link]:J.
Schmidhuber,1991.
[Link].,"Gradientflowinrecurrentnets:thedifficultyoflearninglongtermdependencies,"[Link]
[Link],editors,[Link],2001.
[Link],[Link],"Cresceptron:aselforganizingneuralnetworkwhichgrowsadaptively([Link]
[Link]/~weng/research/[Link]),"[Link],
Baltimore,Maryland,volI,pp.576581,June,1992.
[Link],[Link],"Learningrecognitionandsegmentationof3Dobjectsfrom2Dimages([Link]
[Link]/~weng/research/[Link]),"[Link],Berlin,
Germany,pp.121128,May,1993.
[Link],[Link],"LearningrecognitionandsegmentationusingtheCresceptron([Link]
edu/~weng/research/[Link]),"InternationalJournalofComputerVision,vol.25,no.2,pp.105139,Nov.
1997.
[Link],Bourlard,Renals,Cohen,Franco(1993)"Hybridneuralnetwork/hiddenMarkovmodelsystemsforcontinuous
[Link]/IJPRAI"
[Link].(1992)Arealtimerecurrenterrorpropagationnetworkwordrecognitionsystem,ICASSP.
[Link],Hanazawa,Hinton,Shikano,Lang.(1989)"[Link]
TransactionsonAcoustics,SpeechandSignalProcessing."
[Link],[Link],LiGlass,JimKhudanpur,[Link],[Link],N.O'Shaughnessy,D.(2009)."Research
DevelopmentsandDirectionsinSpeechRecognitionandUnderstanding,Part1".IEEESignalProcessingMagazine.26
(3):[Link].1109/msp.2009.932166.
[Link](1991)."ArtificialNeuralNetworksandtheirApplicationtoSpeech/SequenceRecognition,"[Link],
McGillUniversity,Canada.
[Link],[Link],[Link],M.(1994)."Analysisofcorrelationstructureforaneuralpredictivemodelwith
applicationstospeechrecognition".NeuralNetworks.7(2):[Link].1016/08936080(94)900272.
[Link],[Link],[Link],[Link],M.(2000)."RobustnesstoTelephoneHandsetDistortioninSpeaker
RecognitionbyDiscriminativeFeatureDesign".SpeechCommunication.31(2):[Link].1016/s0167
6393(99)000771.
[Link],[Link],[Link],[Link],[Link],[Link],[Link],[Link],[Link],[Link],T.
Kingsbury,B.(2012)."DeepNeuralNetworksforAcousticModelinginSpeechRecognitionThesharedviewsof
fourresearchgroups".IEEESignalProcessingMagazine.29(6):[Link].1109/msp.2012.2205597.
[Link],[Link],[Link],B.(2013)."Newtypesofdeepneuralnetworklearningforspeechrecognitionand
relatedapplications:Anoverview(ICASSP)".
[Link]:[Link],2013(byGeoffHinton).
[Link]:"AchievementsandChallengesofDeepLearningFromSpeechAnalysisandRecognitionToLanguage
andMultimodalProcessing,"Interspeech,September2014.
[Link],SeppandSchmidhuber,JrgenLongShortTermMemory,NeuralComputation,9(8):17351780,1997
[Link],DouglasEck,NicoleBeringer,andJrgenSchmidhuber(2003).BiologicallyPlausibleSpeechRecognition
[Link],
BioADIT2004,Lausanne,Switzerland,p.175184,[Link]([Link]
[Link],SantiagoFernandez,FaustinoGomez,andJrgenSchmidhuber(2006).Connectionisttemporal
classification:Labellingunsegmentedsequencedatawithrecurrentneuralnets.ProceedingsofICML06,pp.369376.
[Link],AlexGraves,andJrgenSchmidhuber(2007).Anapplicationofrecurrentneuralnetworksto
[Link](2),pp.220229.
[Link],AndrewSenior,KanishkaRao,FranoiseBeaufaysandJohanSchalkwyk(September2015):Googlevoice
search:fasterandmoreaccurate.([Link]
[Link],[Link],[Link](2000).MultiValuedandUniversalBinaryNeurons:Theory,
[Link]&BusinessMedia.
[Link]"deeplearning"postedbyJrgenSchmidhuber(2015)Online(https://
[Link]/100849856540000067209/posts/7N6z251w2Wd?pid=6127540521703625346&oid=10084985654000006
7209)
[Link].,"Learningmultiplelayersofrepresentation,"TrendsinCognitiveSciences,11,pp.428434,2007.
[Link].,"MyFirstDeepLearningSystemof1991+DeepLearningTimeline19622013."Online([Link]
[Link]/~juergen/[Link])
[Link],LiHinton,GeoffreyKingsbury,Brian(1May2013)."Newtypesofdeepneuralnetworklearningforspeech
recognitionandrelatedapplications:Anoverview"[Link].
[Link],ICASSP,2013.
[Link],[Link],[Link],Adeepconvolutionalneuralnetworkusingheterogeneouspoolingfortrading
acousticinvariancewithphoneticconfusion,ICASSP,2013.
[Link].,"ConvolutionalneuralnetworksforLVCSR,"ICASSP,2013.
[Link](2014).LongShortTermMemoryrecurrentneuralnetwork
architecturesforlargescaleacousticmodeling.ProceedingsofInterspeech2014.
[Link],XihongWu(2015).ConstructingLongShortTermMemorybasedDeepRecurrentNeuralNetworksfor
LargeVocabularySpeechRecognitionarXiv:1410.4281([Link]
[Link](2015).UnidirectionalLongShortTermMemoryRecurrentNeuralNetworkwithRecurrent
[Link],pp.44704474.
[Link](2016).SlidesonDeepLearningOnline([Link]
[Link],[Link],[Link],[Link](2011)."Discriminativepretrainingofdeepneuralnetworks,"[Link].
[Link]:DeepLearningforSpeechRecognitionandRelatedApplications,Whistler,BC,Canada,Dec.2009
(Organizers:LiDeng,GeoffHinton,[Link]).
[Link],[Link],L.(2014)."AutomaticSpeechRecognition:ADeepLearningApproach(Publisher:Springer)".
[Link](2015)[Link]
[Link]
72."NvidiaCEObetsbigondeeplearningandVR".VentureBeat.April5,2016.
73."Fromnotworkingtoneuralnetworking".TheEconomist.
[Link],[Link],K.(2004)."GPUimplementationofneuralnetworks".PatternRecognition.37(6):13111314.
doi:10.1016/[Link].2004.01.013.
[Link],K.,Puri,S.,andSimard,P.(2006).Highperformanceconvolutionalneuralnetworksfordocument
[Link].
[Link].,"DeepBigSimpleNeuralNetsforHandwrittenDigitRecognition,"NeuralComputation,22,pp.
32073220,2010.
 [Link],[Link],[Link].,"LargescaleDeepUnsupervisedLearningusingGraphicsProcessors,"Proc.26thInt.
[Link],2009.
[Link],VivienneChen,YuHsinYang,TienJuEmer,Joel(2017)."EfficientProcessingofDeepNeuralNetworks:A
TutorialandSurvey".arXiv:1703.09039 .
[Link],MPoggio,T(1999)."Hierarchicalmodelsofobjectrecognitionincortex".NatureNeuroscience.2(11):
[Link].1038/14819.
[Link],[Link],[Link],[Link],[Link],[Link],[Link].1989Backpropagation
[Link],1(4):541551.
[Link],AndreasandWalther,A..PrinciplesandTechniquesofAlgorithmicDifferentiation,[Link],
2008.
[Link],HenryJ.(1960)."Gradienttheoryofoptimalflightpaths".ArsJournal.30(10):[Link].2514/8.5282.
[Link](1961,April).[Link]
[Link].
[Link],Stuart(1962)."Thenumericalsolutionofvariationalproblems".JournalofMathematicalAnalysisand
Applications.5(1):[Link].1016/0022247x(62)900045.
[Link],Stuart(1973)."Thecomputationalsolutionofoptimalcontrolproblemswithtimelag".IEEETransactionson
AutomaticControl.18(4):[Link].1109/tac.1973.1100330.
[Link],D.E.,Hinton,G.E.&Williams,R.J.,"Learningrepresentationsbybackpropagatingerrors"nature,1974.
[Link](1990).ArtificialNeuralNetworks,BackPropagationandtheKelleyBrysonGradientProcedure.J.
Guidance,ControlandDynamics,1990.
[Link],AlexandSchmidhuber,JrgenOfflineHandwritingRecognitionwithMultidimensionalRecurrentNeural
Networks,inBengio,YoshuaSchuurmans,DaleLafferty,JohnWilliams,[Link],Aron(eds.),
AdvancesinNeuralInformationProcessingSystems22(NIPS'22),December7th10th,2009,Vancouver,BC,Neural
InformationProcessingSystems(NIPS)Foundation,2009,pp.545552
[Link],[Link],[Link],[Link],[Link],[Link],J.(2009)."ANovelConnectionist
SystemforImprovedUnconstrainedHandwritingRecognition".IEEETransactionsonPatternAnalysisandMachine
Intelligence.31(5):[Link].1109/tpami.2008.137.
[Link](2003).HierarchicalNeuralNetworksforImageInterpretation.(PDF).LectureNotesinComputerScience.
[Link].
[Link],P.(1986)."Informationprocessingindynamicalsystems:Foundationsofharmonytheory.".InD.E.
Rumelhart,[Link],&[Link]:Explorationsinthe
[Link].194281.
[Link],[Link],[Link],Y.(2006)."Afastlearningalgorithmfordeepbeliefnets"(PDF).NeuralComputation.
18(7):[Link].1162/neco.2006.18.7.1527.PMID16764513.
[Link],G.(2009)."Deepbeliefnetworks".Scholarpedia.4(5):[Link].4249/scholarpedia.5947.
[Link](25June2012)."HowManyComputerstoIdentifyaCat?16,000.".NewYorkTimes.
[Link],AndrewDean,Jeff(2012)."BuildingHighlevelFeaturesUsingLargeScaleUnsupervisedLearning".
arXiv:1112.6209 .
[Link],[Link],[Link],[Link],[Link],HighPerformanceConvolutional
[Link](IJCAI2011,
Barcelona),2011.
[Link],[Link],[Link],G.N.(2013)."LearningDeepPhysiologicalModelsofAffect".IEEE
ComputationalIntelligence.8(2):[Link].1109/mci.2013.2247823.
[Link],[Link],[Link],[Link].
NeuralNetworks,2012.
[Link],[Link],[Link],[Link]
[Link](NIPS2012),LakeTahoe,2012.
[Link],A.,Sutskever,[Link],G.E.(2012).ImageNetClassificationwithDeepConvolutionalNeural
Networks.NIPS2012:NeuralInformationProcessingSystems,LakeTahoe,Nevada
[Link],[Link],[Link]
ComputerVisionandPatternRecognitionCVPR2012.
[Link],"Distributedhierarchicalprocessingintheprimatecerebralcortex([Link]
[Link]/content/1/1/[Link]+html),"CerebralCortex,1,pp.147,1991.
[Link],"NaturalandArtificialIntelligence:IntroductiontoComputationalBrainMind([Link]
ArtificialIntelligenceIntroductionComputational/dp/0985875720),"BMIPress,ISBN9780985875725,2012.
[Link],"WhyHaveWePassed`NeuralNetworksDonotAbstractWell'?([Link]
[Link]),"NaturalIntelligence:theINNSMagazine,vol.1,no.1,pp.1322,2011.
[Link],[Link],[Link],"WhereWhatNetwork1:WhereandWhatAssistEachOtherThroughTopdown
Connections([Link]
DevelopmentandLearning(ICDL'08),Monterey,CA,Aug.912,pp.16,2008.
[Link],[Link],[Link],"SkullclosedAutonomousDevelopment:WWN7DealingwithScales([Link]
[Link]/~weng/research/[Link]),"[Link],July2728,East
Lansing,Michigan,pp.+19,2013.
[Link],Christian,AlexanderToshev,andDumitruErhan."Deepneuralnetworksforobjectdetection."Advancesin
NeuralInformationProcessingSystems.2013.
[Link],FelixSchraudolph,NicholasSchmidhuber,Jrgen(2002)."LearningprecisetimingwithLSTMrecurrent
networks".JournalofMachineLearningResearch.3:115143.
[Link]
Languages.IEEETNN12(6):13331340,2001.
[Link],[Link],[Link](2014)"SequencetoSequenceLearningwithNeuralNetworks,"[Link].
[Link],OriolVinyals,MikeSchuster,NoamShazeer,YonghuiWu(2016).ExploringtheLimitsofLanguage
[Link]([Link]
[Link],CliffBrunk,OriolVinyals,AmarnagSubramanya(2015).MultilingualLanguageProcessingFromBytes.
arXiv([Link]
[Link].,"Recurrentneuralnetworkbasedlanguagemodel,"Interspeech,2010.
[Link],[Link]."Gradientbasedlearningappliedtodocumentrecognition".ProceedingsoftheIEEE.86(11):2278
[Link].1109/5.726791.
[Link],StuartDreyfus,KenichiNishio(2000).OnderivationofMLPbackpropagationfromtheKelleyBryson
[Link]
Networks(IJCNN2000),ComoItaly,[Link]([Link]
[Link])
[Link],[Link].I:Necessary
[Link].1,11(1963)25442550
[Link].p.578."Themostpopularmethodforlearning
inmultilayernetworksiscalledBackpropagation."
[Link],YuChiHo(1969).Appliedoptimalcontrol:optimization,estimation,[Link]
PublishingCompanyorXeroxCollegePublishing.p.481.
[Link](1976).[Link],16(2),146
160.
[Link](1974).Beyondregression:[Link],
HarvardUniversity.
[Link](1993).[Link]
INSTITUTESTUDIESINTHESCIENCESOFCOMPLEXITYPROCEEDINGS(Vol.15,pp.195195).Addison
WesleyPublishingCo.
[Link]..,"DeepNeuralNetworksforAcousticModelinginSpeechRecognition:Thesharedviewsoffour
researchgroups,"IEEESignalProcessingMagazine,pp.8297,November2012.
[Link]..,"Advancesinoptimizingrecurrentnetworks,"ICASSP,2013.
[Link]..,"ImprovingDNNsforLVCSRusingrectifiedlinearunitsanddropout,"ICASSP,2013.
[Link].,"APracticalGuidetoTrainingRestrictedBoltzmannMachines,"[Link].UTMLTR2010003,Dept.
CS.,[Link],2010.
[Link],GuangBinZhu,QinYuSiew,CheeKheong(2006)."Extremelearningmachine:theoryandapplications".
Neurocomputing.70(1):[Link].1016/[Link].2005.12.126.
[Link],Bernardetal.(2013)."Thenopropalgorithm:Anewlearningalgorithmformultilayerneuralnetworks".
NeuralNetworks.37:[Link].1016/[Link].2012.09.020.
[Link],YannCharpiat,Guillaume(2015)."Trainingrecurrentnetworkswithoutbacktracking".arXiv:1507.07680 .
[Link],Igor,etal."AbriefintroductiontoWeightlessNeuralSystems."ESANN.2009.
[Link](1967).Cyberneticsandforecastingtechniques.
AmericanElsevier,NY.
[Link](1968).Thegroupmethodofdatahandlingarivalofthemethodofstochastic
[Link],13(3):4355.
[Link],[Link],J.(2008)."MultilayeredGMDHtypeneuralnetworkselfselectingoptimumneuralnetwork
architectureanditsapplicationto3dimensionalmedicalimagerecognitionofbloodvessels".InternationalJournalof
InnovativeComputing,InformationandControl.4(1):175187.
133."UnsupervisedFeatureLearningandDeepLearningTutorial".
[Link],ChristianLiu,WeiJia,YangqingSermanet,PierreReed,ScottAnguelov,DragomirErhan,Dumitru
Vanhoucke,VincentRabinovich,Andrew(2014)."GoingDeeperwithConvolutions".ComputingResearchRepository.
arXiv:1409.4842 .
[Link],[Link],A."Learningtaskdependentdistributedrepresentationsbybackpropagationthroughstructure".
NeuralNetworks,1996.,[Link].1109/ICNN.1996.548916.
[Link],RichardLin,CliffNg,[Link],ChristopherD."ParsingNaturalScenesandNaturalLanguage
withRecursiveNeuralNetworks".The28thInternationalConferenceonMachineLearning(ICML2011).
[Link],RichardPerelygin,[Link],JeanChuang,[Link],[Link],AndrewPotts,
Christopher."RecursiveDeepModelsforSemanticCompositionalityOveraSentimentTreebank"(PDF).EMNLP2013.
[Link],DaanWierstra,JulianTogelius,andJrgenSchmidhuber(2009).Evolvingmemorycellstructuresfor
[Link](2),pp.755764.
[Link],AlexGraves,andJrgenSchmidhuber(2007).Sequencelabellinginstructureddomainswith
[Link].
[Link],CarlCase,JaredCasper,BryanCatanzaro,GregDiamos,ErichElsen,RyanPrenger,SanjeevSatheesh,
ShubhoSengupta,AdamCoates,AndrewNg(2014).DeepSpeech:Scalingupendtoendspeechrecognition.
arXiv:1412.5567([Link]
[Link],Y.,Qian,Y.,Xie,F.,andSoong,F.K.(2014).TTSsynthesiswithbidirectionalLSTMbasedrecurrentneural
[Link].
[Link],LijuanWang,[Link],andLeiXie(2015).[Link]
ProceedingsofICASSP2015.
[Link],AlexanderToshev,SamyBengio,andDumitruErhan(2015).ShowandTell:ANeuralImageCaption
[Link]([Link]
[Link],[Link]."Anempiricalevaluationofdeeparchitecturesonproblemswithmanyfactorsofvariation".Proc.
[Link].2007:473480.
[Link].,"TrainingProductofExpertsbyMinimizingContrastiveDivergence,"([Link]
bsps/[Link])NeuralComputation,14,pp.17711800,2002.
[Link],[Link],C.(2014)."TrainingRestrictedBoltzmannMachines:AnIntroduction"(PDF).PatternRecognition.47:
[Link].1016/[Link].2013.05.025.
147.ConvolutionalDeepBeliefNetworksonCIFAR10([Link]
[Link],HonglakGrosse,RogerRanganath,RajeshNg,AndrewY.(1January2009)."ConvolutionalDeepBelief
NetworksforScalableUnsupervisedLearningofHierarchicalRepresentations".[Link].609616.
doi:10.1145/1553374.1553453viaACMDigitalLibrary.
[Link],"PrinciplesofArtificialNeuralNetworks.3rdEdition",WorldScientificPublishers,2013.
[Link],"Largememorystorageandretrieval(LAMSTAR)network,USPatent5920852A",April1996.
[Link],"PrinciplesofArtificialNeuralNetworks.3rdEdition",WorldScientificPublishers,2013,pp.203274.
[Link],[Link],(2004),"Aneuralnetworkbaseddetectionofepilepsy",NeurologicalResearch,26(1):5560.
[Link],[Link],[Link],CW.(2010)."Automatedpredictionofapneaandhypopnea,usingaLAMSTAR
artificialneuralnetwork".AmericanJournalofRespiratoryandCriticalCareMedicine.171(7):727733.
[Link],[Link],[Link],[Link],R.K.(2008)."Blindadaptivefilteringfornoninvasiveextractionofthe
fetalelectrocardiogramanditsnonstationarities".[Link].,UK,PartH:JournalofEngineeringin
Medicine.222(8):[Link].1243/09544119jeim417.
[Link],"PrinciplesofArtificialNeuralNetworks.3rdEdition",WorldScientificPublishers,2013,pp.240253.
[Link],[Link],J.(2002)."ANeuralNetworkforBlindAdaptiveFilteringofUnknownNoisefromSpeech".
IntelligentEngineeringSystemsThroughArtificialNeuralNetworks.12:683688.
[Link],S.(2015)."IrisRecognitionforPersonalIdentificationUsingLAMSTARNeuralNetwork".International
JournalofComputerScienceandInformationTechnology.7(1).
[Link],"PrinciplesofArtificialNeuralNetworks.3rdEdition",WorldScientificPublishers",2013,pp.253274.
[Link],[Link],[Link],T.A.(2003)."RealtimecamerabasedfacedetectionusingamodifiedLAMSTAR
neuralnetworksystem".Proc.SPIE5015,ApplicationsofArtificialNeuralNetworksinImageProcessingVIII.
doi:10.1117/12.477405.
[Link],VSelvan,S.(2007)."IntrusionDetectionusinganImprovedCompetitiveLearningLamstarNetwork".
InternationalJournalofComputerScienceandNetworkSecurity.7(2):255263.
[Link],[Link],(2007),"ControlofunstablenonlinearandnonstationarysystemsusingLAMSTARneural
networks",Proceedingsof10thIASTEDonIntelligentControl,Sect.592,141144.
[Link],"[Link]",WorldScientificPublishers,2016,pp.57110.
[Link],[Link](1996)."NetworkbasedonSOM(selforganizingmap)modulescombinedwithstatistical
decisiontools".[Link].1:471475.
164.D,Graupe,[Link],(1998),"Alargememorystorageandretrievalneuralnetworkforadaptiveretrievaland
diagnosis",InternationalJournalofSoftwareEngineeringandKnowledgeEngineering,1998.
[Link],[Link],DLiu,K."Anovellargememoryneuralnetworkasanaidinmedicaldiagnosisapplications".
IEEETransactionsonInformationTechnologyinBiomedicine.5(3):[Link].1109/4233.945291.
[Link],[Link](2008)."AmodifiedLAMSTARneuralnetworkanditsapplications".Internationaljournalof
neuralsystems.18(4):[Link].1142/s0129065708001634.
[Link],"PrinciplesofArtificialNeuralNetworks.3rdEdition",WorldScientificPublishers,2013,p.217.
[Link],GeoffreySalakhutdinov,Ruslan(2012)."AbetterwaytopretraindeepBoltzmannmachines"(PDF).Advancesin
Neural.3:19.
[Link],GeoffreySalakhutdinov,Ruslan(2009)."EfficientLearningofDeepBoltzmannMachines"(PDF).3:448455.
[Link],YoshuaLeCun,Yann(2007)."ScalingLearningAlgorithmstowardsAI"(PDF).1:141.
[Link],HugoSalakhutdinov,Ruslan(2010)."EfficientLearningofDeepBoltzmannMachines"(PDF):693700.
[Link],PascalLarochelle,HugoLajoie,IsabelleBengio,YoshuaManzagol,PierreAntoine(2010)."Stacked
DenoisingAutoencoders:LearningUsefulRepresentationsinaDeepNetworkwithaLocalDenoisingCriterion".The
JournalofMachineLearningResearch.11:33713408.
[Link](1987).[Link],pages279284.
[Link],LiYu,Dong(2011)."DeepConvexNet:AScalableArchitectureforSpeechPatternClassification"(PDF).
ProceedingsoftheInterspeech:22852288.
[Link],LiYu,DongPlatt,John(2012)."Scalablestackingandlearningforbuildingdeeparchitectures"(PDF).2012
IEEEInternationalConferenceonAcoustics,SpeechandSignalProcessing(ICASSP):21332136.
[Link],Wolpert(1992)."Stackedgeneralization".NeuralNetworks.5(2):[Link].1016/S08936080(05)80023
1.
[Link],Yoshua(2009)."LearningdeeparchitecturesforAI".FoundationsandTrendsinMachineLearning.2(1):1
[Link].1561/2200000006.
[Link],BrianDeng,LiYu,Dong(2012)."Tensordeepstackingnetworks".IEEETransactionsonPattern
AnalysisandMachineIntelligence.115:[Link].1109/tpami.2012.268.
[Link],GeoffreySalakhutdinov,Ruslan(2006)."ReducingtheDimensionalityofDatawithNeuralNetworks".Science.
313:[Link].1126/science.1127647.PMID16873662.
[Link],[Link],[Link],[Link],A.(2012)."ContextDependentPreTrainedDeepNeuralNetworksforLarge
VocabularySpeechRecognition".IEEETransactionsonAudio,Speech,andLanguageProcessing.20(1):3042.
doi:10.1109/tasl.2011.2134090.
[Link],AbdelrahmanDahl,GeorgeHinton,Geoffrey(2012)."AcousticModelingUsingDeepBeliefNetworks".
IEEETransactionsonAudio,Speech,andLanguageProcessing.20(1):[Link].1109/tasl.2011.2109382.
[Link],AaronBergstra,JamesBengio,Yoshua(2011)."ASpikeandSlabRestrictedBoltzmannMachine"(PDF).
JMLR:WorkshopandConferenceProceeding.15:233241.
[Link],AaronBergstra,JamesBengio,Yoshua(2011)."UnsupervisedModelsofImagesbySpikeandSlabRBMs".
Proceedingsofthe28thInternationalConferenceonMachineLearning(PDF).[Link].18.
[Link],TBeauchamp,J(1988)."BayesianVariableSelectioninLinearRegression".JournaloftheAmerican
StatisticalAssociation.83(404):[Link].1080/01621459.1988.10478694.
[Link],HugoBengio,YoshuaLouradour,JerdmeLamblin,Pascal(2009)."ExploringStrategiesforTrainingDeep
NeuralNetworks".TheJournalofMachineLearningResearch.10:140.
[Link],AdamCarpenter,Blake(2011)."TextDetectionandCharacterRecognitioninSceneImageswithUnsupervised
FeatureLearning":440445.
[Link],HonglakGrosse,Roger(2009)."Convolutionaldeepbeliefnetworksforscalableunsupervisedlearningof
hierarchicalrepresentations".Proceedingsofthe26thAnnualInternationalConferenceonMachineLearning:18.
[Link],YuanqingZhang,Tong(2010)."DeepCodingNetwork"(PDF).AdvancesinNeural...:19.
[Link],MarcAurelioBoureau,YLan(2007)."SparseFeatureLearningforDeepBeliefNetworks"(PDF).Advancesin
NeuralInformationProcessingSystems.23:18.
[Link],RichardLin,Clif(2011)."ParsingNaturalScenesandNaturalLanguagewithRecursiveNeuralNetworks"
(PDF).Proceedingsofthe26thInternationalConferenceonMachineLearning.
[Link],GrahamHinton,Geoffrey(2006)."ModelingHumanMotionUsingBinaryLatentVariables"(PDF).Advancesin
NeuralInformationProcessingSystems.
[Link],PascalLarochelle,Hugo(2008)."Extractingandcomposingrobustfeatureswithdenoisingautoencoders".
Proceedingsofthe25thinternationalconferenceonMachinelearningICML'08:10961103.
[Link],CharlesPerfors,AmyTenenbaum,Joshua(2007)."LearningoverhypotheseswithhierarchicalBayesian
models".DevelopmentalScience.10(3):[Link].1111/j.14677687.2007.00585.x.PMID17444972.
[Link],FeiTenenbaum,Joshua(2007)."WordlearningasBayesianinference".[Link].114(2):24572.
doi:10.1037/0033295X.114.2.245.PMID17500627.
[Link],BoPolatkan,Gungor(2011)."TheHierarchicalBetaProcessforConvolutionalFactorAnalysisandDeep
Learning"(PDF).MachineLearning...
[Link],LiFergus,Rob(2006)."Oneshotlearningofobjectcategories".IEEETransactionsonPatternAnalysisand
MachineIntelligence.28(4):[Link].1109/TPAMI.2006.79.PMID16566508.
[Link],AbelDunson,David(2008)."TheNestedDirichletProcess".JournaloftheAmericanStatistical
Association.103(483):[Link].1198/016214508000000553.
[Link],SalakhutdinovJoshua,Tenenbaum(2012)."LearningwithHierarchicalDeepModels".IEEETransactionson
PatternAnalysisandMachineIntelligence.35:[Link].1109/TPAMI.2012.269.
[Link],RakeshPrincipe,Jose(2013)."DeepPredictiveCodingNetworks".arXiv:1301.3541 .
[Link],Volodymyretal.(2015)."Humanlevelcontrolthroughdeepreinforcementlearning".Nature.518:529533.
doi:10.1038/nature14236.PMID25719670.
[Link],Selfsegmentationofsequences:Automaticformationofhierarchiesofsequentialbehaviors.
IEEETransactionsonSystems,Man,andCybernetics:PartBCybernetics,Vol.30,No.3,pp.403418.2000.
[Link],GeoffreyE."Distributedrepresentations."(1984)
[Link],[Link],[Link],"LearningContextFreeGrammars:LimitationsofaRecurrentNeuralNetworkwithan
ExternalStackMemory,"[Link].,p.79,1992.
[Link],M.C.,&Das,S.(1993).Aconnectionistsymbolmanipulatorthatdiscoversthestructureofcontextfree
languages.NIPS5(pp.863870).
[Link],J.(1992)."Learningtocontrolfastweightmemories:Analternativetorecurrentnets".Neural
Computation.4(1):[Link].1162/neco.1992.4.1.131.
[Link],[Link],[Link],J.(2002)."LearningprecisetimingwithLSTMrecurrentnetworks".JMLR.3:
115143.
[Link](1993)."Anintrospectivenetworkthatcanlearntorunitsownweightchangealgorithm".InProc.
[Link],[Link].191195.
[Link],SeppYounger,[Link],PeterR.(2001)."LearningtoLearnUsingGradientDescent".ICANN.
2130:8794.
[Link],Edward,etal."LearningtoTransducewithUnboundedMemory."([Link]
arXiv:1506.02516(2015).
[Link],[Link],Stefan(1995)."Memorybasedneuralnetworksforrobotlearning".Neurocomputing.9
(3):[Link].1016/09252312(95)000336.
[Link],Ruslan,andGeoffreyHinton."Semantichashing."([Link]
[Link])InternationalJournalofApproximateReasoning50.7(2009):969978.
[Link],[Link],Tomas(2014)."Distributedrepresentationsofsentencesanddocuments".arXiv:1405.4053 .
[Link],Alex,GregWayne,andIvoDanihelka."NeuralTuringMachines."arXiv:1410.5401(2014).
[Link],Jason,SumitChopra,andAntoineBordes."Memorynetworks."arXiv:1410.3916(2014).
[Link],Sainbayar,etal."EndToEndMemoryNetworks."arXiv:1503.08895(2015).
[Link],Antoine,etal."LargescaleSimpleQuestionAnsweringwithMemoryNetworks."arXiv:1506.02075(2015).
[Link],Oriol,MeireFortunato,andNavdeepJaitly."Pointernetworks."arXiv:1506.03134(2015).
[Link],Karol,Andrychowicz,MarcinandSutskever,Ilya."NeuralRandomAccessMachines."arXiv:1511.06392(2015).
[Link],"Recurrentcontinuoustranslationmodels,"inEMNLP2013,2013.
[Link],[Link],[Link],"Sequencetosequencelearningwithneuralnetworks,"inNIPS2014,2014.
[Link],[Link],[Link],[Link],[Link],[Link],"Learningphraserepresentations
usingRNNencoderdecoderforstatisticalmachinetranslation,"inProceedingsoftheEmpiricialMethodsinNatural
LanguageProcessing(EMNLP2014),Oct.2014
[Link],Kyunghyun,AaronCourville,andYoshuaBengio."DescribingMultimediaContentusingAttentionbasedEncoder
DecoderNetworks."arXiv:1507.01053(2015).
[Link],Youngmin(2012)."KernelMethodsforDeepLearning"(PDF):19.
[Link],BSmola,Alexander(1998)."Nonlinearcomponentanalysisasakerneleigenvalueproblem".Neural
computation.(44):[Link].1162/089976698300017467.
[Link],[Link],[Link],[Link]."UseofKernelDeepConvexNetworksandEndToEndLearningfor
SpokenLanguageUnderstanding,"[Link],2012
[Link],Philadelphia.
[Link],[Link].(2014)."ConvolutionalNeuralNetworksforSpeechRecognition".IEEE/ACMTransactionson
Audio,Speech,andLanguageProcessing.22(10):[Link].1109/taslp.2014.2339736.
[Link],[Link],J.(2014)."EnsembleDeepLearningforSpeechRecognition".[Link].
[Link],[Link],L.(2010)."RolesofPreTrainingandFineTuninginContextDependentDBNHMMsforRealWorld
SpeechRecognition".NIPSWorkshoponDeepLearningandUnsupervisedFeatureLearning.
[Link],F.,Li,G.,Yu,[Link],
2011.
[Link].,Li,J.,Huang,J.,Yao,K.,Yu,D.,Seide,[Link]
[Link],2013.
[Link],[Link],Xiao(2013)."MachineLearningParadigmsforSpeechRecognition:AnOverview".IEEETransactionson
Audio,Speech,andLanguageProcessing.21:[Link].1109/tasl.2013.2244083.
[Link],[Link],[Link],[Link],[Link],[Link](2010)BinaryCodingofSpeechSpectrogramsUsing
[Link].
[Link],[Link],[Link](2014).AcousticModelingwithDeepNeuralNetworksUsingRawTime
[Link].
[Link],R."HowSkypeUsedAItoBuildItsAmazingNewLanguageTranslator",Wire,Dec.2014.
[Link].(2014)"DeepSpeech:Scalingupendtoendspeechrecognition",arXiv:1412.5567.
237."PlenarypresentationatICASSP2016"(PDF).
[Link](2015)"Accuracy,AppsAdvanceSpeechRecognitionInterviewswithVladSejnohaandLiDeng",
IEEESignalProcessingMagazine,Jan,2015.
[Link]://[Link]/exdb/mnist/.
[Link],[Link],[Link].,"MulticolumnDeepNeuralNetworksforImageClassification,"Technical
ReportNo.IDSIA0412,2012.
[Link],[Link],[Link],[Link](2013).MitosisDetectioninBreastCancerHistologyImages
[Link],2013.
242."TheWolframLanguageImageIdentificationProject".[Link].Retrieved20170322.
[Link].(2014)."ShowandTell:ANeuralImageCaptionGenerator,"arXiv:1411.4555.
[Link].(2014)."FromCaptionstoVisualConceptsandBack,"arXiv:1411.4952.
[Link].(2014)."UnifyingVisualSemanticEmbeddingswithMultimodalNeuralLanguageModels,"
arXiv:1411.2539.
[Link],[Link],[Link],Y."BilinearDeepLearningforImageClassification".Proceedingsofthe19thACM
InternationalConferenceonMultimedia.11:343352.
[Link]"DeepLearning"([Link]
mosacarcomputertrainedwithdeeplearning/)(20150106),DavidTalbot,MITTechnologyReview
[Link],[Link],[Link],[Link].,"ANeuralProbabilisticLanguageModel,"JournalofMachineLearning
Research3(2003)11371155,2003.
[Link],YoavLevy,Omar."word2vecExplained:DerivingMikolovetal.'sNegativeSamplingWordEmbedding
Method".arXiv:1402.3722 .
[Link],RichardManning,Christopher."DeepLearningforNLP"(PDF).Retrieved26October2014.
[Link],RichardBauer,JohnManning,ChristopherNg,Andrew(2013)."ParsingWithCompositionalVector
Grammars"(PDF).ProceedingsoftheACL2013conference.
[Link],Richard(2013)."RecursiveDeepModelsforSemanticCompositionalityOveraSentimentTreebank"(PDF).
EMNLP2013.
[Link],[Link],[Link],[Link],[Link](2014)"ALatentSemanticModelwithConvolutionalPoolingStructure
forInformationRetrieval,"[Link].
[Link],[Link],[Link],[Link],[Link],[Link](2013)"LearningDeepStructuredSemanticModelsforWeb
SearchusingClickthroughData,"[Link].
[Link],[Link],[Link],[Link],[Link],[Link],[Link],[Link],[Link],[Link],[Link],G.
(2015)."Usingrecurrentneuralnetworksforslotfillinginspokenlanguageunderstanding".IEEETransactionson
Audio,Speech,andLanguageProcessing.23(3):[Link].1109/taslp.2014.2383614.
[Link],[Link],[Link],[Link](2014)"LearningContinuousPhraseRepresentationsforTranslationModeling,"
[Link].
[Link],[Link],[Link],[Link],[Link](2014)"ModelingInterestingnesswithDeepNeuralNetworks,"Proc.
EMNLP.
[Link],TraoreI,WoungangI,ObaidatMS."Authorshipverificationusingdeepbeliefnetworksystems([Link]
[Link]/doi/10.1002/dac.3259/full)".[Link].1002/dac.3259
[Link],[Link],[Link](2014)"DeepLearningforNaturalLanguageProcessing:TheoryandPractice(Tutorial),"
CIKM.
[Link],JMiller,P(2013)."Trialwatch:PhaseIIandphaseIIIattritionrates20112012".NatureReviewsDrug
Discovery.12(8):[Link].1038/nrd4090.PMID23903212.
[Link],BKlambauer,GVervoort,LTalloen,WTheQstar,ConsortiumShkedy,ZThas,OBender,AGhlmann,
[Link],S(2015)."Usingtranscriptomicstoguideleadoptimizationindrugdiscoveryprojects:Lessons
learnedfromtheQSTARproject".DrugDiscoveryToday.20:[Link].1016/[Link].2014.12.014.
PMID25582842.
262."AnnouncementofthewinnersoftheMerckMolecularActivityChallenge"
[Link]
[Link],[Link],N.&Salakhutdinov,R.(2014)"MultitaskNeuralNetworksforQSARPredictions,"ArXiv,2014.
264."Toxicologyinthe21stcenturyDataChallenge"[Link]
265."NCATSAnnouncesTox21DataChallengeWinners"[Link]
[Link]
[Link],[Link],[Link],[Link],[Link],[Link],J.K.&Hochreiter,S.(2014)"Deep
LearningasanOpportunityinVirtualScreening"([Link]
onDeepLearningandRepresentationLearning(NIPS2014).
[Link],[Link],[Link],G.&Hochreiter,S.(2015)"ToxicityPredictionusingDeepLearning"([Link]
[Link]/pdf/1503.01445v1).ArXiv,2015.
[Link],[Link],[Link],[Link],[Link],D.&Pande,V.(2015)"MassivelyMultitaskNetworks
forDrugDiscovery".ArXiv,2015.
[Link],IzharDzamba,MichaelHeifets,Abraham(20151009)."AtomNet:ADeepConvolutionalNeuralNetworkfor
BioactivityPredictioninStructurebasedDrugDiscovery".arXiv:1510.02855 .
270."Torontostartuphasafasterwaytodiscovereffectivemedicines".TheGlobeandMail.Retrieved20151109.
271."StartupHarnessesSupercomputerstoSeekCures".KQEDFutureofYou.Retrieved20151109.
272."Torontostartuphasafasterwaytodiscovereffectivemedicines".
[Link],[Link]
andContinuousActionSpace.(April8,2015).[Link]:[Link]
[Link],Aaron,SanderDieleman,andBenjaminSchrauwen."Deepcontentbasedmusicrecommendation."
AdvancesinNeuralInformationProcessingSystems.2013.
[Link],AliMamdouh,YangSong,andXiaodongHe."AMultiViewDeepLearningApproachforCrossDomainUser
ModelinginRecommendationSystems."Proceedingsofthe24thInternationalConferenceonWorldWideWeb.
InternationalWorldWideWebConferencesSteeringCommittee,2015.
[Link],DavideSadowski,PeterBaldi,Pierre(1January2014)."DeepAutoencoderNeuralNetworksforGene
OntologyAnnotationPredictions".[Link].1145/2649387.2649442viaACMDigitalLibrary.
[Link],Aarti(20160101)."SleepQualityPredictionFromWearableDataUsingDeepLearning".JMIR
mHealthanduHealth.4(4):[Link].2196/mhealth.6562.
[Link],EdwardSchuetz,AndyStewart,[Link],Jimeng(20160813)."Usingrecurrentneuralnetworkmodels
forearlydetectionofheartfailureonset".JournaloftheAmericanMedicalInformaticsAssociation:ocw112.
doi:10.1093/jamia/ocw112.ISSN10675027.PMID27521897.
[Link],[Link],D.J.(2002)."Manylayeredlearning".NeuralComputation.14:24972529.
doi:10.1162/08997660260293319.
[Link].,"RethinkingInnateness,"1996.
[Link],[Link],MH(1996)."Dynamicplasticityinfluencestheemergenceoffunctioninasimplecorticalarray".
NeuralNetworks.9(7):[Link].1016/08936080(96)000330.
[Link],SRSejnowski,TJ(1997)."Theneuralbasisofcognitivedevelopment:Aconstructivistmanifesto".Behavioral
andBrainSciences.20(4):[Link].1017/s0140525x97001581.
[Link].,"Inbrain'searlygrowth,timetablemaybecritical,"TheNewYorkTimes,ScienceSection,pp.B5B6,
1995.
284.{BUFILL}[Link],[Link],[Link].,"Humanneotenyrevisited:Thecaseofsynapticplasticity,"AmericanJournal
ofHumanBiology,23(6),pp.729739,2011.
[Link].,"Timinginthedevelopmentofcorticalfunction:Acomputationalapproach,"[Link]
[Link](Eds.),Maturationalwindowsandadultcorticalplasticity,1995.
[Link].,"TheManBehindtheGoogleBrain:AndrewNgandtheQuestfortheNewAI,"
[Link]
[Link].,"Facebook's'DeepLearning'GuruRevealstheFutureofAI,"
[Link]
[Link].,"[Link]"([Link]
289."GoogleAIalgorithmmastersancientgameofGo".NatureNews&Comment.Retrieved20160130.
 [Link],DavidHuang,AjaMaddison,[Link],ArthurSifre,LaurentvandenDriessche,GeorgeSchrittwieser,
JulianAntonoglou,IoannisPanneershelvam,Veda(20160128)."MasteringthegameofGowithdeepneuralnetworks
andtreesearch".Nature.529(7587):[Link].1038/nature16961.ISSN00280836.PMID26819042.
291."AGoogleDeepMindAlgorithmUsesDeepLearningandMoretoMastertheGameofGo|MITTechnologyReview".
MITTechnologyReview.Retrieved20160130.
292."BlipparDemonstratesNewRealTimeAugmentedRealityApp".TechCrunch.
[Link].,"Is"DeepLearning"aRevolutioninArtificialIntelligence?"TheNewYorker,25November2012.
[Link],G.W.(March27,2015)."ArtandArtificialIntelligence".ArtEnt.RetrievedMarch27,2015.
[Link],Paul(February1,2005)."TheImpossibleCoincidence:ASingleSpeciesModelfortheOriginsofModern
HumanBehaviorinEurope"(PDF).EvolutionaryAnthropology:Issues,News,andReviews.RetrievedApril5,2017.
[Link](June17,2015)."Inceptionism:GoingDeeperintoNeural
Networks".GoogleResearchBlog.RetrievedJune20,2015.
[Link](June18,2015)."Yes,androidsdodreamofelectricsheep".TheGuardian.RetrievedJune20,2015.
[Link]'sDeepLearningAlgorithms?(2015)Url:
[Link]
[Link],Anh,JasonYosinski,andJeffClune."DeepNeuralNetworksareEasilyFooled:HighConfidencePredictions
forUnrecognizableImages."arXiv:1412.1897(2014).
[Link],Christian,etal."Intriguingpropertiesofneuralnetworks."arXiv:1312.6199(2013).
[Link],[Link],D."Astochasticgrammarofimages".[Link].2(4):259362.
doi:10.1561/0600000018.
[Link],G.A.,[Link]."Patternconception."PaperforConferenceonpatterndetection,UniversityofMichigan.
1957.
[Link],DeepLearningofRecursiveStructure:GrammarInduction,[Link]
recursivestructuregrammarinduction/58089/

Externallinks
DeepLearningLibrariesbyLanguage([Link]
DataScience:DatatoInsightsfromMIT(deeplearning)([Link]
1:MITProfessionalX+DSx+2016_T1/about)

Retrievedfrom"[Link]

Categories: Deeplearning

Thispagewaslasteditedon3May2017,at17:37.
TextisavailableundertheCreativeCommonsAttributionShareAlikeLicenseadditionaltermsmayapply.
Byusingthissite,[Link]
oftheWikimediaFoundation,Inc.,anonprofitorganization.