Scribd
Upload
6 views76 pages

Notes

The document outlines the installation and configuration of an Apache Hadoop multi-node cluster on Ubuntu 20.04, detailing steps for setting up master and slave nodes, configuring Hadoop files, and starting the necessary daemons. It also covers HDFS commands and YARN administration commands for managing the Hadoop environment. The document concludes with a brief overview of building a MapReduce application for counting word frequency using Eclipse.

Uploaded by

Uzaifa Attar
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
6 views76 pages

Notes

The document outlines the installation and configuration of an Apache Hadoop multi-node cluster on Ubuntu 20.04, detailing steps for setting up master and slave nodes, configuring Hadoop files, and starting the necessary daemons. It also covers HDFS commands and YARN administration commands for managing the Hadoop environment. The document concludes with a brief overview of building a MapReduce application for counting word frequency using Eclipse.

Uploaded by

Uzaifa Attar
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
PageNo | GD) tExpriment wore [ee * Title - Thstallation of Apache HadooP 336 1) Ubuntu 20 44 (mult node Icluster setup) Mulbinode Hadoop cluster as composed oF master lave Architecture to accep ishment of Bigdato processing wich “contains multiple nodes so we oe going ty Consider three machines for setting up Hadeop cluster Here there 15 eonrelaticn betmeen Dumber oF computer in cluster with “ee oF data processing technique | F nich will be our Masternode | setting Up afresh ubnode iadeorma| (2.36) cluster. Fellow the aiven steest prerequicites ~ Inatallation | Configuration oF node. Hadoop single nstall f coal ‘que Single nede Hadoop Cc) ca [Prepare your computer nehoork Cecide number of nodes to sehup cwste [A av @| Basic instatiaton } Configuration — Based cn helping parameter like | Putpcse or Hadoop multinode cuister | Size oF -the nedeset 1 be puceaed labilty: OF machine Step 3A: Hostname ident Acation Your modes to be Configured 14 the [ name Wt as Hadeopmaster and B 2 Further Steps tO mastemode ,we will different slave noder HediE the [ete [hosts Ale with Following eommand gudo gedit Jetcl hosts lt paca Following hostname t-their jp 10 host table — 192 168-24 —Hadooprtaster, Hade op Hixster| Hadesp Steves Skep 98 Create hadeop a: group f | P hduser as user in all machines: Date aa } oN, Kanishka@ Hadcopttaster of sudo addgrcup | _ hadoop Kanisnka@HadoopMaster :1 $5ud0 addus er — ingreup hadoop hdlus SF you require to add hduser te cudeers then file following command sudo _usermed —a ~G sudo “nduser Slep ge: gnstall, rayne for Sharing hadea| Source with rest oll Machines ude apt get install Feync Step 8p 10 Make dbove change; retected we need to Teboot ali of the pnachi nes Sudo reboot © + | Hadoop Configuration steps — | Apelying Common Hadoop — Confiquration “However We will be Confiaurabng Master slave architecture we need te apely the common changes in hadoop | centig Files —_ TACLEST, Warananagar os 2 Changes Update core xml update this le by changing hostname from localhost te HadoopMaser ey -Fire the below Given commard cat | hadoop] e-cite xen! To edit hduser(@)Hadooptaster :/user| etc} hadoop4 sudo gedit ¢ Paste these lines Into CconFigurah en? fag or just update tt by replacing localhost with master & name vfs -deFautt name ha fs t//Hadoop aster goo 0 ai Waren Update this File by updating the Following three properties by updating hostname From localhost tp Hadeoptiaster | hduser@HadoopMaster fuser| locat Tnadoop 4} sudo gedit yara-site xml Update mapred=cite xm) _ Updale this File by updating padding Following properbes = nd user@ HadoopMaster : (user local | Tadoopl kel hadeop 4 sudo gedit rapyved- -site xml 4 propertu> Zname prmmapreduce job tracker 4 arp® create wordcount Reducer. class | import jova io. OEXception $ nd replace with below Code terabhon, teeport java ut Import 0g apache hacloop+ 10.4 5 import era - epeiche hadee? -mapved’ x5 (Pane Ce) pet | [ Public void gmp Wierd Count Reducer __|extends mapeReduceBase_ impleents = Sneed Reducer < Text int Writal _towntable> ae | Public Void reduce CText, key mterator values outputceliector - Text Ent writable> output Repet I reporter) throws TOExcepbon a + | int sum=o5 7* foal result as the key fsum oF Values */ 7 while C values hasnext O) i Gum ta values NEXtO-JetO 5 a output -collect (Fey newantWritable | Csums) 5 _ a a _y | |_| stepo: ; “IF abwe java classes are not Having any syntax error, the corresponding class Fle will be | ‘genera tea cuttomertic a! y as follows [Link]., Warananagar Page No) oa No step 9 - Addibonally before step 6 we have to add dependencies by adding external libraries from Hadoop » fellow J-ine below screenshots and added extemal jar from path . 7 Now click of the Run tab and ¢ chek en New Run - Configuratigns ame | [configuration button end Al the © Project Name and Main class per wereen-shots Sep g: Now @ight click on project ond elect. Export . Under lava , select ee ee a select | “the config file you created instep & f ~ Rerncve” temp Fle generated tv start all required hadoor | e namenode ,datanede s | qracerully | deamons rescurce manager, applicabien manager, secondary dame node TACLEST, Warananagar [owe pie Start pmeess ond check if reqw reed graceFull 4 amo has been & 1 not using ips command . Please pote, if temp files are there and fot femoved 1Few of eleamens will not be started properly —_ step} Run the job Using helow command hadoop jar Word Count Jar] Usr] local] nputThadccpteutput atep 1+ Brows the Hadeop Gut http cl local hoet ' S0070/dF shea tt tab- overview Cutput Fle step 16: Brows the localhost: $0070} explorer htm] +t user/local | hadoop/ output http TRLET. Warananagat step ir: Stop all deamons iF you ave done with job using stop-auish “Conclusion - In this expriment we have Sbidied that building hadeoP Map | [Reduce Application Fer counting Frequenty of word] Phrase 14 ‘ CREATE DATABASE [Link] [30 case a database alteady exist With the same nome, a HOW To avoid error ye Command : we > CREATE TAaBie [Link] - temp. toble (2. 1Nb) 5 or is icon use Fn addition you can also create a table in the database by usifg the “Ere following Cemmand- nveS vse [Link] hive DCREATE TRaLt temp-table crnviney, | You can ape tre form of Friiowing — rein ify PBPROPERIES iy value pair 7 the | We CREATE PATABASE [Link] Wh DECKER TEES Cother! =" Jaspel’, HAIgAM Technology’), + cornpan OERLEST, Warananagar Niewing in database - Meu can view all the database | in particular path oy using _Command « hive> snoy DATABA SEC 5 Dropping patahase — Dropping a databace means delebng it From its Storage locahon “Mme clatabase can be deleted by USING following command: hive> CROP DATABASE ZF EXISTS +temp-database You must Mot 1f 4Ane clatabase to be deleted contain any table you Arst need to Alelete tables Aum “the dabkabase «using the drop commans| Hive allows you to detete tables | [along with cthe database by using | cornreand « [ nive> oROP OATABASE Lemp-database | © C ASC APE peste cee ere eee en em Bitering | (earasaece Alfecing a database means Mahing changes 0 oxisnng clatabase| one con oteeneaaaniee “by using” hive 5 ALTER DATABASE temp database} [creating Tables — _You_can create table in databal jf sé by USING the create coromand | _| Use temp- databace> CREATE TABLE IF NOT Exists employec| eNaMes SFATNG | Salary FLOAT, alesignaben STRING) a ——_ __ | -TROLE PROPERTIESC tomnevic Cereated att =! os joalzols'y | foenn ccvmmteo € TENS “TER MINTED BY we LINES TERMINATED BY Mani tus er|hivelwathouse | LocATION I temp. data base .db/employed | “[ etoRep AS Texter! ee Page No DS ee ot No Parbihioning Tabies t —__Neu_can create par btions ina able “by using the paRTIONEO | [ey Key word CREATE “TABLE _emmpoyee C enome sTRING , Sal FLOAT) | [ PARTITIONER BY Cdesignaton €TRING)5 2 [PROPPING Tables: you Can delete table by fo\Vowing commands: Wive> CROP TABLE IF EXISTS StidentE, + [Altering -mbles + Altering table! means meal Fyig ~g of changing an existing table =e table can be modiFied by the ALTER, _ TABLE Statements » Rename “fable Pe modbry cclumns * Add new columns . "Delete some Columns. change Table Prper tes Alber table - fer adding prpeities. | x) aie ee ee Rename “Table = p_YOu can rename table by * Following command. SSS VED BREW ALTER THBLE employee RENAME To employee -New > Modify eolumns : - HIVE > ALTER TABLE employee Neo CHANGE ColumMN ename , emp-na STRING: Addl celumn : _hive PALTER TABLE emplryee new ADD couumn CSalea month); Replace Colurons : ALTER TRBLE employee -new - REPLACE COULUMNS (ename STRING, | prjectname STRING 4 PlocatHon apr) conclusion — Fo Wis exprin [have “abidied about the overview oF we ppl commands like create. ater 5 clelete etc. (Paene ap) C2) Expiment noise fem 9 sh bee Title- Handling the database using vadeoP_ Hive DML commands like Tneert. HA _dlelete, » update, date, vebrivod a venes cand jo-inner and cuter * DML 19 10 “Hive - aH “Af tev apearying the datahese schem [ and creating a database schema and _ creat ~ing a database fhe data can be modified by using set of pmcedure | mechanism defined by special language known as data manipulabon language ¥ | snsert staternent — “The insert into statement is P insert Mew Tecords in a table used to ID Specifying beth column - name 4 value to be inserted aH insert Inte table-name Ceo sicols —__[vatues 0% 15 vol2 VAIS) 5 @ Jr you are adding “fer all columa oF | fable, you do Not need to specify ¢O Ta as 1h 8Gl query fp rnsert 10t0 tables name Values Val, va wo Delete command - ww Hive son “he The delete operahed aAaable for hive Nv “Pdate operation canbe performed on Aable that support -the AcrD property update Yable name et column= Value [ col=val ..J Cwhere expression) bata retrival queries Hive cuter you te perform data tetvat queries by using select commond along with various types of operator ana clauses * select © Whe + Grupby + having ment most comm | (can ite | Dw OF both eg = select + From employee @| where - - The Where clase is used to search records mF a table on the kas of given condition Daa Mis Clause Feburas a beolean | value [resutt- 3 Ss—~—S~SS for ex- A quety;can return only those Sates records that have an amount greater than isoo fom Us region eX- Select * From Sales WHERE amount 7 1S008 AND region="Us". | ®| Grove ve - The group by Clause f¢ used _| bm put cul the related fecorel s together EEE Core’ s Together| dt_can also be used with aggregate “Function + —— Ex - ¢F you wish to calculate the Gvero. —— 9€ morks obtained by the skident from Shident frm all semester €9- SELECT Gem) avaC Marks) fro __GRouP BY sem, c _ F@ | heving clause - The hoving clause oF is used to | Specify condition on use of gfour by | Clause - the use of having Clause 15 | added 10 the fersien 0.7.0. 0F hive [eg - Select ¥ From mydemotable lint > Executing hive queries — Lets peiform following step to ques Strwke-data File To tetrive data stored in the Steck | [aata table in a particular data you can | ype following Hive @L command an termine! © using yoin in Hive @ Innev Join @ Outer Join nner Join - Zn case of inner join, only the lrecord stahsfuing the giver conditon the other ~vecords leet get Selected out - g ow TLE, Warananagar ——— een eee arene POSES —) 4 |orser Table. _ver orderid 12 BoD i Taspeh 20 - Lise Cust—icl, «cust -1d oO) Ps order id [Link] f Bob as - Soseph a Lise | Outer Ton: yeu feed 40 tebive oul record | From one table te_enly same ¢ fecerd another table from ~ Peg- Select order ,Ordevid , customer Cust__ | Thame from order © join customer F< SS outer JoIO are th © Right outer joie _ @ vert cuter Jon | @ Fut _euker Soin EEE D[ Right outer yoin- (a te—} oP > Order id cust_nome | | 2 bob a Joseph 3 lise 4 monty a) LerE outer Join - Ht All the fecord From ta | the lert Side of Join are vetaine d 6g - select older ofleridl customer cust Fromm Oreler © left Outer join |———syetomer € 00 0. Cust-id edule S| Full Outer son a st — AW the Field frern beth side ie tables are factuded a qa Cd = aa [select order order id, customer [Link] from orde~ o Full outer jain customer _| Go cust-id= corstid); order id cust id y both Joseph aw! 2 3 4 Lise = nus at mony, | conclusion - [eae We have Studied that ail above | [Prive prt command Tike insert delete, | | Upelate + s Joins. ae a I Se EV ree ae Page No. 33 ‘Thonn e/ You 3 auf hadooP 1 Pg. [data using Pigsterage a | Strut = FOREACH “Student GENERATE DUMP output, | oo The result of the preceding acript is | a follows - CTohn, 8rd C Robinson ,2,™) Cwily, GF Cpetter, 7) © | ASSERT — The ASSERT operator assets a [| conebbon on the data, Assertions c | used for ensuring that condibon jc ome "er the fecords: | When assertion -fa)s ASSERT alias by expression [Link] | FO the preceding syntax - Tablas -Refers to the game of relaton 7 8Y Refers +e keyu = expression — Refers to a bodead Pera expression ; message - Refers to error rn 685age | €9- temp=.oAp tdata’as Ghret int, second int, thirdint) Dump temps output : C2534) C4112) CBD CS. 812d Group - Te syntax of GROUP Operator io Fig i6 Similar Fo SQL .but itis] cbfFerent in funcBonatity when compote [ea He Grove BY clause In s@L To pig +he GRovP operater is used grouping data if single multiple yelation a EEE EEE See RACE, Warananagar (om ‘Page No fo ) fee AUAS = GROUP cutiag FALLIBL expre® Poatias AU | By eapressiont T Usana collected! limerge' CPARU OW By particular) CPAPALEL nj, Alias = Referers too name of a table ALL ~ Refers to keyword used For muthiple tnpubrg all types oes SY = ReFers to keyword used for rouping velation py Fret , tuple OF expression PARTITION By Partitioner- Describe the hadeop partiponer used for cantioling the keys. eg - : x = LOAD 'Shident' As Cname chorarray page: int), = DESCIBE x, - xi dame char crray sage vat} oume x) Caeorge 120) truk = C lisa, 2) ¢ Paul, 29) Lf Fo soph ey ENLET, Warananagar O10 operator in P19 13d aoe Ng two of more reriben — _t toner Foin Preece ett * Ouker Foiq Sree auias 3 Join atlas gy pexpression| Ha | "expression Lrexpeession— J)/F salias BY fexpression | 'C/expressioo Cyexpression-..J')'7 [USane ‘repti zate}) 'Skewed' | ‘meme'| ‘merge -spaise J alias — Refers to Name of Relaten BY — Refers to Keyword USING~ ReFers to Keyword teplicated ~ perform replicated sein Skewed — Perform Skewed join merge — petform merge join [tees ie gen Sheree Per korm merger Sparse tty merge -Sparse - Perform merge - sparse joie Inner Toi - FN a inner jo the game dataset 15 used b& cuerent Nays. grant >datal =joad Tmydata! _2fant> datas = load tmydato’ grant >result = join dat 1b1 te, _ _ data 2 by 405 “When you, U_ implement outer SOIT | 1® Pig record that have a cmakch oF the record in relation table are jocluded with Aull value Left outer jon - Je returns all the rows that +Ror left table ,even if there ace no mnatches in fight table are included wieh Hull values- B bys a nestle 2) (2 » tage toP 2) Ct topreman \ Vite 13) lgrant > < =S0iIN A by SO LEFT OUTER | @| crest - The LIMIT operator in Pg ONOWS A User to limit the Nurober OF result. | students! As | second: int third 104 Pig = {| crest DUMP temp 5 Wie temp =LaMsT temp2a Dump Umt-Femp 4 Output = De C2121d (1 22 Se @| SAMPle - TA SAMPLE Operator 19 p Pig | je used for selecting a random data Ty ponding o ample Sample size. 4p Pp —Treturn tre percentage in WS in double values = | een | | abas n3 7 =e EERE 7 ' Refers to mame pf relation + [9% Refers to cample size ,which can be| constant that Fangei from 6 tL __| ef @ Scaler used in expression f © @ Sscalor used fo consider “he example in which table 2 will hav 11 of the dala gecenk (A table xz X= LOAD ‘dato’ as CArst ‘int 5 second : int, third sint)5 == SAMPLE ¥ 0-0) # | conclusion - Io -+this experiment we have | Studied dbout working OF operators IN Apache ‘Pig — FOCEACH » ASSERT | FILTER, 1C¢ROUP » ORPER BY » DISTIN FOTN , SAMPLE - TBtle: The Installation of @ studio 200 Se mPle mentation of basic concepts [ # [How 4 Install Rand Retadlo oh Linux s * | Fnstall Rand Rstudio on linux > __Unux software is often distrub | uted as a source code and then Compiled by package managers like apt Or UUM. Te install @ in ubunti. we “have & 90 through the following Steps: Sudo apt-get install +-base ing the command +a cbf mation prompt Will appear Answer Ni for yes ai Bice ceee ecco eeceaae hing RS pdt >| | “te inate patti, goto dewnlead| Rstudio , click on the downloa button For Retudio , dekstop r¢ the link for the latest R ver io0 | for_your Os and save the -deb File Step2: Download and install the gbeb! Package using the Following Commands sudo apt install qHebi Answer with a 'N'-for yes to confirm when prompted: pees ee Shep 3: Use the following commands “to [step 3; Use sre Pree install the oleb Package sudo _geebi /path| tol tne! Filed deb | Tostatiing R and Re “To jostan eg Ron windo “Go 0 GRAN R project website Jetep2: Click 09 the Download R For windows Tink Stepa: Click on the base subdirectory link or install © for the fest windowd ~GEXW stond for the latest ver isvane sores executable exe File eteps St pun the -exe fle and follow _the _in slatlatien iOShuction “select “the designed language ee NE. Warananagar ——— Sibi Read the licence agreement [and Next, S.c! Select the browser Components You wish to install click Next S-di Enterlbrawse the e Folder [path — YoU wish to Install Rint o0d then confirm by clicking Next S.@; Wait for the Install complete: Zp) etek on finish te Complete the _installabon % [Hostal etude on windows Gtepst: with Rbase installed sleks m1 go te downloads Rstield. K for the “winded “oF Retudio and save Bali Walt For the in InStallaben proces te complete. \8\ \a Installing R and PStidio of Mac OS X To install R and Rsbidio 00 Macosy go through the Following steps Frstati QR on mac: Istepl> Go to CRAN PB project Website K 69 tHe Download forc¢mac) osé im on link for the pkg File, | save Tt File and follow instedlatien Tinetuchens - 7 ee “concepts 10 | Functionalibes» Atithmatic Operations t- — Addition - this operationg «4' 15 used a operator and retuens the additions oF two operands: S+2. # Addition - vision - This operation «> as a operator- Cla herds To]z ¥ division Expene atiati [Fo this operations ‘A! Is os used) cand Tetuin 3 walls. Le) ieee ———____ AS greaponent ation ee Sting in Re Any value uoritken within a par oF cingle quote or double quotes 1" R |'s treated as a Shing “+ fexample of valid Stwings - @< start end end with single quack! print(a) [Sting Manipulation ~ Goncating Slings - pasta o> Funchen. Syntax - — pacte( 21 Sepe” collapse = NULL) pop example - L__fexannple = and shings — [syntax - format CX, oligits nsmalls ecient , Justify = ("Lert wright” mre Counting number of characters 10 & String = Achar() Funchon syntax ~ Nehar CX) [ ex [ result < Achar( "count the number of character") | print Cresult) [eurput > [csod ‘Changing -the » CASE - touppe! OJ blower) funcbon : icone 7 eyataxe +E ___ | touprer Oo [te lower cx) 7 pet — i ee Warananagar = exampl Fesult < toupperc "changing To upper Prink( teste) 3. per) lresw ie ata sohalehallataltab i uit etolowerC" changing to Lower’) | Print Cresult) ging te lower olp— CHANGING To UPPER Changing to lower HE Now *msg! fs back In the workspace| 4sO HH Output "msg" y" veadwoll O Function ~ Fe prompts the user te enter a pame "and ots it IN the Varicble ‘Name’ | mn & as. feger(M) F Cis. nam) £ ne “eeadroll setuen (ny 3 [create Some variable - ees y< so msg < "Hello, wortd _ [RUSE object in the workplace - [ [aso — tt output -"msg "en, ty a HF save the variable ~mag”™ i Workspace rM(msg) /3] | {>} the Ite List Objects again te conArm mag? load | variable mag bact to the | Werkspace From the hile _| Heed message aar) a Namea eadiine € prompt "Enter o @ name’) pant (name) [ Per € readline ( prompt } Enter Percentage ) print Cper) cat CEnter a city" city < scan(Ffle =" " “Dhake charactesllen gth.}o) 1AMax =!) print Carty) #| Conclusion > “| we have stubed that istatrahed OF R studio aNd Implementation of — basic concepts 19 R : We aa 1. Warananagar | | Ovectors — th datasets ite ond R Title - Working withR t —create .tead ,u tables -create,Read write creating pata sets - You can create using the . ec! Functon This Funcbon can take muitipie arguments to create a vector ex- Ne cUi6 6134 21412) @oata frames - pata Frames are Used te : [store tabular date-you can create Q data fomes using the ‘data Fam] functor ex: data ~~ dlatahome (subject = (ha 3,4) clase CC ¢21b2) marksle (gy 4.8979) OMAP, Warananagar

You might also like