7.
Find a procedure to launch virtual machine using trystack (Online
Openstack Demo Version)
Aim:
To Find a procedure to launch virtual machine using trystack.
Steps:
OpenStack is an open-source software cloud computing platform.
OpenStack is primarily used for deploying an infrastructure as a service (IaaS)
solution like Amazon Web Service (AWS). In other words, you can make your
own AWS by using OpenStack. If you want to try out OpenStack, TryStack is the
easiest and free way to do it.
In order to try OpenStack in TryStack, you must register yourself by joining
TryStack Facebook Group. The acceptance of group needs a couple days because
it’s approved manually. After you have been accepted in the TryStack Group, you
can log in TryStack.
[Link] Homepage
I assume that you already join to the Facebook Group and login to the dashboard.
After you log in to the TryStack, you will see the Compute Dashboard like:
OpenStack Compute Dashboard
Overview: What we will do?
In this post, I will show you how to run an OpenStack instance. The instance will
be accessible through the internet (have a public IP address). The final topology
will like:
Network topology
As you see from the image above, the instance will be connected to a local
network and the local network will be connected to internet.
Step 1: Create Network
Network? Yes, the network in here is our own local network. So, your instances
will be not mixed up with the others. You can imagine this as your own LAN
(Local Area Network) in the cloud.
1. Go to Network > Networks and then click Create Network.
2. In Network tab, fill Network Name for example internal and then click Next.
3. In Subnet tab,
1. Fill Network Address with appropriate CIDR, for example
[Link]/24. Use private network CIDR block as the best practice.
2. Select IP Version with appropriate IP version, in this case IPv4.
3. Click Next.
4. In Subnet Details tab, fill DNS Name Servers with [Link] (Google
DNS) and then click Create.
Step 2: Create Instance
Now, we will create an instance. The instance is a virtual machine in the cloud,
like AWS EC2. You need the instance to connect to the network that we just
created in the previous step.
1. Go to Compute > Instances and then click Launch Instance.
2. In Details tab,
1. Fill Instance Name, for example Ubuntu 1.
2. Select Flavor, for example [Link].
3. Fill Instance Count with 1.
4. Select Instance Boot Source with Boot from Image.
5. Select Image Name with Ubuntu 14.04 amd64 (243.7 MB) if you want install Ubuntu
14.04 in your virtual machine.
3. In Access & Security tab,
1. Click [+] button of Key Pair to import key pair. This key pair is a public and
private key that we will use to connect to the instance from our machine.
2. In Import Key Pair dialog,
1. Fill Key Pair Name with your machine name (for example Edward-Key).
2. Fill Public Key with your SSH public key (usually is in
~/.ssh/id_rsa.pub). See description in Import Key Pair dialog box for
more information. If you are using Windows, you can use Puttygen
to generate key pair.
3. Click Import key pair.
3. In Security Groups, mark/check default.
4. In Networking tab,
1. In Selected Networks, select network that have been created in Step 1, for example internal.
5. Click Launch.
6. If you want to create multiple instances, you can repeat step 1-5. I created one
more instance with instance name Ubuntu 2.
Step 3: Create Router
I guess you already know what router is. In the step 1, we created our network,
but it is isolated. It doesn’t connect to the internet. To make our network has an
internet connection, we need a router that running as the gateway to the internet.
1. Go to Network > Routers and then click Create Router.
2. Fill Router Name for example router1 and then click Create router.
3. Click on your router name link, for example router1, Router Details page.
4. Click Set Gateway button in upper right:
1. Select External networks with external.
2. Then OK.
5. Click Add Interface button.
1. Select Subnet with the network that you have been created in Step 1.
2. Click Add interface.
6. Go to Network > Network Topology. You will see the network topology. In the
example, there are two network, i.e. external and internal, those are bridged by a
router. There are
instances those are joined to internal network.
Step 4: Configure Floating IP Address
Floating IP address is public IP address. It makes your instance is accessible from
the internet. When you launch your instance, the instance will have a private
network IP, but no public IP. In OpenStack, the public Ips is collected in a pool
and managed by admin (in our case is TryStack). You need to request a public
(floating) IP address to be assigned to your instance.
1. Go to Compute > Instance.
2. In one of your instances, click More > Associate Floating IP.
3. In IP Address, click Plus [+].
4. Select Pool to external and then click Allocate IP.
5. Click Associate.
6. Now you will get a public IP, e.g. [Link], for your instance.
Step 5: Configure Access & Security
OpenStack has a feature like a firewall. It can whitelist/blacklist your in/out
connection. It is called Security Group.
1. Go to Compute > Access & Security and then open Security Groups tab.
2. In default row, click Manage Rules.
3. Click Add Rule, choose ALL ICMP rule to enable ping into your instance, and then click
Add.
4. Click Add Rule, choose HTTP rule to open HTTP port (port 80), and then click Add.
5. Click Add Rule, choose SSH rule to open SSH port (port 22), and then click Add.
6. You can open other ports by creating new rules.
Step 6: SSH to Your Instance
Now, you can SSH your instances to the floating IP address that you got in the
step 4. If you are using Ubuntu image, the SSH user will be ubuntu.
Result:
Thus the openstack demo worked successfully.
[Link] Hadoop single node cluster and run simple
applications like wordcount.
Aim:
To Install Hadoop single node cluster and run simple
applications like wordcount.
Steps:
Install Hadoop
Step 1: Click here to download the Java 8 Package. Save this file in your
home directory.
Step 2: Extract the Java Tar File.
Command: tar -xvf [Link]
Fig: Hadoop Installation – Extracting Java Files
Step 3: Download the Hadoop 2.7.3 Package.
Command: wget [Link]
[Link]
Fig: Hadoop Installation – Downloading Hadoop
Step 4: Extract the Hadoop tar File.
Command: tar -xvf [Link]
Fig: Hadoop Installation – Extracting Hadoop Files
Step 5: Add the Hadoop and Java paths in the bash file (.bashrc).
Open. bashrc file. Now, add Hadoop and Java Path as shown below.
Command: vi .bashrc
Fig: Hadoop Installation – Setting Environment Variable
Then, save the bash file and close it.
For applying all these changes to the current Terminal, execute the source command.
Command: source .bashrc
Fig: Hadoop Installation – Refreshing environment variables
To make sure that Java and Hadoop have been properly installed on your system
and can be accessed through the Terminal, execute the java -version and hadoop
version commands.
Command: java -version
Fig: Hadoop Installation – Checking Java Version
Command: hadoop version
Fig: Hadoop Installation – Checking Hadoop Version
Step 6: Edit the Hadoop Configuration files.
Command: cd hadoop-2.7.3/etc/hadoop/
Command: ls
All the Hadoop configuration files are located in hadoop-2.7.3/etc/hadoop directory as
you can see in the snapshot below:
Fig: Hadoop Installation – Hadoop Configuration Files
Step 7: Open [Link] and edit the property mentioned below inside
configuration tag:
[Link] informs Hadoop daemon where NameNode runs in the cluster. It contains
configuration settings of Hadoop core such as I/O settings that are common to HDFS &
MapReduce.
Command: vi [Link]
Fig: Hadoop Installation – Configuring [Link]
1
<?xml version="1.0" encoding="UTF-8"?>
2 <?xml-stylesheet type="text/xsl" href="[Link]"?>
3 <configuration>
4 <property>
5 <name>[Link]</name>
<value>hdfs://localhost:9000</value>
6 </property>
7 </configuration>
8
Step 8: Edit [Link] and edit the property mentioned below inside
configuration tag:
[Link] contains configuration settings of HDFS daemons (i.e. NameNode,
DataNode, Secondary NameNode). It also includes the replication factor and block size
of HDFS.
Command: vi [Link]
Fig: Hadoop Installation – Configuring [Link]
1
2 <?xml version="1.0" encoding="UTF-8"?>
3 <?xml-stylesheet type="text/xsl" href="[Link]"?>
<configuration>
4 <property>
5 <name>[Link]</name>
6 <value>1</value>
</property>
7
<property>
8 <name>[Link]</name>
9 <value>false</value>
10 </property>
</configuration>
11
12 9: Edit the [Link] file and edit the property mentioned below
Step
inside configuration tag:
[Link] contains configuration settings of MapReduce application like number
of JVM that can run in parallel, the size of the mapper and the reducer process, CPU
cores available for a process, etc.
In some cases, [Link] file is not available. So, we have to create the mapred-
[Link] file using [Link] template.
Command: cp [Link] [Link]
Command: vi [Link].
Fig: Hadoop Installation – Configuring [Link]
1
<?xml version="1.0" encoding="UTF-8"?>
2 <?xml-stylesheet type="text/xsl" href="[Link]"?>
3 <configuration>
4 <property>
5 <name>[Link]</name>
<value>yarn</value>
6 </property>
7 </configuration>
8
Step 10: Edit [Link] and edit the property mentioned below inside
configuration tag:
[Link] contains configuration settings of ResourceManager and NodeManager
like application memory management size, the operation needed on program &
algorithm, etc.
Command: vi [Link]
Fig: Hadoop Installation – Configuring [Link]
1
2 <?xml version="1.0">
3 <configuration>
4 <property>
5 <name>[Link]-services</name>
<value>mapreduce_shuffle</value>
6 </property>
7 <property>
8 <name>[Link]</
name>
9
<value>[Link]</value>
1 </property>
0 </configuration>
1
Step
1 11: Edit [Link] and add the Java Path as mentioned below:
[Link] contains the environment variables that are used in the script to run
Hadoop like Java home path, etc.
Command: vi hadoop–[Link]
Fig: Hadoop Installation – Configuring [Link]
Step 12: Go to Hadoop home directory and format the NameNode.
Command: cd
Command: cd hadoop-2.7.3
Command: bin/hadoop namenode -format
Fig: Hadoop Installation – Formatting NameNode
This formats the HDFS via NameNode. This command is only executed for the first
time. Formatting the file system means initializing the directory specified by the
[Link] variable.
Never format, up and running Hadoop filesystem. You will lose all your data stored in
the HDFS.
Step 13: Once the NameNode is formatted, go to hadoop-2.7.3/sbin directory and start all the
daemons.
Command: cd hadoop-2.7.3/sbin
Either you can start all daemons with a single command or do it individually.
Command: ./[Link]
The above command is a combination of [Link], [Link] & mr-jobhistory-
[Link]
Or you can run all the services individually as below:
Start NameNode:
The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of
all files stored in the HDFS and tracks all the file stored across the cluster.
Command: ./[Link] start namenode
Fig: Hadoop Installation – Starting NameNode
Start DataNode:
On startup, a DataNode connects to the Namenode and it responds to the requests
from the Namenode for different operations.
Command: ./[Link] start datanode
Fig: Hadoop Installation – Starting DataNode
Start ResourceManager:
ResourceManager is the master that arbitrates all the available cluster resources and
thus helps in managing the distributed applications running on the YARN system.
Its work is to manage each NodeManagers and the each application’s
ApplicationMaster.
Command: ./[Link] start resourcemanager
Fig: Hadoop Installation – Starting ResourceManager
Start NodeManager:
The NodeManager in each machine framework is the agent which is responsible for
managing containers, monitoring their resource usage and reporting the same to the
ResourceManager.
Command: ./[Link] start nodemanager
See Batch Details
Fig: Hadoop Installation – Starting NodeManager
Start JobHistoryServer:
JobHistoryServer is responsible for servicing all job history related requests from client.
Command: ./[Link] start historyserver
Step 14: To check that all the Hadoop services are up and running, run the below command.
Command: jps
Fig: Hadoop Installation – Checking Daemons
Step 15: Now open the Mozilla browser and go
to localhost:50070/[Link] to check the NameNode interface.
Fig: Hadoop Installation – Starting WebUI
Congratulations, you have successfully installed a single node Hadoop cluster
Result:
Thus the Hadoop one cluster was installed and simple applications executed
successfully.