Scaling in the Cloud

Scaling:
 Scalability of systems and applications is an essential feature of cloud computing.
 Computing cost depends on the total volume of resources acquired by an application.
Any acquired and unutilized resource unnecessarily increases computing cost. Again, low
acquisition of resource may affect application performance during higher demand. Hence,
anysystem must run with minimum volume of required resources and should have the ability
to expand itself with growing workload which is critical from business point of view. Again,
a system should also have the ability to reduce itself with declining workload in terms of
acquired resources. Otherwise unnecessary resource acquisition increases the cost.
 This ability of expanding and shrinking of a system as per workload is known as scaling.
 Scaling is the characteristic of a system, model or function that describes its ability of growingor
shrinking whenever required.

 In simple words, scaling is defined as the ability of being enlarged (or shrunk) for
accommodating growth (or fall-off) to fulfill the business needs. A system or application
architecture can be termed as scalable if its performance improves on adding new resources
and the improvement is proportional to the capacity added.
 The scalability of a scalable system is measured by the maximum workload it can competently
handle at any particular moment. The point at which a system or application cannot handle
additional workload efficiently, any more, is known as its limit of scalability.

 Scalability reaches its limit when a system’s architecture does not support scaling anymore or
some critical hardware resource run out. Resource components which generally limit scalability are
processor or memory of application server and disk I/O rate of database server.
 In the traditional static scaling approach, computing system requires a ‘restart’ for the scaling effect to
take place which causes service disruption.
 Dynamic scaling enables a system to keep performing consistently during times of massivedemand by
expanding it at pace with growing demand.
 Service providers can create the illusion of infinite resources during service delivery, as cloudconsumers
remain unaware about the transparent scaling feature.
 Downward scaling is more critical to implement than upward scaling. In downward scaling, the
challenge is to maintain performance while releasing resources. Thepresence of downward
scaling capability has made cloud computing more attractive.

 Scaling in cloud is dynamic and automatic. The auto-scaling facility provides enormous
advantages, both to the service provider and the service consumers.
 They can both gain in terms of service availability (during peaks), cost and business. This
auto- scaling feature in cloud computing has been achieved based on three pillars:

■ Resource virtualization: It eases resource management tasks and reduces the complexity of
system development.
■ Resource sharing: It allows the optimal resource utilization by sharing resources among
multiple users as well as applications.
■ Dynamic resource provisioning: It supplies (or reclaims) the resources on-demand, in
response to the increase (or, decrease) in workload.
A system’s ability of scaling may depend on various parameters. Among them, the two important
 ones are:

■ Capacity planning on regular basis and

■ Load balancing

It is not possible to take full advantage of the scalable computing infrastructure if the application architecture
is not scalable. Both have to work together to maximize the gain.

SCALING STRATEGIES IN CLOUD:

Critical task of dynamic capacity alteration can be done in two ways:

■ Manually: when a system can be scaled while running by executing appropriate commands
through the application interface.
■ Automatically: when this type of scaling of the system can be implemented throughprograms
that can automatically adjust system capacity by observing the actual demand.
The dynamic auto-scaling is generally referred as auto-scaling which is also known as cloud scaling.
Auto-scaling can be implemented in two different ways:

■ Scaling based on a predefined schedule known as proactive scaling.

■ Scaling based on current actual demand known as reactive scaling.
Dynamic scaling of a system can be managed both manually and automatically. Auto-scalingfrees a system
from any manual involvement for adjusting resources.

Proactive Scaling:
Scaling strategy that does not wait for workload to change, rather alters capacity in advance based on
a pre- defined schedule is known as proactive scaling.
These types of known situations of demand change can be categorized into two types. Separate
planning is required to deal with those two types of situations. Hence, the proactive scaling
schedules are implemented in two different ways as
 Proactive cyclic scaling: This type of proactive scaling event takes place at fixed regular
intervals and by pre-defined times of the day, week, month or year.
 Proactive event-based scaling: Major variations in traffic load may occur due to some
scheduled business events like promotional campaigns or new product launch and else.
Proactive scaling strategy does not wait for demand to increase or decrease in expectedcircumstances.
Such standard situations are handled through pre-defined plans.

Reactive Scaling:

 In this strategy, the system reacts immediately to changing demand of resources by adding or removing
 capacity on its own. Here, the decision is taken based on resource utilization. When utilization of
processor or memory or some other resource reaches a certain threshold, more of that resource can be
added into the environment by the system itself without any external intervention.
 Under this scaling technique, depending on a situation where the suitable parameters are identified
at first to activate the auto-scaling process. System scales in response to the changing conditions of
those parameters.
 This eliminates the need for any pre-scheduled action to handle scaling as it always remains
unknown when those conditions may change.

Reactive scaling should be applied as the last layer of protection to scale a system and shouldnot be used
unless unavoidable.

Auto-scaling implementation requires mixture of both reactive and proactive scalingapproaches.

AUTO SCALING IN CLOUD:

In auto-scaling (also called as cloud scaling) mechanism, the system itself can increase the required
resource capacity, automatically (and dynamically), when the demand of workload goes up. The
mechanism can also release and return resources to the free pool when they are no more required.
Auto-scaling facility helps to maximize resource utilization by automatically doing the work of
scaling.
Scaling in these two situations happens in following fashion:

■ Unpredictably, based on specified conditions and

■ Predictably, according to defined schedule
The monitoring and the checking modules of auto-scaling unit play vital roles in the auto- scaling
process. Monitoring module keeps on sending the load status to the checking module at regular
intervals, while checking module decides the appropriate action depending on the current load status
and stored pre-defined schedules.

To protect system from external attack, auto-scaling mechanism is often limited by settingboundaries.
Although, beyond those boundaries the scaling is controlled manually.

TYPES OF SCALING:
There are two different ways of supplying and integrating the hardware resources into a system. Depending
on this, the scaling approaches can be categorized into two types as vertical scaling and horizontal scaling.
1. Vertical Scaling or Scaling Up:
One way of increasing resource capacity is to replace the existing component (processors, memory etc.) with a
more powerful hardware component. Thus, the capacity can be increased (or decreased) by upgrading (or
reducing) a resource quality to support raising (or falling) workload. This type of scaling where a resource
component is powered up through replacementis called scaling up or vertical scaling. For example, a
system with dual-processor capacity canbe scaled vertically by replacing the processor with a quad-
processor.

Advantages: It is not a complex task to replace a component of a system since it does not require
alteration in the system architecture. So, vertical scaling approach has low management complexity
and is less risky.
Disadvantages: High power hardware components or specialized resources are [Link] or
power of any resource component always has an upper limit. Hence, a sufficiently capable resource may
not always be available or if available, the specialized hardware components may not be even
affordable. In vertical scaling, the overall system capacity is enhanced by replacing resource componentswithin
existing nodes.
 2. Horizontal Scaling or Scaling Out:
Resource capacity of a system can be increased by introducing additional resources (and without
elimination of the existing). These new resources work together with the existing components. This type
of scaling where the system is equipped with supplementary resources to expand theresource capacity
is known as scaling out or horizontal scaling. For example, a system with a dual-processor capacity can
be scaled horizontally by adding another dual-processor.

Advantages: This approach does not involve any system downtime. Specialized high-end hardware
components are not required to scale out a system; generally available commodity hardware are used for
horizontal scaling here. And more importantly, this scaling approach is not limited by an individual
component’s capability as multiple similar type components (i.e. multiple processors or multiple memory
units etc.) synchronize together in order to act as a powerful component. Unlike vertical scaling, no resource
components are thrown out of the system (through replacement).
Disadvantages: This approach needs managing large number of distributed nodes whichneedto synchronize
together, and hence involve complexity. Presence of heterogeneous nodes further increases the complexity. The
other issue is that horizontal scaling is appropriate for running distributed software applications; hence existing
applications developed to run in centralized environment are needed to be redesigned for a distributed
computing environment.

Horizontal scaling helps to build an economical computing system as powerful resourcecomponents can be built
by combining multiple ordinary resources.
Cloud Bursting Architecture:

The bursting generally erupts out to a public cloud but it may also tend to a private cloud. The cloud bursting
architecture is suitable for internal computing systems where high volume of traffic occurs for a short period of
time.
 Capacity Planning
 Capacity Planning in computing is basically developing a strategy which guarantees that
at any moment, the available or arranged resources will be sufficient to support the actual
demand for resources and that too at the minimal possible cost.
 The goal of capacity planning is to identify the right amount of resource requirement to
meet the service demands at present and also in the future.
 Capacity Planning in computing is a process that determines the future requirement of
computing resources to provide desired levels of service to a given workload at the least cost.
 In traditional computing, end users had little participation in the capacity planning
process.
 In reality, cloud environment gives the illusion of infinite computing resources available
on-demand. The creation of this illusion about infinite resources marks the difference
between success and failure for a service provider. The delivery model of resources in
cloud computing can indeed provide what may seem to appear to be an infinite supply of
resources on-demand, but it comes at a cost, and that is why we need proper capacity
planning.
 The illusion of infinite computing resources is possible with a business model where one
service provider has many clients and an enormous number of applications to support
their single or multiple connected data centers.
 The service provider needs to have a clear idea about the resource needs of all clients in
advance, so that it can arrange enough resources to support all applications without a
resource crunch at the time of high demand. This is the philosophy behind capacity
planning in cloud computing.
 For successful capacity planning the consumers must also provide a rough idea in
advance to the service provider regarding their varying resource requirements
throughout the year. This helps providers plan accordingly and reserve an adequate
amount of resources for every application during high demand periods.
 In cloud computing, capacity planning is done at two levels. At the first level, each
consumer of a cloud service does his own capacity planning. At the second level, the
cloud service provider analyzes the capacity requirements of all consumers together and
makes the ultimate overall capacity planning.
In this regard, cloud service consumers can be divided into two groups.
■ IaaS consumers generally plan and reserve a fixed capacity of resources they would consume
for a period. This drives them to do their own capacity planning task, but only for the virtual
resources they consume.
■ SaaS and PaaS consumers generally opt for a dynamically metered resource use model.
Hence their declaration about service requirements noted in the service level contracts (SLA
contract) is considered very important for the provider in the capacity planning process.
■ Cloud Computing enables consumers to pay only for what they get. However, consumers should
also manage their demands so that they consume only what they plan to pay for.
Capacity Planning at Different Service Levels

Role of Service Level Agreement

Service level agreement should address the expected and acceptable level of services.
Service providers take the pain of planning for capacity management in cloud to create theillusion of infinite resource
capacity for their consumers.

Traditional fixed cost capacity model suffers from the problem of over-provisioning and under-provisioning of resources.
Both of these are costly for any business.
 The variable cost operating model can closely match the resource capacity with actual demand and it thus decreases cost of
computing by reducing resource wastage.
APPROACHES FOR MAINTAINING SUFFICIENT CAPACITY:
1. Traditional Approach with N+1 Rule
Traditional capacity planning approach suggests maintaining one additional resource nodethan actually required to save
the system during crisis.
2. Cloud Specific Approach
Cloud infrastructure service with virtual resources offer a more flexible and scalable approach, where
auto-scaling mechanism always maintains sufficient resource capacity for applications to serve
legitimate loads. Auto-scaling relieves the capacity planners who are cloud service consumers from the
wearisome job of monitoring and maintainingsufficient capacity, and the system manages capacity itself.
Cloud infrastructure service providers (IaaS providers) build up the service with actual physical
resources and hence cannot go beyond the traditional approach of capacity maintenance.
CAPACITY AND PERFORMANCE:
With capacity, the concern is about how much work a system can do, whereas withperformance, the concern is the rate at
which work gets done.

STEPS FOR CAPACITY PLANNING

Step 1. Determining the expected demand – In the first step of capacity planning process the service
provider must carefully examine the expected overall resource usage patterns as they vary
over a course of period.
Step 2. Analyzing current response to load – Next, the service provider must analyze the available
resource capacity of their system and how the applications respond to load (or overload)
with current capacity, so that any requirement of additional capacity that is to be added can
be identified.
Step 3. Knowing the value of the system – Finally, the service provider must be aware about the
value of the systems to the business, so to know when adding more capacity provides value
and when it doesn’t.
Cloud providers need to maintain a balance between the financial burden of over-provisioning, and the reputation
risk of under-provisioning of capacity.

1. Determining the Expected Demand:

To determine the expected demand for resources of a system, capacity planners need to
determine individual demands of all the applications supported by the system. Later they canaggregate
those results to get idea about traffic pattern of the whole system. Understanding thedemand pattern of
application is important as the traffic pattern generally changes with time.

2. Analyzing Current Load:

System load can be analyzed by measuring the load of different system resources. Several
system metrics are used to represent these loads. In computing, the main resources used are –
processor, memory, and storage and network connectivity. Hence the major system parameters are
processor speed, memory access speed, disk I/O access speed and network I/O access speed. Loads on
each of these resources can be measured by operating-system-specific tools.

It is the goal of a capacity planner to identify the critical resource that has resource ceiling, and resolve the
problem to move the bottleneck to higher levels of demand.

3. Value of System Capacity:

Every application adds some value to the business. It is important to know how more capacity
can help the business before adding any more resources to the system just because it has hit the
stress point.

The decision regarding capacity enhancement depends on, how much it is going to cost to addcapacity against
the value of that additional capacity to the business.
LOAD BALANCING:
 Load balancing in distributed computing systems is an essential technique to
distribute processing and communication activities evenly across the resources in
the network so that no single computing resource gets overloaded.
 Load balancing is especially important for applications which often deal with a large
and unpredictable numbers of service requests. Load balancing is a technique that
distributes load evenly among multiple computing resources such as processor,
memory etc.
 Through load balancing, the incoming service requests are distributed among
available computing resources. An efficient load balancing mechanism improves
average resource utilization rate and therefore enhances the overall performance of a
system.
 The load balancing capability is created in a computing system by crafting
the systemwith suitable architecture. Such architecture helps the system attain
additional capability. For example, with load balancing architecture, the
additional capacity can easily be added into computing system by introducing
multiple instances of similar resources.
 Load balancing shows its real strength by making a system more productive when
additional capacity is induced into the system.
 A load balancer distributes the workloads across multiple similar types of
computing resources, such as processor, memory, storage and network switches.
In cloud computing, service requests can be distributed among multiple servers,
either located within a data center or distributed geographically. The objectives
of load balancing are to optimize resource utilization, maximize the throughput,
minimize the response time and avoid overloading of any resource.
IMPORTANCE OF LOAD BALANCING IN CLOUD COMPUTING:
■ Load balancing offers architectural flexibility and essential help in making a
computing architecture scalable.
■ It ensures efficient utilization of a pool of similar type resources.
■ Efficient resource utilization automatically enhances the performance of the overall
system.
■ The technique decouples applications from its physical implementation during
execution. This creates a layer of abstraction which increases application and
system security.
■ The decoupling of physical resources from direct access of applications also makes
the cloud computing system more tolerant in cases of any component failure.
 Load balancing is one key issue of cloud computing because overloading of any
resource may lead to poor system performance.
 service load balancing which distributes application service requests among
resources is vital for the success of cloud computing.

 The load balancer system in a cloud implementation that directly interfaces with
clients is called the front-end node. All the incoming requests first arrive in this
front end node at the service provider’s end. This node then distributes the
requests towards appropriate resources for further execution. These resources
which are actually virtual machines are called as back- end nodes.
  In cloud computing implementation, when the load balancers at the front-end node
receive multiple requests for a particular service from clients, they distribute those
requests among available virtual servers based on some defined scheduling algorithms.
This scheduling happens depending on some policy so that all of the virtual servers
stay evenly loaded. This ensures maximum as well as efficient utilization of physical and
virtual resources.
 In the figure it has been assumed that all of these three virtual servers are
equally loaded before six similar types of service requests appearing before the
system (from same or different consumers). The incoming requests encounter
the load balancer first. Load balancers use scheduling algorithm and distribute
the requests among these three available virtual servers. Here, the load
balancers act as the front-end-nodes for all of the incoming requests and the
virtual servers act as the back-end-nodes.
 The use of multiple similar resources with load balancing technique incorporated,
instead of a single powerful resource, increases the system’s reliability through
redundancy.

Two Levels of Balancing:

1. VM Provisioning and
2. Resource Provisioning
 VM provisioning is the mapping of service requests with virtual machines.
 Resource provisioning is the mapping of virtual machine loads with physical
resources.
What are to Load Balance:
Following is the list of resources and services which are mainly responsible for load
balancing in cloud computing environment:
■ Application server instances (which include processor and memory)
■ Storage resources
■ Network switches
■ Services such as DNS, TCP, HTTP and HTTPS

GOALS OF LOAD BALANCING:

To improve system performance: In a load balanced system, no resource should get
overloaded when others are under-utilized. All resource components remain almost
evenly loaded or uniformly free. This improves the performance and stability of the
overall system.
To maximize fault tolerance: In a load balanced system, while multiple nodes work
together, that enhances the tolerance of the system against the faults. If some node
stops functioning due to any kind of hardware or software failure the traffic is
automatically redirected among other working nodes by keeping the system up. Thus,
the users of such system remain unaffected and unaware of the failure.
To accommodate scaling: An efficient computing system needs to scale as application
demand grows or declines. One objective of load balancing is to support scaling by
properly redirecting load to the newly introduced nodes in case of growing or by
releasing load from nodes in case of shrinking.
To ensure availability of applications all the time: Another objective of load balancing
is to keep the applications available all the time. As higher level of fault tolerance
can be achieved and overloading of nodes can be avoided by routing traffic across
the less-utilized resources this ensures availability of application all the time.
CATEGORIES OF LOAD BALANCING:
[Link] Approach:
Static load balancing does not use any knowledge base and distributes service requests only
based on the characteristics of the requests. It matches the service request characteristics
to decide about suitable resource type for allocation based on some set of rules. No prior
knowledge about the current state of the running system is needed to implement static load
balancing.

The static approach to load balancing does not consider the current load state of
the systems and allocates loads based on some fixed set of pre-configured rules
related with the nature of the input traffic. The scheduling algorithms are of round
robin and weighted round robin categories which generally have been used with this
approach to schedule the loads among resources.
[Link] Approach:
The dynamic approach of load balancing technique does not distribute loads among
resources depending on any fixed set of rules. Scheduling of tasks usually happen
based on the existing state of the resources or nodes. The advantage of using dynamic
load balancing is that if any node gets fully loaded or fails, it will not halt the whole
system.
This approach is implemented by applying feedback mechanism where the resource
components are monitored continually. The nodes periodically send load and status information
to the load balancer and if any node becomes non-responsive during operation, the load
balancer stops sending traffic to it.
 Dynamic load balancing actively monitors the load levels of resources through a
feedback channel and adjusts load distribution to keep all resources evenly loaded.

Distributed load balancing mechanism provides more robustness to a system at the cost
of some added overhead.
In non-distributed dynamic load balancing approach, all the nodes of a distributed
system do not take part in balancing system load. One or few nodes are assigned the
tasks.

PARAMETERS FOR CONSIDERATION OF LOAD BALANCING:

Resource Utilization: All of the available resources have to be utilized in a planned manner, such
that any single resource does not become over-utilized and could not degrade the
performance of the whole system.
Response Time: It is the time required for a specific load balancing algorithm to respond. Since
service requests appear in large numbers from clients in popular cloud services, the special
attentions are evidently required in improving the performance of the services of the system.
The first step towards this goal is to minimize the response time of the load balancer.
System health: This is related with some sort of component failure. System health should be
maintained by periodically checking the efficiency of its nodes. In case of poor performance
or failure of a node, a load balancer should take prompt action (by releasing or avoiding that
node till recovery) to maintain the system performance.
Associated Overhead: There is always some overhead associated with load balancing
activity. This overhead is introduced by the movement of tasks among nodes and inter-
processor communications being required to execute the tasks. One criteria of designing good
load balancing technique is to minimize this overhead.
Scalability: Load balancing enables scaling while the number of resource nodes in a load
balanced system may increase or decrease with time to support the demand of applications.
Load balancer should have the capability of handling this scalability issue efficiently.
LOAD BALANCING ALGORITHMS:

Class-agnostic load balancing algorithms make decisions without considering the nature
and source of incoming requests.
Content-aware load balancer can redirect similar kind of incoming requests to the same
back- end-server to avoid duplicate processing.
File System and Storage
Cloud computing promises high-performance. Hence, the file system and storage to
support high-performance data processing are critical requirements of cloud
environment.
The data processing model must sync with the distributed file system architecture.
Sophisticated storage techniques are needed to reduce access time.
CHALLENGES BEFORE CLOUD NATIVE FILE SYSTEM:
Multi-tenancy: Cloud system offers multi-tenancy and allows sharing of its
underlying resources among multiple tenants. Especially in a public cloud, consumer
share resources withothers who are unknown to them. Hence, the file system in cloud
must ensure that tenants (tenants’ processes and data) remain isolated from one
another to provide higher degree of security.
Scalability: A cloud file system must scale well so that users can rely upon the system
with their growing storage needs. At the same time downward scaling is also important
to minimize the resource wastage.
Unlimited storage support: File system support for unlimited data storage is another
need of cloud computing for its business success. Moreover, the file system has to be
extremely fault- tolerant. And, all of this needs to be achieved by building storage
network out of inexpensive commodity hardware.
Efficiency: The other performance parameter to be counted over the others is the
system’s output. While dealing with thousands of concurrent operations issued by
many clients, to achieve the requisite performance like the local file-system is a
critical issue.
Compatibility: Compatibility is always an issue when new technologies are
introduced. In the domain of computing file systems, backward compatibility with
existing file system interfaces is important to facilitate the migration to the cloud
seamlessly.
Metered Use: Metered use of resources is one of the basic requirements in cloud
computing and for storage too. The file systems used in cloud have to enable and
promote this capability.
Cloud native file system is one that enables the efficient implementation of cloud
computing capabilities like application scaling, metering of storage usage and else.
MODEL FOR HIGH-PERFORMANCE PROCESSING OF LARGE DATA-SETS:
Map Reduce Programming Model:
‘MapReduce’ is a programming model or framework that supports developers to write those
applications which could process massive amounts of unstructured data in parallel across a
distributed processing environment. It was developed at Google in 2004 in order to quickly
analyze the vast amount of content the search engine giant was collecting through public
websites, to apply their famous PageRank algorithm aiming to decide which websites are most
worthy of showing up in searches.
Development of parallel-programming model needs to handle the tasks like
scheduling, intra-cluster communication and task monitoring by the developer. But, by using
the library routines of MapReduce programming model, the developers can create parallel-
programming effortlessly without worrying about all of these issues. The MapReduce model
takes care of everything. MapReduce framework was inspired by the map and the reduce
primitives of the ‘Lisp’ (List Processing) programming model. Lisp is the second oldest high-
level programming language (after Fortran).
MapReduce is based on the idea for parallel processing of data-intensive applications,
where data-set is split into blocks and assigned to different instances. Those instances run in
parallel to process the data-set. After completing the computation works, the second phase
starts where the intermediate results produced by individual instances are merged to produce
the final result.
These two modules of the framework act in the following manner:
■ Map, a function that divides data processing task in a distributed environment and
generates the intermediate results.
■ Reduce, a function that merges the intermediate results and resolves them into final
result.
These functions accept data as input and then return transformed data as output. Actually,
the master node takes data as input, splits it into smaller sections and sends them to other
associated nodes in the cluster. These nodes may perform the same operation in turn to send
those smaller sections of input to other nodes. MapReduce thus allows for massive scalability
across hundreds or thousands of nodes.
MapReduce model accepts input as a key-value pair and first splits those data
into multiple sets of key-value pairs. These intermediate sets of data are then
distributed across different nodes for further and parallel processing.
At last, those processed data sets are reduced into the required result. The
process can be described well through a simple example. Let there be a file containing
data about daily minimum temperatures in five different cities over a year.
The task is to find the lowest temperature of each city during that year. Such file
can be broken into multiple smaller files; say in ten files and can be distributed among
different nodes for processing. Here, it is to note that each smaller file may contain
data about all of five cities and at the same time each file can have multiple entries for
one city. Now the task of each of the ten processing nodes are reduced to find the
minimum temperature of each city from the data-set as present in smaller file assigned
to it. Let one processing node generates the intermediate result in the forms of
following key-value pairs.
(city 1, 3) (city 2, –10) (city 3, 12) (city 4, –2) (city 5, –8)
Such intermediate result will be generated by each of other nine nodes. The next task is
to accumulate all of these data for all of ten processing nodes which may look like as shown
below.
(city 1, 3) (city 2, –10) (city 3, 12) (city 4, –2) (city 5, –8)
(city 1, 0) (city 2, –4) (city 3, –1) (city 4, 3) (city 5, –9)
(city 1, 6) (city 2, 2) (city 3, 5) (city 4, –7) (city 5, –1)
…
Now the task of finding the minimum temperature of each city for that year is
reduced into finding the result in the above data-set which is very simple. In this way
the MapReduce programming model automatically parallelize processing for large
volume of data-sets.
CLOUD NATIVE FILE SYSTEM:
Any storage system builds up above a file system. Hence, to support high-
performance computing for process-intensive as well as data-intensive tasks, an appropriate
file system development is the first step. Such file systems meet the demand of cloud
computing also as cloud computing is entirely meant for high-performance computing.
[Link] General Parallel File System:
It was one of the earliest effort towards high-performance distributed file system
development. IBM’s General Parallel File System (GPFS) started as the Tiger Shark
file system, a research project at IBM and became available on IBM’s AIX (a series
of proprietary Unix operating systems developed by IBM) in 1998.
[Link] File System:
The real revolution came in high-performance distributed file system development with
emergence of the Google File System (GFS). It was developed during late 1990s. The file
system was a result of an earlier Google effort called as Big Files. The main concern of the
GFS designers was to ensure scalability of the system. They also enhanced reliability by working
out mechanism against failures and errors. The file system was developed after detailed
analyses of the characteristics of large files those are often stored in cloud environment. The
access modelof the system was designed in appropriate manner to minimize the storage and
access time.
GFS connects a very large distributed cluster of inexpensive commodity components using
high-speed network connections. GFS files are collections of fixed-size segments called as chunks.
GFS uses the MapReduce programming model to split each large file into chunks of 64 MB
size. Each chunk consists of 64 KB blocks. The larger chunk size increases the probability of less
number of chunk being accessed to perform some operation and thus reducing the processing
time.
A GFS cluster consists of two different types of nodes (Figure 13.1). One special node which
is called as master server and a large number of other nodes known as chunk servers. The chunk
server stores all of the GFS chunks while the master server maintains information about the
associated chunks.

3. Hadoop Distributed File System

 Hadoop Distributed File System (HDFS) is an open-source implementation of the
Google’s GFS architecture as developed by Apache Software Foundation. The
development was initiated by Yahoo in 2006 being inspired by the Google’s GFS
and MapReduce papers and was looking to develop an open-source based system to
fulfill their storage requirements.
HDFS is a Java-based distributed file system that provides scalable and reliable data
storage. Like GFS, HDFS is also designed to run on large clusters of commodity servers.
The file system emerged as a sub-project of the Apache Hadoop project. HDFS is
extremely fault-tolerant and provides high output access to application data. The file
system is available for consumers on the Amazon EC2 cloud platform.
HDFS is designed to reliably store very large files across multiple machines in a large
cluster. HDFS cluster contains two types of nodes as one single master node called as
NameNode and other slave nodes called as DataNodes. Files are broken into sequence
of blocks (similar to chunks in GFS) of reasonably bigger size (64 MB or 128 MB
generally). These blocks are stored on DataNodes commodity servers (similar to chunk
servers in GFS).
Like other distributed file systems, HDFS also assumes that nodes may fail. Hence,
to increase fault tolerance of the system, it replicates blocks over multiple DataNodes.
By default it uses 3 replicas. The block size and the replication factor are configurable.
During read operation, data is fetched from any one of the replicas. During write
operation, data is sent to all of the DataNodes containing replicas of the file. Master
node usually stores metadata about the blocks.

4. Ghost Cloud File System

Ghost is a fully featured, inexpensive and scalable private cloud file system. It was
designed to run within consumer’s own Amazon web services account. Ghost Cloud
Computing was founded in 2006. Its name is an acronym of Global Hosted Operating
System. In 2009, it released its own cloud file system to run on AWS, known as Ghost File
System.
 5. Gluster File System
The Gluster File System (GlusterFS) is an open-source distributed file system capable of
scaling. GlusterFS was developed originally by Gluster, Inc. and afterward by Red Hat,
Inc., when they purchased Gluster during 2011. Like other cloud file systems,
GlusterFS also allows creating a single volume of storage which spans multiple disks,
multiple machines and even multiple data centers.
6. Kosmos File System
Kosmos File System (KFS) was released in 2007 as an open-source implementation
of Google File System being developed in C++ programming language. It was
developed by American company Kosmix for the UK Government’s G-Cloud
services. It paralleled the Hadoop HDFS project which is developed in the Java.
Currently the file system is known as CloudStore.
STORAGE DEPLOYMENT MODELS:
Depending on the deployment locations, cloud storages can be categorized into three
models as public, private and hybrid. This is just similar to the standard cloud
computing deployment models.
Public cloud storage as the name specifies can be accessed by anyone. It is
provided by reputed service providers. In this model, the consumer enterprise and
storage service provider are two different organizations.
Private cloud storage is arranged by the consumer enterprise itself with help of some
service providers. This type of storage can be set up both on-premises and off-
premises. In this model, the consumer enterprise and storage service provider integrate
each other at enterprise’s data center or provider’s data center.
The hybrid storage model is the combination of both public and private storages as
its name implies. In this model, the enterprises get the opportunity of storing critical
and active data in private cloud storage while public storage can be used for archiving
data.

STORAGE TYPES:
Cloud storage is categorized according to the purpose of use such as general storage
purpose or computing system development purpose.

Managed and Unmanaged Cloud Storage:

 Managed storage system provides raw disk like facility to users. User can divide
or format the storage spaces as per their requirements and can also install the
software. This type of storages is mainly meant for computing system developers.
 Unmanaged cloud storage is like ready-to-use disk drive. Users directly get storage
capacity available for use. All of the primary disk management tasks like
partitioning, formatting and else are managed by the vendors or service providers.
Since users of such storage need not to worry about managing the storage space,
it is called as unmanaged storage. Here, users have very little control. Storage
 provider decides the nature of the storage and applications through which the
space can be accessed.
 Managed cloud storages are delivered as Infrastructure-as-a-Service, whereas
unmanaged storages are delivered as Software-as-a-Service.
POPULAR CLOUD STORAGES FOR DEVELOPERS:
[Link]’s Elastic Block Store
Amazon Elastic Block Store (EBS) provides block level storage volumes for use with
Amazon EC2 (Elastic Compute Cloud) instances (servers). EBS was introduced for
general public by Amazon in 2008. Block level storages can be used to create raw
storage volumes which can be attached with the servers. Variety of file systems like
NTFS (for windows OS) or ext4 (for Linux OS) can be run on the block level storage.
EBS is like a massive SAN (Storage Area Network) under the AWS infrastructure.
Delivered storage volume size goes up to TB in size.

[Link]’s Simple Storage Service

Simple Storage Service (S3) was introduced by Amazon as a cost-effective web service
solution for developers in the year 2006. In S3, files are stored as objects and those
objects are stored into the containers called as ‘buckets’. Object size can go up to few
terabytes and trillions of objects are stored per month as reported by Amazon. S3 can
be used together with Amazon’s virtual server Elastic Compute Cloud (EC2).
[Link]’s Cloud Storage
The persistence storage attached with Google’s cloud server Google Compute
Engine (GCE) is the Cloud Storage of Google. There data are stored as objects and
objects are stored into containers called as the ‘buckets’ as well. Objects can be of
terabytes in size and does not have a size limit. Billing in Google Cloud Storage is
calculated as per storage usages as well as the bandwidth usage on monthly basis.

4. Rackspace’s Cloud Block Storage

Rackspace’s Cloud Block Storage is a block-level storage solution and is associated with
Rackspace Cloud Server. But users can create or delete multiple storage volumes independent
to the cloud servers they are attached with. Storage volume size in Cloud Block Storage may
range up to TB. The storage service is charged as per gigabytes of storage used per month.
POPULAR GENERAL PURPOSE CLOUD STORAGES:
[Link]
Dropbox is a file hosting service offered by Dropbox Inc., United States. The service
was officially launched in 2008. Dropbox provides client applications through which the
storage can be accessed from different personal devices. It is accessible from Windows, Mac,
Linux using desktop applications and also from Android, iOS and the BlackBerry platforms
using mobile apps. In the background, Dropbox uses Amazon’s S3 storage system to store the
files.

[Link] Drive
This is a popular file hosting service offered by Google. The service was launched in
2012. Google Drive provides file sharing facility with other users. Currently, it provides 15 GB
of free storage space for users. Google Drive has the benefit of a built-in office suite where one
can edit documents, spreadsheets and presentations. Google Drive client software is available
for both desktop platform like Windows, Mac as well as mobile platforms like iOS and
Android.
Google Drive storage is for general users and Google Cloud Storage is meant for the
developers.

[Link]
OneDrive is a general purpose cloud storage service provided by Microsoft. The
service was first launched in 2007 and was previously known as ‘SkyDrive’ or ‘Windows
Live Folders’. OneDrive is built into Windows 8 and 8.1 operating systems. It is also
accessible from Mac OS and earlier versions of Windows OS and mobile platforms like
Android, iOS and Windows Phone. Apart from file storage, OneDrive offers the facility for
document creation and collaboration. OneDrive’s biggest strength is that it works closely
with Microsoft Office apps such as Word, Excel or PowerPoint.
[Link]
Box is an online file hosting, sharing and collaboration service. The company Box Inc.
is based in California of United States. While Dropbox like services are popular among
individuals, Box is popular among business and enterprise users. Box offers 3 types
of accounts as Enterprise, Business and Personal. Users can determine how their
content can be shared with other users.

[Link]
Copy cloud storage service was introduced by corporate IT company Barracuda
Networks in 2013. Like other cloud storage services, Copy also has desktop software for
Windows, Mac and Linux in addition to mobile apps for iOS and Android. It provides
15 GB of free storage space.

[Link] Cloud Drive

Amazon Cloud Drive is a file hosting service offered by Amazon. The storage can be
accessed from different mobile devices and computers. It currently provides 5 GB of
free storage space to consumers.