Unit - IV
(Handling large data on a single computer)
Introduction
A large data set is typically defined as a data set that is too large to be
processed by traditional data processing techniques. Handling large data sets
is a complex task that requires a combination of techniques and approaches.
Some of the challenges of managing large data sets include the size of the
data, the complexity of the data, and the speed at which the data is being
generated. One of the biggest challenges in managing large data sets is the
size of the data.
The problems you face when handling large data
In general, a dataset is considered "large" when it exceeds the capacity
of a computer's main memory (RAM). This means that the data cannot be
loaded into memory all at once. A large volume of data poses new
challenges, such as overloaded memory and algorithms that never stop
running. It forces you to adapt and expand your traditional data processing
techniques. But even when you can perform your analysis, you should take
care of issues such as I/O (input/output) and CPU starvation, because
these can cause speed issues.
[Link] Enough Memory
A computer only has a limited amount of RAM. When you try to load
more data into this memory , the OS will start swapping out memory
blocks to disks, which is far less efficient than having it all in memory. The
whole data set load into memory at once, which causes the
out-of-memory error.
[Link] That Never End
While handling large data, in some cases the submitted processes are
not completed quickly and are continuously processed due to the
algorithms, methods and models we use. Moreover, if the process we
submit is not scheduled properly by our computer’s CPU, the process is
running continuously.
[Link] Enough Speed
Traditional processing methods may not be sufficient for handling big
data, as they can be slow and inefficient, Because data is processed and
managed over time. To process large volumes of data quickly and
efficiently, organizations often use distributed computing systems.
General techniques for handling
large volumes of data
Never-ending algorithms, out-of-memory errors, and speed issues are
the most common challenges you face when working with large data.
The solutions can be divided into three categories: using the right
algorithms, choosing the right data structure, and using the right tools.
Choosing the right algorithm
Choosing the right algorithm can solve more problems of handling large
volumes of data. An algorithm that’s well suited for handling large data
doesn’t need to load the entire data set into memory to make predictions.
Ideally, the algorithm also supports parallelized calculations. There are 3
types of algorithms that can do that: online algorithms, block algorithms,
and MapReduce algorithms.
Choosing the right data structure
Algorithms can make or break your program, but the way you store your
data is of equal importance. Data structures have different storage
requirements, but also influence the performance of CRUD (create, read,
update, and delete) and other operations on the data set. There are 3
types of data structure generally used in handling large data. These data
structures are sparse data, tree data, and hash data.
Selecting the right tools
With the right class of algorithms and data structures in place, it’s time
to choose the right tool for the job. The right tool can be a Python library
or R library. Python has a number of libraries that can help you deal with
large data. Most software and tool support a Python interface to their
software. R plays crucial role in the field of data science. Its extensive
set of packages and Libraries for Handling Large data.
General programming tips for dealing
with large data sets
The tricks that work in a general programming context still
apply for data science. Several might be worded slightly
differently, but the principles are essentially the same for all
programmers.
You can divide the general tricks into three parts
1. Don’t reinvent the wheel.
2. Get the most out of your hardware.
3. Reduce the computing need.
Don’t reinvent the wheel
Solving a problem that has already been solved is a waste of
time. As a data scientist, you have two large rules that can help
you deal with large data.
[Link] the power of databases.
[Link] optimized libraries.
[Link] the power of databases
The first reaction most data scientists have when working with large
data sets is to prepare their analytical base tables inside a database. This
method works well when the features you want to prepare are simple.
[Link] optimized libraries
Creating libraries like Mahout, Weka, and other machine learning
algorithms requires time and knowledge. They are highly optimized and
best technologies. Spend your time on getting things done, not on
reinventing and repeating others people’s efforts.
Get the most out of your hardware
Resources on a computer can be idle, whereas other resources are
over-utilized. This slows down programs and can even make them fail.
Sometimes it’s possible to shift the workload from an overtaxed resource to
an underutilized resource using the techniques Feed the CPU compressed
data , Make use of the CPU and multiple threads.
Reduce your computing needs
“Working smart + hard = achievement.” This also applies to the programs
you write. The best way to avoid having large data problems is by
removing as much of the work as possible up front and letting the
computer work only on the part that can’t be skipped.