MODULE 3

Module Introduction
In this module, you will learn about the different components that make up a modern-day data
ecosystem. You will learn about the different types of data structures, file formats, and sources
of data. You will also gain an understanding about the languages data professionals use in their
day-to-day work.

Learning Objectives
After completing this module, you will be able to:

• Describe the different components of a modern data ecosystem.

• Explain the different types of data structures, file formats, and sources of data.
• Explain the features and use of the different languages used by data professionals.
• List some of the tools data analysts use for data wrangling, cleaning, and loading data into
repositories.

Overview of the Data Analyst Ecosystem

A data analyst's ecosystem includes the infrastructure, software, tools, frameworks and processes
used to gather, clean, analyze, mine and visualize data. In this we will go over a quick overview of
the ecosystem before going into the details of each of these topics in subsequent lec. let's first
talk about data. Based on how well-defined the structure of the data is Data can be categorized
as structured, semi-structured or unstructured.
Data that follows a rigid format and can be organized neatly into rows and columns is structured
data. This is the data that you see typically in databases and spreadsheets for example.
Semi-structured data is a mix of data that has consistent characteristics and data that doesn't
conform to a rigid structure. for example, emails. An email has a mix of structured data such as
the name of the sender and recipient but also has the contents of the email which is unstructured
data.
And then there is unstructured data. Data that is complex and mostly qualitative information that
is impossible to reduce to rows and columns for example photos, videos, text files, pdfs and social
media content. The type of data drives the kind of data repositories that the data can be collected
and stored in and also the tools that can be used to query or process the data.
Data also comes in a wide-ranging variety of file formats being collected from a variety of data
sources ranging from relational and non-relational databases to API's, web services, data streams,
social platforms and sensor devices. This brings us to data repositories a term that includes
databases, data warehouses, data marts, data lakes and big data stores. The type, format and

A data ecosystem in data analytics refers to the collection of infrastructure, tools, and applications used to
capture, store, analyze, and leverage data. It allows organizations to analyze raw data, gain insights, and
make data-driven decisions.
 sources of data influence the type of data repositories that you could use to collect, store, clean,
analyze and mine the data for analysis.
If you're working with big data for example, you will need big data warehouses that allow you to
store and process large volume high velocity data and also frameworks that allow you to perform
complex analytics in real time on big data.
The ecosystem also includes languages that can be classified as:
A query language, also known as data query language or database query language,
i. query languages,is a computer language used to make queries in databases and information systems.
ii. programming languages A programming language is a set of instructions and syntax
used to create software programs
iii. shell and scripting languages.
A shell script is essentially a text file containing a sequence of commands that the shell executes. These commands can perform
various operations such as file manipulation, program execution, and text printing
Querying and manipulating data with SQL
To
Developing data applications with python
&
Writing shell scripts for repetitive operational tasks (these are important components in a data
analyst's workbench).
Automated tools, frameworks and processes for all stages of the analytics process are part of the
data analyst's ecosystem. From tools used for gathering, extracting, transforming and loading
data into data repositories to tools for data wrangling, data cleaning, analysis, data mining and
data visualization it's a very diverse and rich ecosystem.
Spreadsheets, Jupiter notebooks and IBM cognos are just a few examples. we will cover some of
the data analytics tools in greater detail in subsequent sections of the course.
1 Types of Data
Data is unorganized information that is processed to make it meaningful. Generally, data
comprises of facts, observations, perceptions, numbers, characters, symbols and images that can
be interpreted to derive meaning. One of the ways in which data can be categorized is by its
structure. Data can be structured, semi-structured or unstructured.
1. Structured data has a well-defined structure or adheres to a specified data model.
o Can be stored in well-defined schemas such as databases and
o In many cases can be represented in a tabular manner with rows and columns.
Structured data is objective facts and numbers that can be collected, exported, stored and
organized in typical databases. Some of the sources of structured data could include:
1 • SQL databases and online transaction processing or
Structured Query Language (SQL) (pronounced S-Q-L; historically "sequel") is a domain-specific language used to manage
data, especially in a relational database management system (RDBMS). It is particularly useful in handling structured data,
OLTP (Online Transaction Processing) systems: Are accessible data processing systems used in enterprises.
Handle day-to-day queries involving inserting, updating, and deleting data. Facilitate real-time transactions in a database.
Manage routine and daily business operations.
Web server logs are text documents that contain a record of all activity related to a specific web server over a defined period of
time
A zipper is a technique of representing an aggregate data structure so that it is convenient for writing programs that
traverse the structure arbitrarily and update its contents, especially in purely functional programming languages.
2• OLTP systems that focus on business transactions
3• Spreadsheets such as excel and google sheets
4 • Online forms
5• Sensors such as global positioning systems or GPS
6• Radio frequency identification or RFID tags
7• Network
8• Web server logs
You can also easily examine structured data with standard data analysis tools and methods.
2. Semi-structured data is data that has some organizational properties but lacks a fixed or
rigid schema. Semi-structured data cannot be stored in the form of rows and columns as
in databases. It contains tags or elements or metadata which is used to group data and
organize it in a hierarchy. Metadata (or metainformation) is "data that provides information about
other data
Some of the sources of semi-structured data could include:
Markup languages are computer languages that are used to
1• Emails structure, format, or define relationships between different
2• XML and other markup languages, parts of text documents with the help of symbols or tags
inserted in the document.
3• Binary executables, TCP/IP is the network protocol used by most networks, and it
4• TCP or IP packets, requires information to be split into packets that contain both a
segment of data to be transferred and the address where the
5 • Zippered & data is to be sent
6• & Integration of data from different sources.
XML and JSON allow users to define tags and attributes to store data in a hierarchical form and
are used widely to store and exchange semi-structured data.
3. Unstructured data is data that does not have an easily identifiable structure and
therefore:
o Cannot be organized in a mainstream relational database in the form of rows and
columns.
o It does not have any particular format, sequence, semantics or rules.
Unstructured data can deal with the heterogeneity of sources and has a variety of business
intelligence and analytics applications. Some of the sources of unstructured data could include:
1• Web pages,
2• Social media feeds,
3• images,
4• Varied file formats such as jpeg, gif and png,
5• video and audio files,
6• documents and pdf files,
7• PowerPoint presentations,
8• media logs, &
NoSQL is a type of database management system (DBMS) that is designed to handle and store large volumes of unstructured
and semi-structured data.

9• surveys.
Unstructured data can be stored in files and documents such as a word document for manual
analysis or in NO SQL databases that have their own analysis tools for examining this type of data.
To summarize structured data is data that is well organized in formats that can be stored in
databases and lends itself to standard data analysis methods and tools. Semi-structured data is
data that is somewhat organized and relies on meta tags for grouping and hierarchy and
unstructured data is data that is not conventionally organized in the form of rows and columns in

2 a particular format. in the next video we will learn about the different types of file structures

Understanding Different Types of File Formats

As a data professional, you will be working with a variety of data file types and formats. It is
important to understand the underlying structure of file formats along with their benefits and
limitations. This understanding will support you to make the right decisions on the formats best
suited for your data and performance needs.
Some of the standard file formats that we will cover in this video include:

1• Delimited text file formats,

2• Microsoft Excel Open XML Spreadsheet or .XLSX,
3• Extensible Markup Language or .XML,
4• Portable Document Format or .PDF,
5• JavaScript Object Notation or. JSON

1. Delimited text files are text files used to store data as text in which each line, or row, has
values separated by a delimiter; where a delimiter is a sequence of one or more characters
for specifying the boundary between independent entities or values. Any character can
be used to separate the values, but most common delimiters are the comma, tab, colon,
vertical bar, and space. Commaseparated values (or CSVs) and tab-separated values (or
TSVs) are the most commonly used file types in this category. In CSVs, the delimiter is a
comma while in TSVs, the delimiter is a tab. When literal commas are present in text data
and therefore cannot be used as delimiters, TSVs serve as an alternative to CSV format.
Tab stops are infrequent in running text. Each row or horizontal line in the text file has a
set of values separated by the delimiter and represents a record. The first row works as a
column header where each column can have a different type of data. For example, a
column can be of date type, while another can be a string or integer type data. Delimited
files allow field values of any length and are considered a standard format for providing
straightforward information schema. They can be processed by almost all existing
applications. Delimiters also represent one of various means to specify boundaries in a
data stream.
requests via HTTP (the network protocol created to distribute web content) or its

2. Microsoft Excel Open XML Spreadsheet, or XLSX, is a Microsoft Excel Open XML file
A web server is computer software and underlying hardware that accepts

format that falls under the spreadsheet file format. It is an XMLbased file format created
by Microsoft. In an .XLSX, also known as a workbook, there can be multiple worksheets.
And each worksheet is organized into rows and columns, at the intersection of which is
the cell. Each cell contains data. XLSX uses the open file format, which means it is generally
accessible to most other applications. It can use and save all functions available in Excel
and is also known to be one of the more secure file formats as it cannot save malicious
Malicious code refers to harmful computer programming scripts that can exploit system
code. vulnerabilities and cause unwanted changes, damage, or unauthorized access to computer systems
3. Extensible Markup Language, or XML, is a markup language with set rules for encoding
data. The XML file format is both readable by humans and machines. It is a self-descriptive
language designed for sending information over the internet. XML is similar to HTML in
some respects, but also has differences. For example, an .XML does not use predefined
tags like .HTML does. XML is platform independent and programming language
independent and therefore simplifies data sharing between various systems.
4. Portable Document Format, or PDF, is a file format developed by Adobe to present
documents independent of application software, hardware, and operating systems, which
secure variant HTTPS

means it can be viewed the same way on any device. This format is frequently used in
legal and financial documents and can also be used to fill in data such as for forms.
5. JavaScript Object Notation, or JSON, is a text-based open standard designed for
transmitting structured data over the web. The file format is a language-independent data
format that can be read in any programming language. JSON is easy to use, is compatible
with a wide range of browsers, and is considered as one of the best tools for sharing data
of any size and type, even audio and video. That is one reason, many APIs and Web Servers

3 return data as JSON.

Sources of Data
As we touched upon in one of our previous videos, data sources have never been as dynamic
and diverse as they are today. In this, we will look at some common sources such as:

• Relational Databases,
• Flat files and XML Datasets,
• APIs and Web Services,
• Web Scraping,
• Data Streams, & Relational Databases
• Feeds Internal sourcing
Typically, organizations have internal applications to support them in managing their day-to-day
business activities, customer transactions, human resource activities, and their workflows. These
systems use relational databases such as SQL Server, Oracle, MySQL, and IBM DB2 to store data
in a structured way. Data stored in databases and data warehouses can be used as a source for
analysis. For example, data from a retail transactions system can be used to analyze sales in
different regions and data from a customer relationship management system can be used for
making sales projections. External sourcing
External to the organization, there are other publicly and privately available datasets. For
example, government organizations releasing demographic and economic datasets on an ongoing
basis. Then there are companies that sell specific data, for example, Point-of-Sale data or financial
data, or Weather data, which businesses can use to define strategy, predict demand, and make
decisions related to distribution or marketing promotions, among other things.
Such data sets are typically made available as flat files, spreadsheet files, or XML documents.
1. Flat files, store data in plain text format, with one record or row per line, and each value
separated by delimiters such as commas, semi-colons or tabs. Data in a flat file map to a
single table, unlike relational databases that contain multiple tables. One of the most
common flat file formats is CSV in which values are separated by commas.
2. Spreadsheet files are a special type of flat files, that also organize data in a tabular format
– rows and columns. But a spreadsheet can contain multiple worksheets, and each
worksheet can map to a different table. Although data in spreadsheets is in plain text, the
files can be stored in custom formats and include additional information such as
formatting, formulas, etc. Microsoft Excel, which stores data in .XLS or .XLSX format is
probably the most common spreadsheet. Others include Google sheets, Apple Numbers,
and LibreOffice. Hierarchical databases are useful when you need to represent data in a tree like
hierarchy
3. XML files, contain data values that are identified or marked up using tags. While data in
flat files is “flat” or maps to a single table, XML files can support more complex data
structures, such as hierarchical. Some common uses of XML include data from online
surveys, bank statements, and other unstructured data sets. Many data providers and
websites provide APIs, or Application Program Interfaces, and Web Services, which
multiple users or applications can interact with and obtain data for processing or analysis.
APIs and Web Services typically listen for incoming requests, which can be in the form of
web requests from users or network requests from applications and return data in plain
text, XML, HTML, JSON, or media files.
Let’s look at some popular examples of APIs being used as a data source for data analytics:

• The use of Twitter and Facebook APIs to source data from tweets and posts for
performing tasks such as opinion mining or sentiment analysis, which is to summarize
the amount of appreciation and criticism on a given subject, such as policies of a
government, a product, a service, or customer satisfaction in general.
• Stock Market APIs used for pulling data such as share and commodity prices, earnings
per share, and historical prices, for trading and analysis.
• Data Lookup and Validation APIs, which can be very useful for Data Analysts for
cleaning and preparing data, as well as for co-relating data—for example, to check
which city or state a postal or zip code belongs to.
 • APIs are also used for pulling data from database sources, within and external to the
organization.
• Web scraping is used to extract relevant data from unstructured sources. Also known
as screen scraping, web harvesting, and web data extraction, web scraping makes it
possible to download specific data from web pages based on defined parameters. Web
scrapers can, among other things, extract text, contact information, images, videos,
product items, and much more from a website. Some popular uses of web scraping
include: collecting product details from retailers, manufacturers, and eCommerce
websites to provide price comparisons, generating sales leads through public data
sources, extracting data from posts and authors on various forums and communities,
and collecting training and testing datasets for machine learning models.
1 2 3 4
Some of the popular web scraping tools include Beautiful Soup, Scrapy, Pandas, and Selenium.
Data streams are another widely used source for aggregating constant streams of data
flowing from sources such as:
1• Instruments,
2• IoT devices and applications,
3• GPS data from cars,
4• Computer programs,
5• Websites, &
6• Social media posts
a digital record of the time of occurrence of a particular event:
This data is generally timestamped and also geo-tagged for geographical identification.
Some of the data streams and ways in which they can be leveraged include:
1• Stock and market tickers for financial trading,
2• Retail transaction streams for predicting demand and supply chain management,
3• Surveillance and video feeds for threat detection,
4• Social media feeds for sentiment analysis,
5• Sensor data feeds for monitoring industrial or farming machinery,
6• Web clicks feed for monitoring web performance and improving design, &
7• Real-time flight events for rebooking and rescheduling.
Some popular applications used to process data streams include:
A-Pa-Sh
• Apache Kafka, Apache Kafka is an open-source distributed event streaming platform that provides a high-
performance data pipeline and stream analytics
• Apache Spark Streaming, Apache Spark is a lightning-fast unified analytics engine for big data and machine
learning
• and Apache Storm Apache Storm is a free and open source software that processes unbounded streams
of data in realtime.
• RSS (or Really Simple Syndication) feeds, are another popular data source.
RSS (RDF Site Summary or Really Simple Syndication) is a web feed that allows users and
applications to access updates to websites in a standardized, computer-readable format
 These are typically used for capturing updated data from online forums and news sites where
data is refreshed on an ongoing basis. Using a feed reader, which is an interface that converts RSS

4 text files into a stream of updated data, updates are streamed to user devices.

Languages for Data Professionals

In this, we will learn about some of the languages relevant to the work of data professionals.
These can be categorized as – query languages, programming languages, and shell scripting.
Having proficiency in at least one language in each category is essential for any data professional.
Simply stated:

• Query languages are designed for accessing and manipulating data in a database; for
example, SQL.
• Programming languages are designed for developing applications and controlling
application behavior; for example, Python, R, and Java; and
• Shell and Scripting languages, such as Unix/Linux Shell, and PowerShell, are ideal for
repetitive and time-consuming operational tasks.
We will examine these languages in greater depth.
1. SQL, or Structured Query Language, is a querying language designed for accessing and
manipulating information from, mostly, though not exclusively, relational databases. Using
SQL,
1 o We can write a set of instructions to perform operations such as Insert, update,
and delete records in a database;
2 o Create new databases, tables, and views; &
3 o Write stored procedures-which means you can write a set of instructions and call
them for later use.
Here are some advantages of using SQL:
movable
1• SQL is portable and can be used independent of the platform,
inspecting
2• It can be used for querying data in a wide variety of databases and data repositories,
although each vendor may have some variations and special extensions,
3• It has a simple syntax that is similar to the English language,the arrangement of words and phrases to create
well-formed sentences in a language
4• Its syntax allows developers to write programs with fewer lines than some of the other
programming languages using basic keywords such as select, insert, into, and update,
5• It can retrieve large amounts of data quickly and efficiently,
6• It runs on an interpreter system, which means code can be executed as soon as it is
basic model
written, making prototyping quick and easy.
SQL is one of the most popular querying languages. Due to its large user community and the sheer
volume of documentation accumulated over the years, it continues to provide a uniform platform,
worldwide, to all its users.
 2. Python is a widely-used open-source, generalpurpose, high-level programming language.
1 o Its syntax allows programmers to express their concepts in fewer lines of code, as
compared to some of the older languages. Python is perceived as one of the
easiest languages to learn and has a large developer community.
2 o Because of its focus on simplicity and readability, and a low learning curve, it’s an
ideal tool for beginning programmers.
3 o It is great for performing high-computational tasks in vast amounts of data, which
can otherwise be extremely time-consuming and cumbersome. Python provides
libraries like NumPy and Pandas, which eases this task by the use of parallel
processing.
4 o It has inbuilt functions for almost all of the frequently used concepts.
5 o Python supports multiple programming paradigms, such as object-oriented,
imperative, functional, and procedural, making it suitable for a wide variety of use
cases.
Now let’s look at some of the reasons that make Python one of the fastest-growing
programming languages in the world today.

1• It is easy to learn - With Python, you have the advantage of using fewer lines of code to
accomplish tasks compared to other languages.
2• It is open-source — Python is free and uses a community-based model for development.
3• It runs on Windows and Linux environments and can be ported to multiple platforms.
4• It has widespread community support with plenty of useful analytics libraries available.
5• It has several open-source libraries for data manipulation, data visualization, statistics, and
mathematics, to name just a few.
Its vast array of libraries and functionalities also include:
i. Pandas for data cleaning and analysis,
ii. NumPy and Scipy, for statistical analysis,
iii. Beautiful soup and Scrapy for web scraping,
iv. Matplotlib and Seaborn visually represent data in the form of bar graphs, histogram,
and pie-charts,
v. Opencv for image processing.

3. R is an open-source programming language and environment for data analysis, data

visualization, machine learning, and statistics. Widely used for:
1 o Developing statistical software and,
2 o Performing data analytics,
3 o It is especially known for its ability to create compelling visualizations, giving it an
edge over some of the other languages in this space.
Some of the key benefits of R include the following:
1• It is an open-source - platform-independent programming language,
2• It can be paired with many programming languages, including Python,
3• It is highly extensible, which means developers can continue to add functionalities by
defining new functions,
4• It facilitates the handling of structured as well as unstructured data which means it has a
more comprehensive data capability,
5• It has libraries such as Ggplot2 and Plotly that offer aesthetic graphical plots to its users,
6• You can make reports with the data and scripts embedded in them; also,
7• Interactive web apps that allow users to play with the results and the data,
8• It is dominant among other programming languages for developing statistical tools.

4. Java is an object-oriented, class-based, and platform-independent programming language

originally developed by Sun Microsystems.
1 o It is among the top-ranked programming languages used today.
2 o Java is used in a number of processes all through data analytics, including cleaning
data, importing and exporting data, statistical analysis, and data visualization.
3 o In fact, most of the popular frameworks and tools used for big data are typically
written in Java, such as Hadoop, Hive, and Spark.
4 o It is perfectly suited for speed-critical projects.
5. Unix/Linux Shell is a computer program written for the UNIX shell. It is a series of UNIX
commands written in a plain text file to accomplish a specific task. Writing a shell script is
fast and easy. It is most useful for repetitive tasks that may be time-consuming to execute
by typing one line at a time.
Typical operations performed by shell scripts include:

1 • File manipulation,
2 • Program execution,
3 • System administration tasks such as disk backups and evaluating system logs,
4 • Installation scripts for complex programs,
5 • Executing routine backups,
Configuration frameworks are used to automate IT processes and give developers
6 • Running batches. and administrators control over the way applications run
6. PowerShell is a cross-platform automation tool and configuration framework by Microsoft
that is optimized for working with structured data formats, such as JSON, CSV, XML, and
REST APIs, websites, and office applications.
1 o It consists of a command-line shell and scripting language.
2 o PowerShell is object-based, which makes it possible to filter, sort, measure, group,
compare, and many more actions on objects as they pass through a data pipeline.
3 o It is also a good tool for data mining, building GUIs, and creating charts,
dashboards, and interactive reports.
Summary and Highlights
A data analyst ecosystem includes the infrastructure, software, tools, frameworks, and processes
used to gather, clean, analyze, mine, and visualize data.
Based on how well-defined the structure of the data is, data can be categorized as:
o Structured Data, that is data which is well organized in formats that can be stored in
databases.
o Semi-Structured Data, that is data which is partially organized and partially free form.
o Unstructured Data, that is data which cannot be organized conventionally into rows and
columns.
Data comes in a wide-ranging variety of file formats, such as delimited text files, spreadsheets,
XML, PDF, and JSON, each with its own list of benefits and limitations of use.
Data is extracted from multiple data sources, ranging from relational and non-relational
databases to APIs, web services, data streams, social platforms, and sensor devices.
Once the data is identified and gathered from different sources, it needs to be staged in a data
repository so that it can be prepared for analysis. The type, format, and sources of data influence
the type of data repository that can be used.
Data professionals need a host of languages that can help them extract, prepare, and analyze
data. These can be classified as:
o Querying languages, such as SQL, used for accessing and manipulating data from
databases.
o Programming languages such as Python, R, and Java, for developing applications and
controlling application behavior.
o Shell and Scripting languages, such as Unix/Linux Shell, and PowerShell, for automating
repetitive operational tasks.