Data Visualisation using Data Intelligence and

PowerBI Apache Spark

Data Visualisation
using PowerBI
Let's introduce the concept of "Data
in Azure."
Think of Azure as a vast digital landscape, designed to manage data in all its
forms—from ingestion and storage to processing and analysis. Within this
landscape, Power BI becomes the lens that transforms raw data into
meaningful, interactive visualizations. Azure ensures that data is collected
securely, processed efficiently, and made accessible in real time, while Power
BI translates this into dashboards and reports that empower students, staff,
and decision-makers with actionable insights.

Azure Data Lake Storage (ADLS) is a highly scalable and cost-effective data
lake solution designed for big data analytics.
Imagine a traditional data warehouse – it's structured, organized, and great
for querying specific information. Azure Data Lake is similar and is designed
to store massive amounts of data in its native, raw format, whether it's
structured, semi-structured (like JSON or CSV), or unstructured (like text
documents, images, videos, and more).

There are two primary generations of

Azure Data Lake Storage:
1. Azure Data Lake Storage Gen1 (ADLS Gen1):
This was the first generation, built on a Hadoop Distributed File
System (HDFS).
It offered massive scalability and was optimized for big data
workloads.
It had a simpler pricing model focused on storage and throughput.
However, it had some limitations around security and integration
with other Azure services compared to Gen2.
Microsoft has now retired ADLS Gen1. New deployments should
focus on Gen2.

2. Azure Data Lake Storage Gen2 (ADLS Gen2): ADLS Gen2 acts as the
central, scalable, and flexible repository for all your learning data, ready
to be explored and analysed by various Azure services.
This is the recommended and future-forward data lake solution in
Azure.
It's built on Azure Blob Storage, inheriting its scalability, cost-
effectiveness, and tiering capabilities (hot, cool, archive).
Key Enhancement: Hierarchical Namespace (HNS): This is the
crucial difference from regular Blob Storage. HNS introduces a file
system-like structure with directories and files, which significantly
improves:
Data Organization: Makes it easier to organize and navigate vast
amounts of data.
Performance: Enables atomic file and directory operations,
which are essential for many big data analytics frameworks like
Apache Spark and Hadoop.
Security: Allows users to set access control at the directory and
file level using Azure Active Directory (Azure AD) and POSIX-like
permissions (Access Control Lists - ACLs).

Integration: ADLS Gen2 seamlessly integrates with a wide range of

Azure analytics services, including:
 Azure Databricks: For big data processing and machine learning.
Azure Synapse Analytics: For data warehousing and big data
analytics.
Azure Data Factory: For data ingestion and ETL/ELT.
Power BI: For data visualization and reporting.
Azure Machine Learning: For building and deploying machine
learning models.
Open Formats: It's designed to work well with open data formats like
Parquet, ORC, and Avro, which are commonly used in big data
ecosystems for efficient storage and processing.

Why is a Data Lake Important?

Store Diverse Data: Unlike a data warehouse with its rigid schema, a
data lake can store any type of data in its native format, allowing users to
capture everything without upfront transformation.
Scalability: Designed to handle petabytes and even exabytes of data.
Cost-Effective: Especially for large volumes of data, the storage costs are
optimized. Tiered storage options further help manage costs based on
data access frequency.
Flexibility for Analytics: Enables various types of analytics, from simple
reporting to advanced machine learning and real-time analysis. Users can
explore and transform the data as needed for different use cases.
Data Exploration: Allows data scientists and analysts to explore raw
data and discover new insights without predefined schemas.

Implement Azure Data Lake Architecture:

Ingest: This layer is responsible for ingesting data into the various data
storage targets which form the data lake, typically called the raw zone.
The storages used could be object store which stores raw files or
databases. Ingestion layer has the capability to connect to diverse data
sources and can ingest batch or streaming data into the lake. Ingestion
Pipelines can be triggered based on a pre-defined schedule, in response
to an event, or can be explicitly called via REST APIs.
Store: This layer is responsible for storing the raw data (raw zone) as well
as processed data (curated zone). It should be scalable as well as cost
effective to allow us to store vast quantities of data. It also has security,
high availability, data archival and backup capabilities.
Process and Enrich: This layer validates, transforms, and moves your
datasets into your Curated zone in your data lake. These layers have the
 data processing pipelines and the means to orchestrate the data flow. It
has mechanisms for auditing, reconciliation of data at various stages in
the pipeline. At this stage Machine Learning Models can also be invoked
to enrich your datasets and generate further business insights.
Serve: This is the layer which provides tools that support analysis
methods, including SQL, batch analytics, BI dashboards, reporting, and
ML.
Security, Monitoring: This layer ensures security to the various
components of all layers in the data analytics architecture. It has means
for authentication, authorisation, encryption, monitoring, logging, alerts.
Discover and Govern: This layer registers all user data sources,
organises them and allows auto-discovery of new datasets, cataloguing
and metadata updating. It stores data classification, data sensitivity, data
lineage information. It can help us maintain a business glossary with the
specific business terminology required for users to understand the
semantics of what datasets mean and how they are meant to be used
across the organisation.

Services used in the enclosed graphics in

its various layers are as follows:
Azure Event Hubs: Fully managed, real-time data ingestion service which
can stream millions of events per second from any source to build
dynamic data pipelines. It enables us to automatically capture the
streaming data in Event Hubs in an Azure Blob storage or Azure Data
Lake Storage Gen 1 or Gen 2.
Azure Data Lake Storage Gen2: Azure Data Lake Storage Gen2 builds
on Azure Blob storage and enhances on performance, management, and
security aspects. Especially the hierarchical namespace significantly
improves the overall performance of many analytics jobs. This storage
layer service is scalable and cost effective as well.

Other Key Azure Services for Data

Platforms:
Azure Synapse Analytics: End to End analytics service that brings
together data integration, enterprise data warehousing, and big data
analytics. Supports both data lake and data warehouse use cases. It is a
combination of multiple services some of which are Azure Synapse
Pipelines (can be used for data ingestion/processing), Data explorer pools
(for near real time analytics), Spark pools (for data processing workloads),
Serverless and Dedicated SQL pools (for creating enterprise data
warehouse, running T-SQL queries).
Azure Data Factory: Though not shown in the sample architecture, data
factory can also be used to build scalable data ingestion and data
processing pipelines.
Azure Stream Analytics: For real-time insights, use a Stream Analytics
job to implement the “Hot Path” of the Lambda architecture pattern and
derive insights from the stream data in transit. Easily enable scenarios
like low-latency dashboarding, streaming ETL, and real-time alerting.
Azure Cognitive Services: This offers a vast set of services which can be
used for data enrichment/data quality like entity recognition, sentiment
analysis, identify and analyse content in images or videos, content
moderator, anomaly detection. It can also be used in your serve layer to
improve user experience by using features like personaliser, text to
speech, speech to text, question-answering etc.
Azure Machine Learning: As part of your data enrichment process we
can create and use custom ML models from Azure ML, in addition to pre-
 defined models available in Azure cognitive services.
Azure Data Share: Data can also be securely shared to other business
units or external trusted partners using Azure Data Share. We can control
what is shared, who receives the data, and the terms of use.
Power BI Premium: We can load relevant data from the data lake into
Power BI datasets for data visualisation and exploration. Power BI
models implement a semantic model to simplify the analysis of business
data and relationships. Business analysts use Power BI reports and
dashboards to analyse data and derive business insights.
Azure Purview: Can be used for data governance – discovery and
insights on your data assets, data classification, and sensitivity, which
covers the entire organisational data landscape.
Azure Monitor: Can be used for monitoring of the entire data platform.
Monitor performance, usage as well as availability of various stages and
provides alerting and auto-scaling capabilities to respond to the various
problems.
Azure Active Directory: Is an enterprise identity service that provides
single sign-on, multi-factor authentication, and conditional access to data
in our data lake.

Learn more at the following link: [Link]

us/azure/architecture/data-

Business Intelligence using Power BI

Power BI is a powerful business intelligence (BI) and data visualization tool from
Microsoft. Think of it as the tool that helps users to take raw, often messy, data
and transform it into clear, understandable, and actionable insights through
interactive dashboards and reports.
Imagine users have spreadsheets filled with sales figures, website analytics, or
student performance data. These numbers can be overwhelming and may be
difficult to interpret. Power BI will interpret to make the data sensible.

Top 10 key benefits of using Power BI

 

PowerBI

What Power BI does?

1. Connects to Various Data Sources: Power BI can pull data from a vast arra
of sources, including:
Spreadsheets (like Excel)
Databases (like SQL Server, Azure SQL Database, Cosmos DB)
Cloud services (like Salesforce, Google Analytics, Adobe Analytics)
Files (like CSV, JSON)

2. Transforms and Models Data: Once connected, Power BI allows users to

clean, shape, and transform the data.
Filtering and sorting data
Combining data from multiple sources
Creating relationships between different datasets
Calculating new metrics and measures (like total sales, average score,
growth rate)
Defining data types and formats

This data modelling process ensures user data is structured correctly for
analysis and visualization.
 3. Visualizes Data: It offers a rich library of interactive charts, graphs, maps, 
and other visuals to represent your data in an engaging and easy-to-
understand way. Instead of looking at rows and columns of numbers, users
can see trends, patterns, and relationships at a glance.
4. Creates Interactive Dashboards and Reports: Users can combine these
visualizations onto interactive dashboards and multi-page reports. These
dashboards provide a high-level overview of key metrics, while reports allow
for more detailed exploration of the data. Users can often filter, drill down,
and interact with these visuals to uncover deeper insights.
5. Shares and Collaborates: Power BI enables users to easily share the
dashboards and reports with colleagues and stakeholders. This fosters data
driven decision-making across teams and the entire organization. Users can
publish reports to the Power BI service (the cloud-based platform) and
control who has access.

Power BI as data storytelling tool: It helps users take the raw facts and figures
and weave them into a compelling narrative using visuals. This makes it easier fo
everyone, regardless of their technical background, to understand the data and
make informed decisions.

Key Components of Power BI

1. Power BI Desktop: A free Windows application where you connect to data
sources, transform data, build data models, and design your visualizations
and reports.
2. Power BI Service: A cloud-based platform where you publish your reports
and dashboards, share them with others, collaborate, and set up automated
data refreshes.
3. Power BI Mobile Apps: Apps for iOS and Android devices that allow users to
view and interact with the Power BI dashboards and reports on the go.
4. Power BI Report Server: An on-premises solution for organizations that
need to keep their Power BI reports within their own infrastructure.

Different User Interface of PowerBI

 

Different User Interface of PowerBI

Progress: 0%

Copyright and
References Assessments

About UP Education
© 2025 Yoobee College of Creative Innovation is one of the trading names for the registered