Data Engineering with

Microsoft Azure
Session-2
Ujwal Iyer
Skugan Venkatesan
July 31, 2020
INTERNAL
 Agenda

1. About this session

2. Analytical data stores

3. SaaS/PaaS offerings- MSFT vs SAP

4. Data platform reference architecture & alternatives

5. Demo

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 2

Session outline

What to expect from this session

1. Overview of data services in Azure:

– Analytical data stores

2. Compare and contrast few offerings of Azure vs SAP products

3. Analysis of an enterprise data architecture, and considerations behind the same

4. Demo- Synapse Analytics (formerly SQL Data Warehouse)

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 3

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 4
Analytical Data Stores
Analytical Data Stores- an introduction
There is NO single best data management
This is the “Serving” layer in the lambda architecture. This arch is choice for all data storage tasks.
divided into layers:
Desired capabilities at the 3 layers:
• Speed layer - processes real-time data streams,
• Speed layer: random writes
• Batch layer - precomputes results
• Batch layer: batch writes
• Serving layer – output from batch & speed stored here,
responds to ad-hoc queries • Serving layer: random reads

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 6

Analytical Data Stores- technologies

Approaches to serving layer in General: Options for data serving storage in Azure:
1. Kimball-style relational DW(dimensional star 1. Azure Synapse Analytics
schema)
2. Azure SQL Database
2. Low latency NoSQL – Hbase, Cassandra
3. SQL Server in Azure VM
3. Many more..
4. HBase on HDInsight
5. Azure Analysis Services
6. Azure Cosmos DB

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 7

Analytical Data Stores- how to select?

Development Considerations Operational Considerations Commercial Considerations

• Data structure, type, blob • Availability SLA, RPO, RTO • Licensing

• Consistency- eventual, strong. • Failure recovery • Support
• Query model- key-value, • Routine maintenance • SLA
analytics, geo-spatial, text
• Data replication support • Training of employees
• Data integration with- BI,
• AutnN- LDAP, DB user, IAM
analytics tools, Hadoop
• AuthZ- granular, doc attribute,
• Language used for row level
development- availability of
drivers in the DBs • Encryption- data in motion, rest
• Provisioning, upgrades
• Disk utilization mgmt.

Legacy integration, self-service, TCO, TTV, AI-ML, preferred ecosystem, scalability, flexibility

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 8

Analytical data stores- a comparison
Capability SQL DB Azure Synapse HBase on Hive on HDInsight Azure Analysis Cosmos DB
HDInsight Services

Primary database Relational Relational tables Wide column store In-Memory Tabular/MOLAP Document, graph, KV,
model with columnar semantic models wide column
storage
Optimized for speed Yes No Yes Yes No Yes
serving layer
Redundant regional Yes Yes Yes No No Yes
servers for high
availability
Supports query scale No Yes Yes Yes Yes Yes
out General

scale up Yes Yes No No Yes Yes

Scalability
Supports in-memory Yes Yes No Yes Yes No
caching of data
Security
Authentication SQL / Azure SQL / Azure AD local / Azure AD local / Azure AD Azure AD database users / Azure
Active AD via access control
Directory (IAM)
(Azure AD)
Data encryption at Yes Yes Yes Yes Yes Yes
rest
Row-level security Yes Yes Yes Yes Yes (through object- No
level security in
model)
Dynamic data Yes Yes Yes Yes No No
masking
© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 9
Offering Analysis: MSFT vs SAP
 Power BI & SAP Analytics Cloud

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 11

Power BI
Provides tools for: visualization, reporting, & exploration

1. Weak on-premises version: Compared with the

Power BI Pro cloud service, on-premises offering has
Power BI significant functional gaps. For e.g., none of these
• Connect to hundreds of data sources functions are supported in on-premise: dashboards,
• Use Power BI Embedded to integrate with apps streaming analytics, prebuilt content, natural

• Author on desktop for free with PBI Desktop, SaaS language Q&A, augmentation.
2. Connectivity: Power BI offers a very wide range of
with PBI Service, go mobile with PBI Mobile
data connectors, but query performance of on-
• Create offline, publish online
premises data gateways is variable and requires lots
• Plenty of 3rd party visualizations than SAC
of effort to optimize.
• Create own visuals using their open source ◼ Connectivity to SAP BW and HANA: direct
framework queries is problematic — a known issue that
MSFT is working on.

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 12

SAP Analytics Cloud

1. No offline desktop access, only cloud. Regular updates

2. ONE tool for visualizations, planning, and predictive

functions 1. Less connectors: when compared with PBI

3. Insight to Action: change data in S/4 from SAC

4. BOBJ investments safe- no rework, no rebuilding

5. Predictive capabilities: smart predict, smart grouping,

easy for citizen data scientists

6. Collaborative enterprise- planning, forecasting, budgeting

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 13

Benefits of SAP Analytics Cloud

1. Closed-loop capability: Integrated functionality for planning,

Azure Analysis Services
analytical and predictive capabilities in a unified, single platform.
• Models: Enterprise grade data models.
SAP Digital Boardroom unique “what if?” analyses and simulations.
• Structure: 2 modes: Tabular in-memory data
structure based on DAX, or multi-dimensional
2. Augmented and advanced analytics: Strong functionality for OLAP based on MDX.
NLG, NLP, automated insights, and automated time-series • Query: Data can either be in-memory(cached
mode) or Direct Query mode.
analysis.
• Compare: Akin to Universes in SAC & BOBJ
3. Breadth of capability: Pre-built content covers a range of
industries & LOB functions. This includes data models, data stories
and visualizations, templates for SAP Digital Boardroom agendas,
and guidance on using SAP data sources.

4. Excellent support for SAP solutions (including live query), and

non-SAP ingestion

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 14

 Synapse Analytics & SAP Data Warehouse Cloud

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 15

DWC Capabilities

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 16

Unified Analytics- Battle of the modern DW + enterprise analytics

Synapse Analytics Data Warehouse Cloud

Data Factory Data Intelligence Embedded *

Spark + SQL Data HANA + Data

engine Warehouse Sybase IQ Warehouse

Power BI SAP Analytics

Analytics Analytics
Cloud
(external)

Data Lake Data Lake

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 17

*GA in H2-2020
Comparing DWC and Synapse

• Paradigm shift in analytics, by unifying database, data mgmt., analytic cloud services

• MSFT embedding PowerBI and ML service in DDM stack. & SAP embedding Data Intelligence in DWC

Synapse Analytics Data Warehouse Cloud

• Lego based- integrate any Azure services • Broader & deeper data mgmt. capabilities:
data quality, metadata mgmt, & master
• Industry solutions: ADRM acquisition(data data mgmt.
models), SAS partnership(analytics, AI)
• Industry solutions: NA. Separate industry
cloud

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 18

 Azure Data Factory & Databricks vs SAP Data Intelligence

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 19

Comparing Data Factory + Databricks with SAP Data Intelligence

Azure Data Factory

SAP Data Intelligence
• SAP integration not robust enough
• Watermarking events not supported
• Good fit for Azure-centric environment • Cant handle recovery from node failures (replay relevant events,
handle windowing results)
• Works well with Databricks

• Node failures easily handled in Spark • Can integrate with Az Databricks via Spark submit job

• Less holistic compared to SAP DI • Pricing: single integrated solution, with a simpler pricing
• Deployment: same product available aaS or BYOL on any
hyperscaler, private cloud or on-prem
Azure Databricks
• Investment safeguarding: integration of SAP data and reuse of SAP
• Full blown ML platform on prem tools (Data Services, BW, SLT, HANA)
• E2E integration with transactional processes (with CPI and the Cloud
Platform Integration Suite)
SAP Data Intelligence =
• pre-built business content (not GA yet)
Az Data Factory + Az Databricks + • Data preparation + data catalog + pipelining + ML + data quality
Az Data Catalog

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 20

Actual Reference Architecture
Kennametal production use case scenario
Solution Stack
Business:
1. Azure Data Lake Storage
• Kennametal delivers industrial parts and services to global
2. Azure IoT Edge
customers in industries from aerospace to energy.
3. Azure IoT Hub
Problem:
4. Azure SQL Database
• Optimize factory outputs i.e. build a “Smart Factory” 5. Azure Stream Analytics
• Operators used paper forms to manage a fleet of machines 6. Power Apps
that produce metallic cutting, milling, and lathing parts for
7. Power BI
industrial applications

• Streamline order tracking & machine diagnostics Results:

1. Improved machine setup times by 50%
View Story
2. Increased press availability by 50 percent
3. Historical high of pieces produced per day

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 22

Kennametal- System Architecture

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 23

An advanced data platform architecture..

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 24

A few alternatives

Azure services been used in the architecture: A few alternatives to the previous architecture:

1. Azure Data Factory 1. Azure Data Factory is the service responsible for data pipeline

2. Azure Data Lake Gen2 orchestration. Azure Databricks can also be used through the

3. Azure Databricks execution of nested notebooks.

4. Azure Cosmos DB
2. Azure Stream Analytics is the service responsible for processing
5. Azure Cognitive Services
streaming data. Azure Databricks can also be used through the
6. Azure Event Hubs
execution of notebooks.
7. Azure Stream Analytics
8. Microsoft Power BI 3. Azure Databricks is used to invoke Cognitive Services. Azure
Functions can also be used here to invoke Azure Cognitive Services
from an Azure Data Factory Pipeline.

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 25

Demo
Synapse Analytics demo

1. Explore cold data scenario of ingesting raw data Demo Resources

files (CSV, parquet, JSON) Workspace resource group
Workspace name
2. Pipelines used for ingestion
Storage Account
3. Data transformation using tools such as data flows, Default file system container
Synapse Spark, and Synapse SQL (both SQL Pool
provisioned and serverless) SQL Serverless Endpoint
Azure Key Vault
4. Azure Synapse Studio(embedded Data Factory) to
author notebooks to further process data, create
datasets, train, and create machine learning
models

© 2020 SAP SE or an SAP affiliate company. All rights reserved. ǀ INTERNAL 27

Q&A