Data Management and Security

in System Design

1. Data Management

● Data management refers to collecting, storing, organizing, and maintaining data

efficiently
● It ensures data is accurate, available, and secure
● Important for performance, scalability, and decision-making

2. Types of Data in Systems

● Structured data → stored in tables (databases)

● Unstructured data → text, images, videos
● Semi-structured data → JSON, XML
● Real-time data → streaming data (IoT, sensors)

3. Data Storage Techniques

● Relational Databases (RDBMS)

○ Example: MySQL, PostgreSQL
○ Uses tables and schemas
● NoSQL Databases
○ Example: MongoDB, Cassandra
○ Flexible schema, scalable
● Distributed Storage
 ○ Data stored across multiple machines
○ Improves reliability and availability
● Cloud Storage
○ Example: AWS S3, Google Cloud Storage
○ Scalable and cost-effective

4. Data Modeling

● Process of designing data structure

● Entities → objects (User, Product)
● Attributes → properties (Name, ID)
● Relationships → connections between entities
● Helps in efficient database design

5. Data Consistency and Integrity

● Consistency ensures same data across system

● Integrity ensures correctness of data
● Types of consistency:
○ Strong consistency → immediate update everywhere
○ Eventual consistency → updates spread over time
● ACID properties:
○ Atomicity → all or nothing
○ Consistency → valid state
○ Isolation → no interference
○ Durability → permanent storage

6. Data Backup and Recovery

 ● Backup → copy of data for safety
● Types:
○ Full backup
○ Incremental backup
○ Differential backup
● Recovery strategies:
○ Disaster recovery plans
○ Replication (real-time copies)

7. Data Security Basics

● Protects data from unauthorized access

● Ensures confidentiality, integrity, availability (CIA Triad)
● Confidentiality → only authorized users access data
● Integrity → data is not altered
● Availability → data accessible when needed

8. Authentication and Authorization

● Authentication → verifying identity (login, password)

● Authorization → granting permissions
● Methods:
○ Password-based
○ Multi-factor authentication (MFA)
○ Biometric authentication

9. Encryption Techniques
 ● Encryption converts data into unreadable format
● Types:
○ Symmetric encryption → same key (AES)
○ Asymmetric encryption → public/private key (RSA)
● Used in:
○ Data at rest
○ Data in transit (HTTPS, SSL/TLS)

10. Access Control Mechanisms

● Role-Based Access Control (RBAC)

● Attribute-Based Access Control (ABAC)
● Least privilege principle → minimum required access

11. Network and System Security

● Firewalls → block unauthorized access

● Intrusion Detection Systems (IDS)
● Intrusion Prevention Systems (IPS)
● Secure APIs and endpoints

12. Data Privacy and Compliance

● Protect user personal data

● Follow regulations:
○ GDPR
 ○ HIPAA
○ Indian IT Act
● Techniques:
○ Data masking
○ Anonymization

13. Logging and Monitoring

● Tracks system activities

● Helps detect security breaches
● Tools:
○ Log management systems
○ SIEM (Security Information and Event Management)

14. Best Practices in Data Management and Security

● Regular backups
● Strong password policies
● Encrypt sensitive data
● Regular security audits
● Update and patch systems
● Use secure coding practices

15. Challenges

● Handling large-scale data

● Maintaining security in distributed systems
● Balancing performance and security
 ● Managing real-time data

Transaction Processing

1. Introduction to Transaction Processing

● A transaction is a sequence of operations performed as a

single logical unit
● Used in databases and distributed systems
● Ensures data consistency and reliability

Example:

● Bank transfer → debit from one account + credit

to another

2. Properties of Transactions (ACID)

● Atomicity
○ Transaction is all-or-nothing
○ If one step fails → entire transaction fails
 ● Consistency
○ Database remains in a valid state
● Isolation
○ Transactions do not interfere with each other
● Durability
○ Once committed → permanently stored

3. Transaction States

● Active → transaction is executing

● Partially Committed → last statement executed
● Committed → successfully completed
● Failed → error occurred
● Aborted → rolled back

4. Transaction Lifecycle

Start
|
v
Active ---> Failed ---> Aborted
|
 v
Partially Committed
|
v
Committed


5. Concurrency Control

● Ensures multiple transactions run safely

● Problems:
○ Dirty Read → reading uncommitted data
○ Lost Update → overwrite problem
○ Non-repeatable Read
● Techniques:
○ Lock-based protocols
○ Timestamp ordering
○ Optimistic concurrency control

6. Lock-Based Protocols

● Shared Lock (Read Lock)

 ● Exclusive Lock (Write Lock)
● Two-Phase Locking (2PL):
○ Growing Phase → acquire locks
○ Shrinking Phase → release locks

7. Serializability

● Ensures concurrent execution = serial execution

8. Deadlock in Transactions

● Occurs when transactions wait for each other

Example:

● T1 holds A, waits for B

● T2 holds B, waits for A

9. Deadlock Handling

● Prevention → avoid circular wait

● Detection → wait-for graph
● Recovery → abort one transaction
10. Recovery Mechanisms

● Log-based recovery
● Checkpoints
● Undo and Redo operations

11. Transaction Log

[T1 Start]
[T1 Write A]
[T1 Commit]

● 
Used for recovery after failure

12. Distributed Transactions

● Transactions across multiple systems

● Challenges:
○ Network failure
○ Partial commit
13. Two-Phase Commit (2PC)

● Phase 1 → Prepare
● Phase 2 → Commit / Abort

14. Three-Phase Commit (3PC)

● Extension of 2PC
● Adds extra phase to avoid blocking

15. Real-World Examples

● Banking systems
● E-commerce order processing
● Airline reservation systems

16. Key Challenges

● High concurrency
● Fault tolerance
● Scalability
● Network delays in distributed systems
Conflict Serializability and
Recoverability

1. Introduction

● In concurrent systems, multiple transactions execute

together
● Goal → maintain correctness and consistency
● Two key concepts:
○ Conflict Serializability
○ Recoverability

2. Conflict Serializability

● A schedule is conflict serializable if it can be

transformed into a serial schedule
● Transformation is done by swapping non-conflicting
operations
● Conflict serializability is a database concurrency
control property ensuring a non-serial schedule is
equivalent to a serial schedule by swapping non-
conflicting operations.

3. Conflicting Operations

Two operations conflict if:

● They belong to different transactions

● They operate on the same data item
● At least one is a write

Types of conflicts:

● Read–Write (R-W)
● Write–Read (W-R)
● Write–Write (W-W)

4. Non-Conflicting Operations

● Read–Read (R-R) → no conflict

● Operations on different data items → no
conflict
 5. Example Schedule

T1: R(A) ---- W(A)

T2: R(A) ---- W(A)

● 
This may lead to conflict → order matters

6. Precedence Graph (Serialization Graph)

● Used to check conflict serializability

● Nodes → transactions
● Edge T1 → T2 if:
○ T1 executes before T2 on conflicting operation

7. Graph Example
T1 --------> T2

● If graph has no cycle → Serializable

● If graph has cycle → Not Serializable

8. Recoverability
● Ensures system can recover from failures without
inconsistency
● Focuses on commit order of transactions

9. Types of Schedules Based on Recoverability

1. Recoverable Schedule

● T2 commits only after T1 commits

● If T2 reads data written by T1

T1: W(A) ------ Commit

T2: R(A) -------- Commit


2. Non-Recoverable Schedule

● T2 commits before T1

T1: W(A) ----------- (not committed)

T2: R(A) ---- Commit ❌

● 
Dangerous → may cause inconsistency
3. Cascading Rollback

● Failure of one transaction causes others to roll

back

T1: W(A) ---- Fail

T2: R(A) ---- Fail


4. Cascadeless Schedule

● Transactions read only committed data

T1: W(A) ---- Commit

T2: R(A) ---- Commit

● 
No cascading rollback

5. Strict Schedule

● Strongest form
● No transaction reads/writes until previous
transaction commits
 10. Comparison

● Conflict Serializability
○ Focus → correctness of execution order
○ Tool → precedence graph
● Recoverability
○ Focus → safe recovery from failure
○ Ensures proper commit order

Concurrency Control Techniques

1. Introduction

● Concurrency control manages simultaneous execution

of transactions
● Goal → maintain consistency, isolation, and
correctness
● Prevents anomalies when multiple users access data

2. Need for Concurrency Control

● Multiple users access same database

 ● Improves system performance
● Prevents data inconsistency

3. Common Problems (Anomalies)

● Dirty Read
○ Reading uncommitted data
● Lost Update
○ One transaction overwrites another
● Non-repeatable Read
○ Same query gives different results
● Phantom Read
○ New rows appear in repeated queries

4. Example:

Case: Lost Update

T1: Read(A=100)
T2: Read(A=100)

T1: A = A + 10 → 110

T2: A = A - 20 → 80

Final A = 80 (Incorrect)
 

5. Concurrency Control Techniques

1. Lock-Based Protocols

The most widely used technique.

● Types of Locks
● Shared Lock (S) → Read-only
access
● Exclusive Lock (X) → Read & write
access
● Two-Phase Locking (2PL)
● Growing Phase → Locks are
acquired
● Shrinking Phase → Locks are
released

★ Ensures conflict serializability

★ Can cause deadlocks

2. Timestamp-Based Protocols

Each transaction gets a unique timestamp.

Rules:

● Older transactions get priority

● Ensures operations follow timestamp order
 Types:

● Basic Timestamp Ordering

● Thomas Write Rule (ignores outdated writes)

Timestamp Example:

T1 (TS=1) → older

T2 (TS=2)

If conflict:
T1 proceeds, T2 waits or aborts

★ Deadlock-free
★ May cause transaction rollback (starvation
possible)

3. Optimistic Concurrency Control (OCC)

OCC is Also called Validation-Based Protocol

Phases:

1. Read Phase → Transaction executes without

restrictions
2. Validation Phase → Check for conflicts
3. Write Phase → Commit if valid

OCC Diagram:
Transaction
|
v

Read → Validate → Write



★ No locks → high concurrency

★ High rollback rate if conflicts are frequent

4. Multiversion Concurrency Control (MVCC)

Maintains multiple versions of data.

● Readers access old versions
● Writers create new versions

MVCC Diagram:

A1 (Old Version) ← Read by T1

A2 (New Version) ← Written by T2



★ No read-write conflicts
★ High performance in read-heavy systems
Example:
● Amazon DynamoDB
● PostgreSQL
 Comparison
Technique Locks Used Deadlock B
Lock-Based Yes Possible General
Timestamp No No Real-tim
OCC No No Low con
MVCC No (logical No Read-he
versions)

6. Deadlock in Concurrency Control

● Transactions wait for each other indefinitely

Deadlock Example

T1 holds A → waits for B

T2 holds B → waits for A



7. Deadlock Handling

● Prevention → avoid circular wait

 ● Detection → wait-for graph
● Recovery → abort one transaction

Data Recovery Mechanisms

1. Introduction

● Data recovery ensures system can restore data after

failure
● Important for reliability and fault tolerance
● Used in databases and distributed systems

2. Types of Failures

● Transaction Failure
○ Logical error (invalid input)
○ System error (deadlock, abort)
● System Crash
○ Power failure, OS crash
● Disk Failure
○ Hardware failure, data corruption
 ● Network Failure (in distributed systems)
○ Message loss, partition

3. Recovery Techniques Overview

● Log-based recovery
● Checkpointing
● Shadow paging
● Backup and restore

4. Log-Based Recovery

● Maintains a log file of all operations

● Stored on stable storage

5. Log Example

[T1 Start]
[T1 Write A: 100 → 200]
[T1 Commit]


6. Write-Ahead Logging (WAL)

● Log is written before actual data update

● Ensures recovery is possible

7. Recovery Operations

● Undo
○ Reverts uncommitted transactions
● Redo
○ Reapplies committed transactions

8. Undo/Redo Diagram
Crash occurs
|
v
Check Log
/ \
Undo Redo
(Uncommitted) (Committed)


9. Checkpointing

● Saves system state periodically

● Reduces recovery time

10. Checkpoint Diagram

---Log Records---[Checkpoint]---Log Records---
| |
Before Crash After Crash


11. Shadow Paging

● Keeps two versions of data:

○ Current page
○ Shadow (backup) page
● No logs required
12. Shadow Paging Diagram
Before Update:
Page Table → Old Pages

After Update:
Page Table → New Pages
Shadow → Old Pages


13. Backup and Restore

● Full backup → entire data

● Incremental backup → changes only
● Differential backup → changes since last full
backup

14. Backup Strategy Diagram

Full Backup → Incremental → Incremental
|
Restore Process

15. Recovery in Distributed Systems

● Uses protocols like:

○ Two-Phase Commit (2PC)
○ Three-Phase Commit (3PC)
● Handles partial failures

16. ARIES Recovery Algorithm

● Advanced recovery technique

Phases:

● Analysis → identify transactions

● Redo → repeat committed actions
● Undo → rollback incomplete transactions

17. ARIES Diagram

Crash
|
v
Analysis → Redo → Undo

 18. Comparison of Techniques

● Log-based → flexible, widely used

● Checkpointing → faster recovery
● Shadow paging → simple but costly
● Backup → long-term safety

Data Security

1. Introduction

● Data security protects data from unauthorized access,

misuse, or theft
● Critical in system design for trust, privacy, and
compliance
● Applies to data at rest, in transit, and in use

2. Goals of Data Security (CIA Triad)

● Confidentiality
○ Only authorized users can access data
● Integrity
○ Data remains accurate and unaltered
● Availability
 ○ Data is accessible when required

3. CIA Triad Diagram

 Confidentiality
/\
/ \
/ \
/ \
/ \
/ \
Integrity -------- Availability


4. Types of Data Security

● Data at Rest → stored data (databases, disks)

● Data in Transit → data being transmitted
(network)
● Data in Use → data being processed in memory

5. Authentication
 ● Verifies identity of users

Methods:

● Password-based
● Multi-Factor Authentication (MFA)
● Biometric authentication

6. Authentication Flow

User → Login → Verify Credentials → Access

Granted/Denied


7. Authorization

● Determines what actions a user can perform

Models:

● Role-Based Access Control (RBAC)

● Attribute-Based Access Control (ABAC)

8. Authorization Diagram (RBAC)

User → Role → Permissions

 |
(Read, Write, Execute)


9. Encryption

● Converts data into unreadable form

Types:

● Symmetric Encryption → same key (AES)

● Asymmetric Encryption → public/private key (RSA)

10. Encryption Process

Plain Text → Encryption → Cipher Text

|
Key

Cipher Text → Decryption → Plain Text

|
Key

11. Hashing

● Converts data into fixed-length value

● One-way function (cannot reverse)

Used for:

● Password storage
● Data integrity

12. Hashing Diagram

Input Data → Hash Function → Hash Value


13. Digital Signatures

● Ensures authenticity and integrity

● Uses private/public key

14. Digital Signature Flow

Sender → Hash → Encrypt (Private Key)
|
Signature
Receiver → Decrypt (Public Key) → Verify


15. Network Security Mechanisms

● Firewalls → block unauthorized access

● IDS (Intrusion Detection System)
● IPS (Intrusion Prevention System)
● VPN (Virtual Private Network)

16. Secure Communication

● HTTPS (HTTP Secure)

● SSL/TLS protocols

17. Data Backup & Recovery (Security Aspect)

● Prevents data loss due to attacks/failures

● Regular backups ensure availability

18. Data Privacy

 ● Protects personal and sensitive information

19. Common Security Threats

● Malware (virus, ransomware)

● Phishing attacks
● SQL injection
● Man-in-the-middle attack

20. Best Practices

● Use strong passwords

● Enable MFA
● Encrypt sensitive data
● Regular software updates
● Access control with least privilege
● Regular security audits