Scribd
Upload
1 views3 pages

Scalable Data Processing API Assignment

The document outlines an engineering assignment focused on scalable data processing, requiring the joining of two large datasets under strict memory constraints. The first assignment involves performing an INNER JOIN on two 500 MB CSV files using a Python script that must operate within a 256 MB RAM limit. The second assignment extends this solution into a non-blocking API using FastAPI, allowing users to trigger the join operation without blocking the server, while implementing logging and concurrency management.

Uploaded by

eternityparadox001
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
1 views3 pages

Scalable Data Processing API Assignment

The document outlines an engineering assignment focused on scalable data processing, requiring the joining of two large datasets under strict memory constraints. The first assignment involves performing an INNER JOIN on two 500 MB CSV files using a Python script that must operate within a 256 MB RAM limit. The second assignment extends this solution into a non-blocking API using FastAPI, allowing users to trigger the join operation without blocking the server, while implementing logging and concurrency management.

Uploaded by

eternityparadox001
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Engineering Assignment: Scalable Data

Processing API

In real-world data engineering and backend systems, we often need to process data that is larger
than the available system memory, and we need to do so in a way that doesn't crash our web
servers when multiple users request processing at the same time.

🛠️ Assignment 1: Out-of-Core Data Join

You are tasked with joining two datasets based on a common key ( user_id ).

• Dataset A (Users): ~500 MB .csv file


• Dataset B (Transactions): ~500 MB .csv file

(Note: A simple Python script is provided below to generate these files locally).

⚠️ The Constraint (Crucial)

You must write a Python script to perform an INNER JOIN on these two files and output a final
[Link] file. However, you must assume your runtime environment is heavily restricted:

• Artificial RAM Limit:


Assume your script is running on a container with only
256 MB of RAM
.
• Restriction:
You cannot simply load both datasets entirely into memory using [Link]()
or .read_csv() . Your script must process the files efficiently (e.g., via chunking,
streaming, or external sorting).
• Tooling:
You can use frameworks for this as well.

🚀 Assignment 2: Non-Blocking API

Extend your solution from Assignment 1 to serve as a backend API.


📌 Scenario

Users will hit an API endpoint to trigger the join operation. Because the join takes time and CPU
resources, doing this synchronously will block the web server and cause timeouts.

📦 Requirements

• Build a simple API (using FastAPI) with an endpoint (e.g., POST /trigger-join ).
• Concurrency: When a user hits the endpoint, the system should immediately return a
success message or a job_id , while the large data join (from Assignment 1) processes in
the background.
• Implementation: Implement at least 2 approaches with a proper pros and cons for all the
approaches you use.
• Logging: The background task should log when it starts and when it successfully finishes
writing [Link] .
Data Generation Script

import pandas as pd
import numpy as np
import os

def generate_data():
print("Generating synthetic datasets (approx 500MB each)... This may take a
minute.")

# Generate 5 Million Users


num_users = 5_000_000
users = [Link]({
'user_id': [Link](1, num_users + 1),
'name': ['User_' + str(i) for i in range(num_users)],
'signup_date': pd.date_range(start='2020-01-01', periods=num_users,
freq='min')
})
users.to_csv('[Link]', index=False)
print("[Link] created.")

# Generate 10 Million Transactions


num_transactions = 10_000_000
transactions = [Link]({
'transaction_id': [Link](1, num_transactions + 1),
'user_id': [Link](1, num_users + 1, size=num_transactions),
'amount': [Link](5.0, 500.0, size=num_transactions).round(2)
})
transactions.to_csv('[Link]', index=False)
print("[Link] created.")

if __name__ == "__main__":
generate_data()

You might also like