Data processing and

analysis
PROF DR. OMNIA EL BARBARY
A PI s a nd D a t a Co ll ec t io n

 Sessi on
• Introduction to APIs and Data Collection
• APIS(application programming interface)
• Types of API

Introduction to APIs and Data Collection
Data Collection
• Data collection is a fundamental step in the fields of Artificial Intelligence (AI) and Data
Science, forming the basis for model training, analysis, and decision-making.

[Link] of Data Collection

• Foundation for Models: High-quality data is essential for training accurate and reliable AI
models.

• Insights and Decision-Making: In Data Science, collected data helps in extracting

patterns, making predictions, and driving business decisions.

• Performance Improvement: The quantity and quality of data influence the performance
and generalization of AI algorithms.

1
Data Collection
2. Types of Data Collected
• Structured Data: Organized data in tabular forms like databases (e.g., sales data)

• Unstructured Data: Includes text, images, audio and video

• Semi-Structured Data: Data that doesn’t fit into strict rows and columns but has some
organizational properties (e.g., JSON, XML).

• Time-Series Data: Data collected over time, often used in forecasting.

1
Data Collection
3. Methods of Data Collection
• Manual Data Collection: Human-driven methods like surveys, interviews, and
observations

• Automated Data Collection:

• Web Scraping: Extracting data from websites.

•APIs (Application Programming Interfaces): Pulling data from platforms like social
media, financial services, or weather systems.

1
Data Collection
4. Data Collection Process
• Define Objectives: Clearly outline what data is needed and why.
• Identify Data Sources: Determine where and how the data can be collected
• Select Collection Methods: Choose between manual or automated techniques
based on requirements.
• Data Cleaning and Preprocessing: Remove inaccuracies, duplicates, and
irrelevant information to ensure data quality.
• Data Storage: Store data securely in databases, data lakes, or cloud storage.
• Compliance and Ethics: Ensure adherence to legal standards (like GDPR) and
ethical considerations for privacy

1
Data Collection
5. Challenges in Data Collection
• Data Quality: Dealing with missing, incorrect, or biased data.
• Scalability: Collecting and managing large volumes of data.
• Privacy and Security: Ensuring that data collection respects privacy regulations.
• Access to Data: Sometimes, data is difficult to access due to proprietary
restrictions or costs.

1
Importance of APIs for Data Gathering
• APIs (Application Programming Interfaces) play a crucial role in data gathering for
AI, Data Science, and other data-driven fields
• They provide a structured and efficient way to access, collect, and integrate data
from various platforms and services.

1. Efficient and Automated Data Collection

• Real-Time Data Access: APIs allow for the collection of real-time data, which is critical for
applications like stock market analysis, weather forecasting, or social media monitoring.

• Automation: APIs enable automated data collection, reducing manual effort and
minimizing errors. Scripts and programs can be written to regularly pull data without
human intervention.

1
Importance of APIs for Data Gathering
2. Access to Large and Diverse Data Sources
• Global Reach: APIs provide access to data from multiple platforms, such as social media
(Twitter, Facebook).

• Diverse Data Types: APIs can retrieve various types of data, including text, images,
video, and structured data like JSON or XML.

3. Data Consistency and Reliability

• Standardized Formats: APIs deliver data in consistent formats (like JSON or XML),
simplifying data parsing and processing

• Reliable Access: APIs from established platforms are stable and provide consistent data
access, reducing the risk of errors.

1
Importance of APIs for Data Gathering
4. Cost and Time Efficiency
• Reduces Manual Work: Automating data collection via APIs saves time and reduces
operational costs.
• Quick Integration: APIs enable faster integration with other systems or applications,
accelerating the data analysis process .

5. Examples of API Usage in Data Gathering

• Social Media Analysis: Using Twitter API to collect tweets for sentiment analysis.
• Weather Data: Collecting meteorological data using weather APIs for forecasting models.

1
Types of APIs
• APIs (Application Programming Interfaces) come in different types based on their
accessibility, usage, and intended audience.

• The three main types are Public APIs, Private APIs, and Partner APIs. Understanding
these types is crucial for selecting the right API for data gathering and integration
needs.

1
Types of APIs: Public APIs (Open APIs)
• Public APIs are open and accessible to anyone.
• They are designed for external developers and third parties, often made
available by organizations to encourage innovation, data sharing, and
integration.
Key Features:
• Open Access: Available to the public with minimal restrictions.

• Documentation Provided: Detailed documentation to guide developers on how to use

the API.

• Authentication: May require basic registration or API keys but are generally easy to
access.

• Promotes Innovation: Encourages third-party developers to build applications or services

using the API.

1
Types of APIs: Public APIs (Open APIs)
Examples of Public APIs:
•Twitter API: For accessing tweets and user data (within set limits).

•OpenWeather API: For retrieving weather data.

•Google Maps API: For embedding maps and location data in apps.

•Spotify API: For accessing music data and playlists.

1
Common Data Formats
• In data gathering and exchange—especially in AI, Data Science, and Web
Development—structured data formats are essential for organizing, storing, and
transferring information.
• Three of the most commonly used data formats are JSON, XML, and CSV.

1. JSON (JavaScript Object Notation)

• JSON is a lightweight, text-based, and human-readable data format used primarily for
data interchange between servers and web applications .

Advantages of JSON:
•Easy to Read and Write: Simple syntax and human-readable.
•Language Agnostic: Supported by most programming languages.
•Compact: Consumes less bandwidth.
•Ideal for APIs: Preferred for RESTful APIs.
Disadvantages of JSON:
•No Schema Definition: Less strict than XML, which can lead to data inconsistencies.
•Limited Metadata Support: Not as robust for complex structures as XML
1
Common Data Formats
2. XML (eXtensible Markup Language)
XML is a markup language used to store and transport data. It is designed to be self-
descriptive and follows a tree-structure format.

Advantages of XML:
•Highly Structured: Great for complex and hierarchical data.
•Extensible: Custom tags allow flexibility in data representation.
•Self-Descriptive: Clearly defines data with tags and attributes.
•Schema Support: Ensures data consistency and validation.
Disadvantages of XML:
•Verbose: Larger file size due to extensive tagging.
•Slower to Parse: Requires more processing power and bandwidth compared to JSON.
•More Complex: Harder to read and write manually.

1
Common Data Formats
3. CSV (Comma-Separated Values)
•CSV is a simple text format used to store tabular data, where each line corresponds
to a row and each value is separated by a comma.

Advantages of CSV:
•Simple and Easy: Easy to understand and process.
•Highly Compatible: Supported by most data analysis tools and spreadsheets (like Excel).
•Compact: Small file sizes, fast to read and write.
Disadvantages of CSV:
•No Support for Complex Data: Cannot handle hierarchical or nested structures.
•Lacks Metadata: No way to describe data types or schema.
•Data Ambiguity: Special characters (like commas) in data can cause issues if not
properly handled.

1
Common Data Formats

When to Use Each Format

•Use JSON when working with web APIs, modern web applications, or when needing lightweight,
structured data.
•Use XML when data validation, complex data structures, or extensive metadata is required.
•Use CSV for simple, tabular data exports or when working with spreadsheets and data analysis tools.
1
Real-world examples of API usage in data analysis
• APIs play a crucial role in Data analysis by enabling access to real-time data,
automating data collection, and integrating data from multiple sources.

1. Social Media Analysis

API Used: Twitter API, Facebook Graph API, Instagram API

Use Case:
•Sentiment Analysis: Collect tweets or posts to analyze public sentiment about brands,
events, or products using Natural Language Processing (NLP).
•Trend Detection: Track trending hashtags and topics over time.
•User Behavior Analysis: Understand user engagement and demographics.

Example:
A marketing agency uses the Twitter API to collect data about customer sentiments during
a product launch, identifying positive and negative feedback in real-time.

1
Real-world examples of API usage in data analysis
2. Financial Data Analysis
API Used: Alpha Vantage API, Yahoo Finance API, IEX Cloud API

Use Case:
•Stock Market Prediction: Gather historical and real-time stock price data for machine
learning models.
•Portfolio Management: Track financial indicators to optimize investment portfolios.
•Market Sentiment Analysis: Integrate with news APIs to correlate market trends with
financial news.

Example:
A financial analyst builds an AI model using Alpha Vantage API to predict stock price
movements based on historical trends and technical indicators .

1
Real-world examples of API usage in data analysis
3. Weather Forecasting and Analysis
API Used: OpenWeatherMap API, Weatherstack API

Use Case:
•Agriculture Forecasting: Provide farmers with weather predictions to optimize planting and
harvesting schedules.
•Supply Chain Optimization: Analyze weather patterns to plan logistics and avoid delays.
•Disaster Prediction: Use historical data for predicting natural disasters like floods or storms.

Example:
A logistics company uses the OpenWeatherMap API to optimize delivery routes by
avoiding areas with severe weather forecasts.

1
Real-world examples of API usage in data analysis
4. Sports Analytics
API Used: [Link] API, ESPN API

Use Case:
•Player Performance Analysis: Track performance statistics to develop predictive models.
•Fan Engagement: Create interactive dashboards showing live stats.
•Game Predictions: Use historical data to forecast game outcomes.

Example:
A sports analytics company uses the [Link] API to collect player statistics for building
machine learning models that predict match outcomes .

1
APIS(application programming interface)
What is APIS
• API stands for application programming interface.

• In essence, an API acts as a communication layer, or interface, that allows different systems
to talk to each other without having to understand exactly what the others do.

• Is a set of rules and protocols that allows different software components to

communicate with each other

1
Types of APIs in Python
• Web APIs: Used to interact with web services(e.g RESTful APIs)

• Library APIs: Provided by Python libraries (e.g TensorFlow, Numpy)

• Operating System APIs : For interacting with the system (e.g os module)

• Database APIs: Used to connect to databases(e.g MySQL, SQLite)

1
Status Codes
• Status codes are one of the most important pieces of information to look for in any API
response.
• They tell you if your request was successful, if it’s missing data, if it’s missing credentials, and
so on.
• here’s a list with some of the most common status codes that you’ll find:

1
What is an API Endpoint?
• An API endpoint is a specific URL (Uniform Resource Locator) where an API can be
accessed by a client to perform a specific action, such as retrieving or updating data.
• Structure of an API Endpoint:

Examples of API Endpoints:

• [Link] → Fetch all users.
• [Link] → Fetch the user with ID 1
• [Link] → Create or retrieve posts.

1
What are API Parameters?
API parameters are options you can pass to an endpoint to modify the request and get
specific data.
There are different types of parameters:
a. Path Parameters
• Part of the endpoint's URL and used to identify a specific resource .
b. Query Parameters
• Used to filter or sort data.
• Added at the end of the URL using a ? with multiple parameters joined by &
c. Body Parameters
• Used in POST, PUT requests to send data to the server in the body of the request.
• Typically, in JSON format.

1
Working with web APIs in Python
• Python provides several libraries to interact with APIs:
• requests (most commonly used for RESTful APIs)
• Flask (for creating APIs)

a) Making API Requests with requests

1
Working with web APIs in Python
• Example: Making a GET Request Using requests
• Purpose: Retrieve data from a server.
• Example: Fetching user information. (Path parameter)

1
Working with web APIs in Python
• Example: Making a GET Request Using requests
• Purpose: Retrieve data from a server.
• Example: Fetch posts from user ID 1 and limits result (query parameter)

1
Working with web APIs in Python
• Example: Sending Data Using POST Request (Body Parameters)
• Purpose: Send data to the server to create a new resource.
• Example: Creating a new user or post.

1
Working with web APIs in Python
• Example: Updating a user's profile information using PUT (Body Parameters)
• Purpose: Update an existing resource or create if it doesn’t exist.
• Typically replaces the entire resource.
• Use when sending complete data for an update.

1
Working with web APIs in Python
• Example: Deleting a specific post using DELETE
• Purpose: Remove a resource from the server.
• Usually returns a 204 No Content status code if successful.

1
Working with web APIs in Python
• Example: Creating API with Flask

1
API Authentication Methods
When interacting with APIs, authentication ensures that only authorized users can access specific
resources.
API Keys
• What is it?
An API key is a simple string (like a password) that identifies the client accessing the API.
• How it works:
• The API provider generates a unique API key.
• The client includes this key in the request (usually in the header or query parameters).
• The server validates the key and processes the request.

1
Working with Popular APIs
• Web Data APIs (e.g., OpenWeather)
• Step 1: Sign Up and Get an API Key
• Go to the [Link]
• Create a free account
• After logging in, navigate to the API Keys section
• Click on "Create Key" and name it.
• You'll receive an API key (keep this secure).
•

1
Working with Popular APIs
• Web Data APIs (e.g., OpenWeather)
• Step 2: Basic API Usage in Python
• Example: Get Current Weather by City Name