Topics Covered:
● Pycharm
● Anaconda
● MongoDB
● Fast API
● Machine Learning
● NumPy
● Pandas
PyCharm
Pycharm is an integrated development environment(IDE) for Python,
developed by JetBrains. Supports web development frameworks, scientific
computing and data science.
Key Features :
1. Code Assistance:
a. Smart Code Completion.
b. Real-time code Analysis and error highlighting.
c. Quick documentation access
2. Project Navigation:
a. Easy project Management with a structured view.
b. Search Functionalities for files and Symbols.
3. Integrated Development Tools:
a. Built-in terminal
b. Version control integration(Git,SVN etc)
c. Database tools for managing SQL databases.
4. Debugging and Testing:
a. Powerful debugger for step-by-step code execution.
b. Support for unit testing frameworks.
5. Support for Web Frameworks:
a. Django, Flask, FastAPI Support.
b. HTML,CSS, and JavaScript support for web development.
6. Jupyter Notebooks Integration:
a. Allows Running Jupyter notebooks directly within
Pycharm(Professional Edition)
7. Environment Management:
a. Easily create and manage Virtual environments.
b. Integration with Anaconda for Package Movement.
8. Customization:
a. Highly Customizable interface and Settings.
b. Extensive plugins available for additional functionality.
Versions:
● Community Edition:Free, open- source, suitable for basic
Python Development.
● Professional Edition:Paid version with advanced features,
including web development support and database tools.
Installation:
● Download from the “JetBrains Website”
● Follow installation instructions for your OS.
Creating a New Project:
● Open Pycharm and Select “New Project”
● Choose a project interpreter.
Running Code:
● Create a new python file, write your code, and run it using the
“Run” button.
Debugging:
● Set breakpoints by clicking in the gutter next to the line number.
● Start the debugger to step through your code.
Using Jupyter Notebooks:
● Open or create a .ipynb file to use Jupyter Functionality within
Pycharm.
Tips:
● Utilize keyboard shortcuts to improve productivity.
● Explore plugins to extend functionality based on your needs.
● Regularly update PyCharm to access the latest features and
improvements.
These notes provide a foundational understanding of PyCharm and its role in
Python development, particularly for data science.
Anaconda
A distribution of Python and R for scientific computing, datascience,
machine learning and Big [Link] package management and
deployment.
Key features:
1. Package Management:
a. Conda:A Package manager that installs, runs, and updates
packages and their dependencies.
b. Supports both Python and R Packages.
2. Environment Management:
a. Create isolated environments for different projects using
Conda, avoiding package Conflicts.
b. Easy Switching between environments.
3. Pre-installed Libraries:
a. Comes with a wide-range of data science libraries(e.g.
NumPy,Pandas).
4. Anaconda Navigator:
a. A Graphical user interface(GUI) for managing packages,
environments and launching applications like Jupyter
Notebook and Spyder.
5. Cross-Platform Capability:
a. Available for Windows, macOS, and Linux.
6. Integration with IDE:
a. Works seamlessly with IDE,including Jupyter, Sypder and
Pycharm.
Installation:
● Download from the “Anaconda Website”
● Follow the installation instructions for your OS.
Creating a New Environment:
● Open a terminal or Anaconda prompt.
● Use this Command:
conda create --name myenv python=3.8
● Activate the Environment:
conda activate myenv
Installing Packages:
● Install packages with conda:
conda install numpy pandas
● You can also use PIP if needed:
pip install package_name
Launching Jupyter Notebook:
● In the terminal, run:
jupyter notebook
Common Commands:
● List Environments:
conda env list
● Deactivate Environment:
conda deactivate
● Remove an Environment:
conda env remove --name myenv
Tips:
● Regularly update Anaconda to access the latest packages and
features.
● Use the Anaconda Navigator for a user-friendly interface to
manage environments and packages.
● Check the Anaconda documentation for troubleshooting and
advanced features.
MongoDB
A noSQL database that stores data in flexible, JSON- like documents.
Designed for scalability, high-performance, and ease of development.
Key features :
1. Document-oriented Storage:
a. Data is stored in BSON(Binary JSON)format, allowing for
flexible Schema design.
2. Scalability:
a. Supports horizontal scaling through sharding, distributing data
across multiple servers.
3. High Performance:
a. Efficient read and write operations, optimized for handling large
volumes of data.
4. Flexible Schema:
a. Allows for dynamic schema changes, accommodating varied data
structures without requiring a predefined schema.
5. Rich Query Language:
a. Supports powerful queries, including filtering, sorting, and
aggregation.
6. Indexing:
a. Multiple indexing options (single field, compound, geospatial) to
enhance query performance.
7. Replication:
a. Supports automatic data replication for high availability and
disaster recovery.
8. Aggregation Frameworks:
a. Provides a powerful way to process data and return computed
results.
Installation:
● Download from the “MongoDB website”.
● Follow installation instructions for your operating system.
Starting the MongoDB Server:
● Use the command:
mongod
● Default data directory is /data/db.
Connecting to MongoDB:
● Open a new terminal and run:
mongo
● This starts the MongoDB shell for interacting with the database.
Basic Commands:
● Create a Database:
use myDatabase
● Insert a Document:
[Link]({ name: "Alice",
age: 25 })
● Find a Document:
[Link]({ age: { $gt: 20 }
})
● Update a Document:
[Link]({ name:
"Alice" }, { $set: { age: 26 } })
● Delete a Document:
[Link]({ name:
"Alice" })
Common Use cases:
● Real-time analytics.
● Content management systems.
● Internet of Things (IoT) applications.
● Mobile applications.
Tips :
● Regularly back up your data to prevent loss.
● Use indexing wisely to optimize performance, but avoid
excessive indexing as it can slow down write operations.
● Monitor performance metrics to identify bottlenecks.
FastAPI
A modern, fast (high-performance) web framework for building APIs with
Python. Based on standard Python type hints and asynchronous
programming.
Key Features:
1. Fast performance:
a. One of the fastest Python web frameworks, thanks to
asynchronous support (built on Starlette).
b. Comparable in performance to [Link] and Go.
2. Easy to Use:
a. Simple to set up and start building APIs quickly.
b. Automatic interactive API documentation (Swagger UI and
ReDoc) generated from code.
3. Data Validation:
a. Uses Pydantic for data validation, ensuring that input data meets
specified types and constraints.
4. Asynchronous Support:
a. Fully supports asynchronous programming, making it suitable for
handling concurrent requests.
5. Dependency Injection:
a. Built-in support for dependency injection, allowing for better
organization and testing of code.
6. Type Hints:
a. Leverages Python type hints for parameter and return type
validation, improving code readability and error detection.
7. Cross-origin Resource Sharing(CORS):
a. Built-in support for handling CORS, making it easier to work
with frontend applications.
Installation:
Install FastAPI and an ASGI server (like Uvicorn) using pip:
pip install fastapi uvicorn
Creating a Basic API:
from fastapi import FastAPI
app = FastAPI()
@[Link]("/")
def read_root():
return {"Hello": "World"}
@[Link]("/items/{item_id}")
def read_item(item_id: int, q: str = None):
return {"item_id": item_id, "query": q}
Running the Application:
● Use Uvicorn to run the FastAPI application:
uvicorn main:app --reload
● Access the interactive API docs at
[Link]
Common Commands:
Run the Server:
uvicorn filename:app --reload
Access API Documentation:
Swagger UI: [Link]
ReDoc: [Link]
Use Cases:
● Building RESTful APIs.
● Microservices architecture.
● Prototyping APIs quickly.
● Handling high-load applications.
Tips:
● Use Pydantic models for complex request and response data
structures.
● Utilize middleware for cross-cutting concerns like logging,
authentication, and error handling.
● Organize your code using routers to keep the application
modular.
Machine Learning
A subset of artificial intelligence (AI) that enables systems to learn from data,
identify patterns, and make decisions without explicit programming.
Key Concepts:
Types of Machine Learning:
Supervised Learning:
● Models are trained on labeled data (input-output
pairs).
● Examples: Regression, classification.
Unsupervised Learning:
● Models are trained on unlabeled data, finding
patterns or groupings.
● Examples: Clustering, dimensionality reduction.
Reinforcement Learning:
● Models learn by interacting with an environment,
receiving rewards or penalties.
● Used in robotics, game playing, etc.
Common Algorithm:
● Linear Regression: Predicts continuous values.
● Logistic Regression: Used for binary classification.
● Decision Trees: Non-linear models that split data
based on feature values.
● Support Vector Machines (SVM): Finds
hyperplanes that separate classes.
● Neural Networks: Inspired by the human brain, used
for complex pattern recognition.
● K-Means Clustering: Groups data into k clusters
based on feature similarity.
Key Terms:
● Training Set: The subset of data used to train a
model.
● Test Set: The subset used to evaluate model
performance.
● Overfitting: When a model learns noise instead of
the underlying pattern.
● Underfitting: When a model is too simple to capture
the data structure.
● Feature Engineering: The process of selecting,
modifying, or creating features from raw data to
improve model performance.
Machine Learning Process:
Data Collection:
● Gather data from various sources (databases, APIs,
etc.).
Data Processing:
● Clean and prepare the data (handling missing values,
normalization, encoding categorical variables).
Exploratory Data Analysis(EDA):
● Analyze data distributions and relationships through
visualizations.
Model Selection:
● Choose appropriate algorithms based on the problem
type and data characteristics.
Training:
● it the model to the training data.
Evaluation:
● Use metrics like accuracy, precision, recall, F1-score,
and AUC-ROC to assess model performance on the
test set.
HyperParameter Tuning:
● Optimize model parameters for better performance
(e.g., using grid search or random search).
Deployment:
● Integrate the trained model into a production
environment for real-time predictions.
Tools and Libraries:
● Programming Languages: Python, R.
● Popular Libraries:
○ Scikit-learn: For traditional ML algorithms.
○ TensorFlow: For deep learning.
○ Keras: High-level neural networks API.
○ PyTorch: Deep learning framework with dynamic computation
graph.
Use Cases:
● Image and speech recognition.
● Natural language processing (NLP).
● Fraud detection.
● Recommendation systems.
● Predictive analytics.
Tips:
● Always validate models with separate test data to prevent
overfitting.
● Use cross-validation for more robust performance evaluation.
● Continuously monitor and update models to adapt to new data
patterns.
Numpy
A fundamental package for numerical computing in Python. Provides support
for arrays, matrices, and a variety of mathematical functions.
Key Features :
1. N-Dimensional Arrays:
a. Central data structure is the ndarray (N-dimensional array).
b. Supports array operations and broadcasting.
2. Mathematical Functions:
a. Provides a wide range of mathematical functions (e.g.,
trigonometric, statistical, algebraic).
3. Performances:
a. Optimized for performance with low-level optimizations and
compiled C code, making operations faster than standard Python
lists.
4. Linear Algebra:
a. Built-in functions for linear algebra operations (e.g., dot product,
matrix multiplication).
5. Random Number Generation:
a. Tools for generating random numbers and distributions.
6. Interoperability:
a. Works well with other scientific libraries (e.g., SciPy, Pandas,
Matplotlib).
Basic operations:
1. Importing Numpy:
import numpy as np
2. Creating Arrays:
a. From a List:
import numpy as np
b. Zero Array:
zeros = [Link]((2, 3)) # 2x3 array
of zeros
c. Ones Array:
ones = [Link]((2, 3)) # 2x3 array of
ones
d. Random Array:
random_arr = [Link](2, 3) #
2x3 array with random values
3. Array Reshaping:
reshaped = [Link]((3, 2)) # Change
shape to 3x2
4. Slicing and Indexing:
arr[0] # Access first element
arr[1:3] # Slice elements from index 1 to 2
Use Cases:
● Data manipulation and analysis.
● Scientific computing and simulations.
● Image processing.
● Machine learning data preprocessing.
Tips:
● Use NumPy arrays instead of Python lists for numerical data to
improve performance.
● Familiarize yourself with broadcasting rules for efficient array
operations.
● Leverage built-in functions for mathematical operations instead
of writing custom loops for efficiency.
Pandas
A powerful open-source data manipulation and analysis library for Python.
Provides data structures and functions for working with structured data.
Key Features
1. Data Structures:
○ Series: One-dimensional labeled array capable of holding any
data type (similar to a list).
○ DataFrame: Two-dimensional labeled data structure with
columns of potentially different types (similar to a table or
spreadsheet).
2. Data Manipulation:
○ Easy to filter, group, and aggregate data.
○ Supports merging and joining of datasets.
3. Data Cleaning:
○ Functions for handling missing data, duplicates, and data type
conversions.
4. Time Series Support:
○ Built-in functionality for working with time series data, including
date ranges and frequency conversions.
5. Input/Output:
○ Read and write data from various formats (CSV, Excel, SQL
databases, JSON, etc.).
Basic Operations
Importing Pandas:
import pandas as pd
1. Creating Data Structures:
From a List:
s = [Link]([1, 2, 3, 4])
From a Dictionary:
df = [Link]({'A': [1, 2], 'B': [3,
4]})
2. Reading Data:
From CSV:
df = pd.read_csv('[Link]')
From Excel:
df = pd.read_excel('[Link]')
3. Viewing Data:
Head and Tail:
[Link]() # First 5 rows
[Link]() # Last 5 rows
Info:
[Link]() # Summary of DataFrame
4. Data Selection:
Selecting Columns:
df['A'] # Select column A
df[['A', 'B']] # Select multiple columns
Selecting Rows:
[Link][0] # Select first row
[Link][0] # Select row by index label
Filtering Data:
df[df['A'] > 1] # Filter rows where column
A is greater than 1
5. Common Functions
Descriptive Statistics:
[Link]() # Summary statistics
Handling Missing Values:
[Link]() # Drop missing values
[Link](0) # Fill missing values with 0
Group By:
[Link]('A').mean() # Group by column A
and calculate mean
Use Cases
● Data analysis and exploration.
● Data cleaning and preprocessing.
● Time series analysis.
● Statistical modeling and machine learning.
Tips
● Familiarize yourself with the difference between loc (label-based) and
iloc (integer-location based) indexing.
● Use vectorized operations instead of loops for better performance.
● Leverage built-in plotting capabilities with Matplotlib for quick
visualizations.