Scribd
Upload
31 views7 pages

Advanced Python Cheat Sheet PDF

Uploaded by

aman
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
31 views7 pages

Advanced Python Cheat Sheet PDF

Uploaded by

aman
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

# [ Advanced Python for Data Science ] ( CheatSheet )

1. Advanced Data Structures

● List Comprehensions with Conditional Logic: squared_even = [x**2 for x


in range(10) if x % 2 == 0]
● Dictionary Comprehensions: squared_dict = {x: x**2 for x in range(10)}
● Set Comprehensions: unique_squared = {x**2 for x in range(-5, 5)}
● Nested Dictionary Comprehensions: matrix = {x: {y: x*y for y in range(3)}
for x in range(3)}
● Defaultdict for Default Values: from collections import defaultdict; dd =
defaultdict(int)
● Counter to Count Hashable Objects: from collections import Counter;
counts = Counter(my_list)
● OrderedDict to Maintain Insertion Order: from collections import
OrderedDict; od = [Link]('abcde')
● Deque for Efficient Stack and Queue: from collections import deque; dq =
deque([1, 2, 3])
● ChainMap to Combine Dictionaries: from collections import ChainMap;
combined = ChainMap(dict1, dict2)
● Namedtuple for Readable Tuples: from collections import namedtuple; Point
= namedtuple('Point', ['x', 'y'])
● Heapq for Priority Queues: import heapq; [Link](heap, item)
● Itertools for Complex Iterations: import itertools;
[Link]('ABCD')
● Bisect for Array Bisection Algorithms: import bisect;
bisect.bisect_left(a, x)
● Functools for Higher-order Functions: from functools import reduce;
reduce(lambda x, y: x+y, [1, 2, 3])
● Zip for Parallel Iteration: for x, y in zip(list1, list2):

2. Functional Programming

● Lambda Functions: multiply = lambda x, y: x * y


● Map for Function Application: squared = list(map(lambda x: x**2,
numbers))
● Filter to Extract Elements: evens = list(filter(lambda x: x % 2 == 0,
numbers))
● Reduce for Cumulative Operation: from functools import reduce; product =
reduce(lambda x, y: x * y, numbers)

By: Waleed Mousa


● Partial Functions for Arguments: from functools import partial; add_five
= partial(add, 5)
● Itertools for Advanced Iteration: cyclic = [Link]('ABCD')
● Generators for Lazy Evaluation: (x**2 for x in range(10))
● Decorator Functions for Meta-programming: @cache def fibonacci(n):
● Use of Closure to Enclose State: def outer(x): return lambda y: x + y
● Any and All for Condition Checking: any([True, False]), all([True, True])

3. Concurrency and Parallelism

● Threading for I/O-bound Tasks: from threading import Thread; thread =


Thread(target=function, args=(arg,))
● Multiprocessing for CPU-bound Tasks: from multiprocessing import Process;
process = Process(target=function, args=(arg,))
● Concurrent Futures for Async Execution: from [Link] import
ThreadPoolExecutor; [Link](function, arg)
● Asyncio for Asynchronous Programming: import asyncio; [Link](main())
● Use of Locks in Threading: from threading import Lock; lock = Lock()
● Semaphore for Controlling Access: from threading import Semaphore;
semaphore = Semaphore(2)
● Condition Variables for Synchronization: from threading import Condition;
condition = Condition()
● Event for Signaling Between Threads: from threading import Event; event =
Event()
● Queue for Thread-safe Data Exchange: from queue import Queue; queue =
Queue()
● Using map with ProcessPoolExecutor: with ProcessPoolExecutor() as
executor: results = [Link](func, args)

4. Debugging and Testing

● Use Assert Statements for Quick Checks: assert x > 0, 'x must be
positive'
● Logging for Debugging and Monitoring: import logging;
[Link]('Debugging information')
● Pdb for Interactive Debugging: import pdb; pdb.set_trace()
● Timeit for Timing Code Execution: import timeit; [Link]('func()',
setup='from __main__ import func')
● CProfile for Performance Profiling: import cProfile; [Link]('func()')
● Memory Profiler for Memory Usage: from memory_profiler import profile;
@profile def my_func():
By: Waleed Mousa
● Using PyTest for Unit Testing: def test_function(): assert func(x) ==
expected
● Mock for Testing in Isolation: from [Link] import Mock; mock =
Mock()
● Coverage for Test Coverage Measurement: coverage run -m pytest; coverage
report
● Use Type Hints for Static Type Checking: def greet(name: str) -> str:

5. Performance Optimization

● Using NumPy for Efficient Numeric Computation: import numpy as np;


np_array = [Link]([1, 2, 3])
● Pandas for Efficient Data Manipulation: import pandas as pd; df =
[Link]({'col1': [1, 2], 'col2': [3, 4]})
● Cython for Compiling Python: import cython; @[Link]
● JIT Compilation with Numba: from numba import jit; @jit def
sum_array(arr):
● Use of Cache to Avoid Recomputation: from functools import lru_cache;
@lru_cache(maxsize=None) def fib(n):
● Vectorization to Replace Loops (NumPy, Pandas): df['col3'] = df['col1']
+ df['col2']
● Use of Pandas Categoricals for Memory Efficiency: df['col'] =
df['col'].astype('category')
● Memory Views for Large Data Manipulation: memoryview([Link]([1, 2, 3]))
● Batch Processing for Large Datasets: for batch in pd.read_csv('[Link]',
chunksize=1000):
● Using HDF5 or Feather Format for Large Data Storage:
df.to_hdf('data.h5', 'table')

6. Advanced File Handling

● Read/Write JSON Files: import json; with open('[Link]', 'r') as f:


data = [Link](f)
● Working with CSV Files: import csv; with open('[Link]', newline='') as
f: reader = [Link](f)
● Manipulating ZIP Files: from zipfile import ZipFile; with
ZipFile('[Link]', 'r') as zip_ref:
zip_ref.extractall('path_to_extract')
● Handling Large Files with Generators: def read_large_file(file_object):
yield from file_object

By: Waleed Mousa


● Use Pickle for Object Serialization: import pickle; [Link](obj,
file)
● Working with Binary Data: with open('[Link]', 'wb') as f:
[Link](b'Hello World')
● Use Glob for File Path Pattern Matching: from glob import glob;
file_paths = glob('*.txt')
● Handling XML Data with ElementTree: import [Link] as ET;
tree = [Link]('[Link]')
● Working with HDF5 Files for Large Datasets: import h5py; f =
[Link]('data.h5', 'r')
● Using Pandas to Read/Write Excel Files: df.to_excel('[Link]',
index=False); df_read = pd.read_excel('[Link]')

7. Advanced Pandas Techniques

● MultiIndex DataFrame Operations: df.set_index(['level_1', 'level_2'])


● Conditional Operations Using [Link]: df['new_col'] = [Link](df['col']
> 0, 'positive', 'negative')
● Vectorized String Operations: df['col'].[Link]()
● Pandas SQL-like Queries: [Link]('col > 0')
● Pivot Tables for Data Summarization: df.pivot_table(values='D',
index=['A', 'B'], columns=['C'])
● Window Functions for Rolling and Expanding Calculations:
df['col'].rolling(window=5).mean()
● Merging, Joining, and Concatenating DataFrames: [Link]([df1, df2]);
[Link](df1, df2, on='key')
● Apply Functions for Custom Operations: [Link](lambda row: row['A'] +
row['B'], axis=1)
● Time Series Specific Operations: [Link]('M').mean()
● Categorical Data Handling for Memory Optimization: df['col'] =
df['col'].astype('category')

8. Advanced Visualization Techniques

● Interactive Plots with Plotly: import [Link] as px; [Link](df,


x='x', y='y')
● Advanced Matplotlib Customizations: fig, ax = [Link](); [Link](x,
y)
● Creating Dashboards with Dash or Streamlit: import streamlit as st;
st.line_chart(df)

By: Waleed Mousa


● Seaborn for Statistical Data Visualization: import seaborn as sns;
[Link](x='x', y='y', data=df)
● 3D Plotting with Matplotlib: ax = fig.add_subplot(111, projection='3d')
● Heatmaps for Correlation Visualization: [Link]([Link]())
● Pairplot for Multi-variable Analysis: [Link](df, hue='class')
● Facet Grids for Conditional Plots: g = [Link](df, col='col',
row='row'); g = [Link]([Link], 'val')
● Network Graphs with NetworkX: import networkx as nx; G = [Link]();
G.add_edge('A', 'B')
● Geospatial Data Visualization: import geopandas as gpd; world =
gpd.read_file([Link].get_path('naturalearth_lowres'))

9. Machine Learning Pipeline Optimization

● Automating Pipeline with Pipeline: from [Link] import Pipeline;


pipeline = Pipeline(steps=[('scaler', StandardScaler()), ('clf',
LogisticRegression())])
● Grid Search for Hyperparameter Tuning: from sklearn.model_selection
import GridSearchCV; GridSearchCV(pipeline, param_grid=param_grid)
● Feature Selection Techniques: from sklearn.feature_selection import
SelectFromModel; SelectFromModel(estimator)
● Model Serialization with Joblib for Deployment: from joblib import dump,
load; dump(model, '[Link]')
● Cross-Validation Strategies for Robust Model Evaluation: from
sklearn.model_selection import cross_val_score; cross_val_score(model,
X, y, cv=5)

10. Advanced Statistical Techniques

● ANOVA for Feature Selection: from scipy import stats;


stats.f_oneway(df['group1'], df['group2'])
● Linear Regression Diagnostics: import [Link] as sm; [Link](y,
sm.add_constant(X)).fit().summary()
● Kernel Density Estimation for Data Distribution: [Link](data)
● Principal Component Analysis for Dimensionality Reduction: from
[Link] import PCA; PCA(n_components=2).fit_transform(X)
● Time Series Decomposition: from [Link] import
seasonal_decompose; seasonal_decompose(series, model='additive')
● Bayesian Inference with PyMC3: import pymc3 as pm; with [Link]() as
model: # Define priors and likelihood

By: Waleed Mousa


● Survival Analysis for Time-to-Event Data: from lifelines import
KaplanMeierFitter; kmf = KaplanMeierFitter(); [Link](durations,
event_observed)
● Non-Parametric Tests for Independent Samples: [Link](x, y)
● Multivariate Regression Analysis: [Link](y,
sm.add_constant(X)).fit().summary()
● Hierarchical Clustering for Unsupervised Learning: from
[Link] import dendrogram, linkage; Z = linkage(X,
'ward')

11. Advanced Neural Network Techniques with TensorFlow/Keras

● Custom Layers for Specific Operations: class


MyCustomLayer([Link]): # Define computations
● Callbacks for Monitoring Training Process: [Link](X, y,
callbacks=[[Link]()])
● TensorBoard for Training Visualization: tensorboard_callback =
[Link](log_dir='./logs')
● Custom Training Loops for Granular Control: for epoch in range(epochs):
# Manually iterate over batches
● Implementing Attention Mechanisms for NLP: class
AttentionLayer([Link]): # Define attention computations
● Using Transfer Learning and Fine-Tuning Pre-trained Models: model =
[Link].VGG16(include_top=False); [Link] = False
● Generative Adversarial Networks for Data Generation: class
GAN([Link]): # Define generator and discriminator
● Recurrent Neural Networks for Sequence Data: model =
[Link]([[Link](128),
[Link](1)])
● Normalization Techniques for Faster Convergence:
[Link]()
● Custom Loss Functions and Metrics: def custom_loss(y_true, y_pred): #
Define custom logic

12. Advanced Python Tips and Tricks

● Using Walrus Operator for Assignment Expressions: if (n := len(a)) > 10:


print(f"List is too long ({n} elements)")
● Unpacking for Efficient Variable Assignment: a, *rest, b = range(10)
● Using pathlib for Filesystem Path Manipulation: from pathlib import Path;
p = Path('/usr/bin'); p.is_dir()

By: Waleed Mousa


● Dictionary Merging with ** Operator: merged_dict = {**dict1, **dict2}
● Using dataclasses for Boilerplate-free Data Structures: from dataclasses
import dataclass; @dataclass class Point: x: int; y: int
● Using Generators for Memory-efficient Loops: (x**2 for x in range(10))
● Context Managers for Resource Management: with open('[Link]') as f:
contents = [Link]()
● Using functools.lru_cache for Memoization:
@functools.lru_cache(maxsize=None) def fib(n):
● Async/Await for Asynchronous Programming: async def fetch_data(): data =
await get_data()
● Type Hints for Improved Code Clarity: def greet(name: str) -> str:

By: Waleed Mousa

Common questions

Powered by AI

ProcessPoolExecutor is designed for CPU-bound tasks and takes advantage of multiple CPU cores by using separate processes, while ThreadPoolExecutor is used for I/O-bound tasks using threads within the same process. This distinction is crucial as CPU-bound tasks benefit from multiple processes to bypass Python's Global Interpreter Lock. An example usage is 'with ProcessPoolExecutor() as executor: results = executor.map(func, args)', where 'func' is executed concurrently across multiple processes, thus efficiently handling tasks like image processing or mathematical computations .

ChainMap is beneficial when you want to search through multiple dictionaries as one combined mapping without combining them physically. This is useful for runtime scopes or configuration where multiple settings dictionaries might overlay each other. An example is 'from collections import ChainMap; combined = ChainMap(dict1, dict2)', which allows accessing keys across 'dict1' and 'dict2' seamlessly. If a key exists in the first dictionary, its value is used; otherwise, it checks the next dictionary .

Decorators in Python are a powerful meta-programming feature that allows the modification of functions or methods using wrappers, enhancing their behavior without changing their actual code. They can aid in cross-cutting concerns like logging, access control, or memoization. An example is '@cache def fibonacci(n):', where 'cache' is a decorator that memoizes the 'fibonacci' function to optimize performance by storing computed results, thus influencing the function by adding a caching behavior to it .

Context managers in Python simplify resource management by providing a structured way to allocate and release resources using the 'with' statement. This ensures that resources such as files or network connections are properly managed, including automatic cleanup after use. An example is 'with open('file.txt') as f: contents = f.read()', where the file is automatically closed after its contents are read, even if an error occurs, thus preventing resource leaks and simplifying exception handling .

The async/await syntax in Python enables asynchronous programming by allowing code execution to pause and resume based on I/O-bound operations without blocking the rest of the program. This is particularly beneficial for tasks like web requests or database operations that would otherwise wait for completion. An example is 'async def fetch_data(): data = await get_data()', where 'fetch_data' asynchronously calls 'get_data()', pausing its execution until 'get_data()' completes, improving efficiency by allowing other operations to proceed during the wait time .

List comprehensions with conditional logic allow for the concise and efficient creation of lists by integrating filtering conditions directly within the comprehension syntax. This approach reduces the need for loops and conditional statements, leading to cleaner and often faster-executing code. An example of their application is creating a list of squared values for even numbers: 'squared_even = [x**2 for x in range(10) if x % 2 == 0]', where only even numbers are squared and included in the list .

Cython is used to optimize Python performance by compiling Python code to C, which executes faster due to reduced overhead and potential for direct access to C libraries. This is particularly advantageous for computationally intensive sections of code where Python's execution speed is a bottleneck. A potential application is in numerical computations, where 'import cython; @cython.cfunc' can compile functions that involve heavy mathematical calculations, substantially improving their execution times by leveraging C-level optimizations .

A defaultdict is preferred over a regular dictionary when you need a dictionary that provides a default value for missing keys, avoiding the need for explicit checks or exceptions when a key doesn't exist. It is particularly useful in scenarios involving accumulation where missing keys are frequent, such as counting occurrences. For example, using 'from collections import defaultdict; dd = defaultdict(int)' initializes a defaultdict that assigns a default integer value of 0 to any missing key, simplifying the process of counting occurrences without checking for key existence .

functools.partial allows you to fix a few arguments of a function and generate a new function. This is advantageous for creating more flexible and customizable function calls without repeating argument specifications, thus enhancing code reuse. For example, 'from functools import partial; add_five = partial(add, 5)' creates a function 'add_five' that adds 5 to its argument. This enhances flexibility by allowing functions to be partially applied and reused in various contexts without redefining them .

NumPy and Pandas are optimized for performance through C-optimized functions and vectorized operations, which significantly improve the efficiency of data manipulation compared to native data structures like lists and dictionaries that require Python loops. NumPy provides multi-dimensional arrays and mathematical functions that operate on these arrays, while Pandas offers flexible, fast data structures designed for data analysis. For example, vectorization in Pandas allows operations on entire data columns in one go without explicit loops, making operations faster and code concise .

You might also like