>>> 'doesn\'t' # use \' to escape the single quote...

The lifetime of a request can be simplified as below:

>>> print('C:\some\name') # here \n means newline! 1. User types URL in browser
>>> print(r'C:\some\name') # note the r before the quote 2. Check cache (Y/N)
3. DNS lookup of IP address
Operator Operation Operation Meaning
+ Addition < strictly less than
4. Browser initiates TCP connection
- Subtraction <= less than or equal 5. Browser sends HTTP request
* Multiplication > strictly greater than 6. Server handles the request
/ Division >= greater than or equal 7. Browser receives HTTP response
// Floor division == equal 8. Browser renders content
** Exponentiation != not equal
% Modulus is object identity Pandas
is not negated object identity
[Link]() removes rows/columns with missing values.
fillna() replaces missing values with specified values
variables and types String a sequence of characters
1 string_a = "foo" - a sequence of Boolean True or False
iloc is purely integer-based location indexing.
characters Float Decimal numbers .loc is label-based indexing
2 int_x = 100 Integer Whole numbers use the [Link][x,y] operation to pick a specific cell in a table and
3 truthy = True Complex Complex number manipulate the value
4 falsey = False describe() gives count, mean, std deviation, min, 25th percentile,
median, 75th percentile, and max. df[x].describe()
Lists comma separate values inside [ ], Mutable arrays [Link]() function to print all values as true or false with false variables
Dictionaries Unordered, unique key-value pairs. Mutable {} having data and true values being nulls/none/empty
my_dict = {"key1": "value1", "key2": "value2"}
Tuples Immutable sequence, comma separated ( )
Sets Unordered collections, unique values email_pattern = r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"
email = [Link]+filter@[Link]
x+y sum of x and y -x x negated
x-y difference of x and y +x x unchanged
x*y product of x and y abs(x) absolute value or
x/y quotient of x and y magnitude of x
x // y floored quotient of x and y int(x) x converted to integer
x%y remainder of x / y float(x) x converted to floating
x ** y x to the power y point
pow(x, y) x to the power y
Tools like Selenium can automate browsers to capture such content.
complex(re, im) a complex number with real part re, imaginary part im. im
defaults to zero.
[Link]() conjugate of the complex number c
SQL
divmod(x, y) the pair (x // y, x % y) CREATE Creates a new table, view, or other database objects.
INSERT Adds new records into a table.
Data handling SELECT Retrieves data from a database.
f = open("[Link]", "r") WHERE Filters records based on one or more condition.
There are a few different flags that can be passed to open(), here’s a summary: UPDATE Modifies existing records in a table.
r Read DELETE Removes records from a table.
a Append (to the end of the file)
DROP Deletes an existing table in a database.
w Write. If the file doesn’t exist, it will also create the file
x Create a file. If the file already exists, it
produces an error GIT Commands JOIN Combines rows from two or more tables based on related
t Tells open() we’re dealing with a text file Update code columns.
b Tells open() we’re dealing with binary file Write code GROUP BY Groups rows with the same values in specified columns.
HAVING Filters the result of a GROUP BY operation.
for loop – Measure word Lenth git add ORDER BY Sorts the result set based on specified columns.
>>> # Measure some strings: git commit -m "" ALTER Modifies an existing table, such as adding or deleting columns
... words = ['cat', 'window', 'defenestrate'] git push
>>> for w in words:
... print(w, len(w))
SQL Injection Prevention
Cheack if up to date
cat 3 1. Sanitize Inputs: Ensure all user inputs are sanitized before
git Status
window 6 they're processed.
defenestrate 12 View Current Branch 2. Parameterized Queries: Use parameterized queries or
git branch prepared statements to separate SQL logic and data,
1 f = open("[Link]", "r") eliminating the risk of malicious data altering the query
2 data = [Link]() New Branch structure.
3 [Link]() git checkout -b new-branch- 3. Least Privilege Principle: Give the minimum required
name permissions to the database accounts. If a user doesn’t need
import csv
list = [] to drop tables, they shouldn’t have that permission.
Switch between Branches
with open("[Link]") as csvfile: git checkout branch-name 4. Regular Audits: Periodically review and audit your code
reader = [Link](csvfile, delimiter = ",") and databases for vulnerabilities.
for row in reader: see diffrence between files
[Link](float(row[6])) git diff name-of-branch 2D visualizations are often preferable to 3D for several
list = list[1:] Undo add reasons:
print(sum(list) / len(list)) git reset 1. Clarity: 2D visualizations tend to be more straightforward and easier
git reset filename to interpret.
# Sorting
print(df.sort_values(by='Value', ascending=False, na_position='first')) 2. Distortion:3D can sometimes distort data, giving a misleading
# For Loop example # While Loop example representation.
for item in my_list: count = 0 [Link]: 2D charts and graphs are generally more mobile-friendly
print(item) while count < 3: and accessible.
print(count) [Link]: In many cases, adding a third dimension doesn't
count += 1 provide additional clarity and instead makes the graph more difficult to
understand.
 Numpy:
NumPy, short for Numerical Python, provides the data structures,
algorithms, and library glue needed for most scientific applications
involving
numerical data in Python. NumPy contains, among other things:
A fast and efficient multidimensional array object ndarray
Functions for performing element-wise computations with arrays or
mathemati-al operations between arrays
Tools for reading and writing array-based datasets to disk
Linear algebra operations, Fourier transform, and random number
generation
Array Creation Functions [Link](), [Link](), [Link](),
[Link](), etc.
Array Manipulation Functions [Link](), [Link]()
Array Mathematical Functions [Link](), [Link](), [Link](),
[Link](), etc.
Array Statistical Functions [Link](), [Link](), [Link](), and
Split words from paragraph [Link]().
wordLines = [[Link](" ") for x in lines] Array Input and Output Functions [Link](), [Link](), [Link](),
dict = {} etc
for line in wordLines:
for word in line: Databases:
if word in [Link](): Non-Relational
Merits - Their schema-free nature makes managing and storing vast
dict[word] += 1
volumes of data easier. They can also be easily scaled horizontally.
else: Data is not too
dict[word] = 1 complex and can be distributed among several distinguished nodes for
better accessibility.
Regular Expressions serve as powerful tools for text processing and Demerits - Since they have no specific structure or schema for the
pattern matching. Use the re module for regex operations in Python. data stored, you cannot rely on your data for a particular field because
[Link]() searches a string for a match, and returns a Match object if it might not
found. Metacharacters: have it. Having no relations makes it very hard to update the data, as
? – non-greedy; [^ ] – matches what’s not in the set; \S – any non- you will have to update every detail separately.
whitespace; \s – any whitespace; $ - end of line; * – zero or more; + –
one or more Relational
Merits - follow a strict schema, each new entry must have different
JSON data format follows a hierarchical, tree-like structure with components that fit in that preformed template. It enables the data to
nested objects and arrays. Use the “import json” library. be predictable and easily assessable. ACID compliance Is a must.
Well structured and significantly reduce the chances of errors.
Beautiful soup - library for web scraping navigating web pages Demerits: The exact nature, strict schemas, and constraints of
through the Document Object Model (DOM) relational databases make storing the numbers required for today’s
mammoth internet data nearly impossible. Iimpossible to scale
Data Provenance means data ownership, or in simple words; what horizontally as relational databases follow a particular schema.
happened to the data before it’s arrived on our doorstep. Although vertical scaling seems like the obvious answer, it’s not.
Vertical scaling has a limit, and, in this time, and age, the data
Classes are a way of collecting data and functions together. It also collected via the internet daily is too large to imagine that vertical
provides a means for getting a pre-existing set of functions, data, and scaling would work for long. Schema constraints also impede data
extending upon it. We can reuse bits and pieces as want. migration to and from different RDBMS. They need to be identical;
otherwise, it will not simply work
Creating a visualization – 3 Aims
Diagnostics: to test assumptions or hypotheses about our data. In the context of databases, tables are viewed as sets, and records
Exploratory analysis: summarizing characteristics within those tables are seen as objects in a set
Findings: Multi tiered analysis
Pip Install - the command to install the libraries we need
Exploratory data analysis (EDA) is a method of analyzing and pip freeze - that shows all the libraries in use right now
summarizing data sets before making any assumptions or hypotheses pip list- that shows you all the current packages in use in your program

EDA Steps: Test Driven Development:

Data Collection: Retrieved historical stock market data from Yahoo Three laws - You may not write production code unless you've first
Finance, including daily prices, volumes, and relevant metrics. written a failing unit test; You may not write more of a unit test than is
Data Cleaning: Removed inconsistencies, missing values, and sufficient to fail; You may not write more production code than is
outliers to ensure data quality and consistency across sources. sufficient to make the failing unit test pass.
Data Visualization: Utilized Python libraries like Matplotlib to generate Setup - assertEqual is a method of the TestCase class found in the
visualizations, such as time series plots, to identify patterns and unittest module in Python, which is used for writing tests for your code.
[Link] Analysis: Calculated summary statistics (mean, This method is used to compare two values, and it tests if they are =
median, standard deviation, correlation coefficients) to understand Import module, create class from [Link], inside class
central tendency, dispersion, and variable relationships. create method named test which has assert statement
Risk Analysis: Evaluated key risk metrics (volatility, beta) using
historical stock prices to assess investment risks. Return Analysis: 3 features of a development environment
Computed stock returns based on historical prices and compared Integrated Development Environment (IDE):
them with market benchmarks (e.g., S&P 500) to gauge relative Version Control System (VCS):
performance. Build Automation Tools:
Data Visualization Challenges: Describe two ethical issues associated with web scraping.
Overplotting: Occurs when many data points overlap, obscuring •Copyright infringement/theft of intellectual
patterns and trends, impacting interpretation, especially in scatter and property through excessive scraping e.g. "how
density plots. much is too much?" discussion varies between
Choosing the Right Visualization: Selecting suitable chart types for different types of media. [- Excessive requests
datasets is crucial to avoid inaccurate representations, such as using e.g. manifesting as a Denial of Service attack.
inappropriate charts like pie charts for comparing many categories.
Color and Aesthetics: Picking appropriate colors and aesthetics is Viable technique for scraping
essential for effective communication; poor choices may lead to Valid technique e.g. headless browser or other library
misinterpretations or accessibility issues for color-blind users. that handles dynamic events- simulating a browser and embedded
Handling Large Datasets: Python libraries like Matplotlib and Seaborn functionality. Selenium headless could be used to automate clicks
struggle with efficiently rendering large datasets, resulting in slow and trigger events so that appropriate HTML is accessible during
rendering times and cluttered plots, hindering exploration and different states of interaction such as logged in state.
communication.
Interactive Visualizations: Creating dynamic plots for user Assume we have a dataset with missing or erroneous data. Describe
exploration requires additional effort but enhances engagement and three strategies to identify erroneous data
enables deeper data exploration compared to static visualizations. Look for outliners – data outside the range
Visualise data – use graphs to see incorrect data
Web Scraping Overview: Validate data against other available data sets
Web scraping is a method to extract unstructured data from websites Explore inconsistent datatypes – eg strings, where type should be int
and convert it into structured data for various purposes such as data
mining, analysis, visualization, and machine learning. Key Steps: Find data duplication
- summing, counting values vs expected values
Identify Target Website: Choose the website to extract data from. - compare against other dataset eg static database and webscraping
Inspect Page: Use browser developer tools to understand the page's - dataframe solution, [Link]( subset = none, keep= “first”)
structure (HTML, CSS). Unique Identifier Check: Ensure uniqueness of identifiers like
Select Data Elements: Identify specific elements (text, images, customer or transaction IDs.
tables) for scraping. Exact Match Check: Compare rows or columns to detect exact
Write Code: Develop a web scraper using languages like Python or duplicates. - Regex solution
JavaScript. Fuzzy Matching: Use algorithms to find similar records, even with
Send HTTP Requests: Use libraries like requests to fetch webpage variations.
content.
Parse HTML: Extract relevant data using parsing libraries (e.g., Mutable Objects: Immutable Objects:
Beautiful Soup). Mutable objects are those Immutable objects, on the other
Store Data: Save extracted data locally or in a database. whose state can be modified hand, are those whose state
Handle Pagination: Manage scraping across multiple pages if needed. after creation. cannot be modified after
Respect [Link]: Adhere to website scraping rules outlined in This means that you can change creation. Once an immutable
[Link]. their internal state or contents object is created, its state
Error Handling: Account for potential errors like missing elements or without changing the identity of cannot be changed. Any
connection issues. the object itself. operation that appears to
Regular Maintenance: Periodically update the scraper to adapt to Examples of mutable objects modify the object actually
changes on the website. include lists, dictionaries, sets, creates a new object with the
and objects of custom classes modified state. Examples of
Find longest word in file where the attributes can be immutable objects include
def find_longest_word(filename): modified. integers, floatingpoint numbers,
with open(filename, 'r') as file:
strings, tuples, and frozen sets
words = [Link]().split()
longest_word = max(words, key=len) Pandas – 5 Characteristics
return longest_word Tabular Structure: Data organized in rows and columns, akin to a
spreadsheet.
Find longest word in sentence Heterogeneous Data Types: Supports mixed data types within
def find_longest_word(sentence): columns.
word_list = [Link]() # Split the sentence Indexing and Selection: Allows selection of data using labels or
into words integers.
longest_word = '' # Initialize the longest word Missing Data Handling: Provides methods for detecting and handling
for word in word_list: missing values.
if len(word) > len(longest_word): Data Alignment and Broadcasting: Aligns data based on index labels
longest_word = word # Update longest_word and supports broadcasting operations.
if the current word is longer
return longest_word Check if Password meets requirements
def check_password(password):
if len(password) < 4 or len(password) > 10:
calculate the number of upper and lower case characters return False
Str="GeeksForGeeks" # Example usage (Password):
has_upper = False
lower=0 has_lower = False password = input("Enter your
upper=0 has_number = False password: ")
for i in Str: for char in password: ifcheck_password(password:
if([Link]()): if [Link](): print("Password meets the
lower+=1
has_upper = True criteria.")
elif [Link](): else:
else:
has_lower = True print("Password does not
upper+=1 elif [Link]():
meet the criteria.")
print("The number of lowercase characters is:",lower) has_number = True
print("The number of uppercase characters is:",upper) return has_upper and has_lower and has_number
VersionControl Identify erroneous data Web scraping
It is a system that tracks and manages • Identify outliers or unusual patterns Process of extract data from websites.
changes to files and code, often used in through summary statistics (mean, Ethical issues: Copyright infringement/ Theft
software development. It helps teams median, standard deviation) and of intellectual [Link]: Scraping
collaborate, maintains a history of changes, visualizations (histograms, box plots, and republishing copyrighted content, such
and allows for code rollback when issues scatter plots). as articles.
arise. Git is a popular version control tool • Example: In a dataset of people's Terms of service Example: Scrap content
widely used in the industry. ages where the mean is 25, but from a website where itsterm of service
Git has a distributed architecture, it means there are values like 200. prohibit web [Link] number of
that every dev has a copy of the code • Use pattern recognition or regex to requests may overload theserver. For
repository. identify data that do not conform. example: manifest a Denial of Service attack.
Centralized VCS vs Distributed VCS • Example: In a dataset of phone Techniques: Regular Expressions (Regex).
• DVCS allows devs to work offline, CVCS numbers, there may be some value Regex are patters used to match and extract
require connection to a central server. that lacks digits or contains special specific data from alarger string. It's useful for
• DVCS offers effortless branching and characters. extract data when patterns are well defined.
merging. CVCS involves more complex • Explore inconsistent data types. HTTP Requests and HTML Parsing. It's simply
branching and merging workflows. • For example: In a dataset, there making a request to the webpage URL using
• DVCS enhances data security and may be some value that is a string libraries in Python (for example), and parse
redundancy with full repository copies where an int is expected. the content.
on each developer's machine. CVCS
offers security risks since it only relies 2D visuali1zation vs 3D visualization Test Driven Development
from one central server. • 2D Advantage: Clearer and simpler, it's more It's a programming methodology that
Missing data intuitive for most people. consists in writing tests for some code
It's the absence or lack of values in a Disadvantage: Lack of ability to represent before actually implementing the code itself.
dataset. data with depth or volumetric information. Its steps are: write a test, write the code,
How to deal with it: • 3D Advantage: Can represent spatial run the test, and refactor (if needed).
1. Replace the missing values with NaN. relationships and data with depth or Advantages of TDD
2. Then we can either: volumetric information. • Improved code quality: the rigorous
• Drop the rows/columns that have Disadvantage: More complex and clutter, testing ensures that the code functions
some missing value. specially with large datasets. correctly.
• Drop rows/columns that only Overcome 3D issues - allow users to • Easier to refactor code: old code can be
contains missing values. manipulate and explore the data in real-time, refactored easily using tests suites.
• Fill with some value e.g. the mean enabling them to zoom in, rotate, and pan
value. around the visualization Python has 3 distinct data types
Integers (int) - are whole numbers without a
Data visualization NLP- Natural Language Toolkit steps: decimal point
• Line plot 1. Segmentation = break data into sentences Floating Point Numbers (Float) - are numbers
import [Link] as plt sentences = nltk.sent_tokenize(text) with a decimal point.
x = [1, 2, 3, 4, 5] 2. Tokenizing = break sentence into words Complex numbers - numbers with a real part
y = [10, 12, 5, 8, 7] words = nltk.word_tokenize(sentence) and an imaginary part
[Link](x, y) 3. Stop Words = mark down 'Verb to be.,
[Link]('X-axis') prepositions, ...etc...'
[Link]('Y-axis') stop_words = set([Link]('english'))
[Link]('Line Plot') filtered_words = [word for word in words if
[Link]() [Link]() not in stop_words]
• Bar Chart 4. Stemming = same words with different
import [Link] as plt prefix or suffix stemmer = PorterStemmer()
# Sample data words = ["jumping", "jumps", "jumped",
categories = ['Category A', 'Category B "jumper"]
values = [25, 40, 30] stemmed_words = [[Link](word) for
# Create a bar chart word in
[Link](categories, values) words]stem(word) for word in words]
[Link]('Categories') 5. Lemmatization = learning that multiple
[Link]('Values') words can have the same meaning (is, am, are
[Link]('Bar Chart') >>> be)
[Link]() lemmatizer = WordNetLemmatizer()
• Histogram words = ["running", "flies", "better"]
import [Link] as plt lemmatized
# Sample data
data = [2, 3, 3, 4, 4, 4, 5, 5, 6, 7] #Extract the first 9 records from the 'cars'
# Create a histogram DataFrame first_9_records = [Link](9)
[Link](data, bins=5)
[Link]('Values') Remove whitespaces from a string
[Link]('Frequency') text = " Hello, World! "
[Link]('Histogram') cleaned_text = [Link](" ", "")
[Link]() Or you can do this
cleaned_text = "".join([Link]())
Function that finds and shows the length of
values contained in a dictionary: print out the second hottest day temp
length_dict = {} sorted_temps = sorted(temps, reverse=True)
for key, value in [Link](): print(sorted_temps[1])
length_dict[key] = len(value)
return length_dict