Python: The Basics Lists [ ] Non-permanent methods Diccionarios { key : value , } zip() Control statements

create an empty list dictionary = {x:y} composed of a unique key(x) zip(iterable1, iterable2) creates a list of tuples from if ... elif ... else
Extended variables by text and a value(y) (any type of data) pairs of elements from the two lists (while you Establish a condition for the code to execute that
can it is below the if. *it must be indented*
CONCATENATION len(lista) returns the number of elements dict() elif to check more conditions after an if
min(lista)/max(lista) extracts the minimum and maximum value variable = dict(x=y, m=n)crear un diccionario [Link]() sorts the tuples of the zip by the first else group the conditions that have not been met; it cannot
To chain text element bring new conditions
[Link]() returns the number of elements that are in
categoria1 = "verde" [Link]() create a copy
the list of a certain value in the() if x > y:
color_detalle = categoria1 + ' ' + 'oscuro" len(dicc) returns the number of elements (x:y) present in
sort(lista) to sort a list from smallest to largest the dictionary
Sets {} x is greater than y
elif x == y:
print(categoria1 + ' dark') no duplicates allowed, they are unordered x is equal to y
[Link]() to make a copy of the list sorted(dicc) sorts the keys; use with .items()
print(category1, 'dark') to sort tuples of the elements or .values() else:
set = {x,y}
to sort the values alone x and y are equal
type() and isinstance() Methods with indices
set(iterable) only allows one iterable argument; removes
duplicates while
[Link](x) returns the index of x in the list
float/int/str(variable) changes the data type/type Dictionaries–Methods to check whether there is an element
repeat the code while
it will stop when the condition is False
the condition is True, that is
list[i] returns the element at index i
type(variable) returns: class 'float/int/str' Conditions can be included with if... elif... else
[start:stop:step] Obtain information from a dictionary len(set) returns the number of elements
they can be infinite (if the condition does not become
list[i:j:x] returns the elements within the range of i [Link]() returns all the keys False
isinstance(variable, float/int/str) check the type
a j (includes i but not j) jumping by x Expand a set
of the date (returns True/False) [Link]() returns all the values
lista[-i:-j] returns the elements by the indices while x < 5:
[Link]() returns tuples of the key:value [Link](x) #add an element
negatives (includes –j but not –i) x is greater than 5
check if a key exists/not
Algebraic Operations [Link](x, y) returns the value associated with the key x,
[Link](set or list) #add one or more elements with
[] or {} or a variable of list or set type
add / divide or if it does not exist return the output and For loops
subtract divider and round (modulus) Lists - Permanent Actions dicc["key"] returns the value of the key (see below Remove elements from a set They are used to iterate through all the elements of a variable.
multiply remainder of a division (floor that has more uses
to raise division) [Link]() #removes a random element what has to be an iterable (list, dictionary, tuple,
Expand a list set, or string
round(x) [Link](x) #removes the element x
[list1, list2] merges lists but they remain as Expand a dictionary [Link](x) # removes the element x (and does not return They can be combined with if ... elif ... else, while, or another.
separated lists
Binary Operations list1 + list2 makes a longer list
.update() error if it does not exist for loop
[Link]({x:y}) #to insert new elements [Link]() #empties the set In default dictionaries, it iterates over the keys; we can
== check if values match use [Link]() to access the values
.append() dicc["key"] = value # to insert a new key or
is to check if values are exactly equal value, or change the value of a key for i in list:
!= check if values are different [Link](x) #adds a single element (list, Operations with two Sets
dicc. setdefault(x, y) # returns the value of the key print("hello world")
is not to check if values are not exactly string, integer, or tuple to the list x, or if the key x does not exist, it creates it and assigns the
equals [Link](set2) returns the union of the two sets: all
.extend() value and by default
greater than (greater than or equal to)
[Link](lista2) #adds the elements of one
the least duplicate elements. List comprehension
< (<=) less than (less than or equal to) [Link](set2) returns the common elements of
list at the end of the list
both true Remove elements from a dictionary the two sets Its main use is to create a new list from a for
or gossips or just a true one .insert() [Link](x) #removes the key x (and returns it) loop in a single line of code
[Link](set2) returns the sets that are in set1
in/not check if there is a value in a list etc. .insert(i, x) # inserts an element (x) at an index (i) [Link]() # removes the last key:value pair but not in set2 (subtract) [ what we want to obtain iterable condition (optional) ]
[Link]() # empties the dictionary
Sort a list set1.symmetric_difference(set2) returns all the
elements that are not in both
String Methods .sort() try ... except
[Link](set2) check if all elements of
[Link]()zUPPERCASE
[Link]() # sorts from smallest to largest, use with Tuples (,) immutable, indexed two sets are different
(reverse=True) to sort from highest to lowest They are used to prevent our code from stopping due to an error.
[Link]()lowercase [Link](set2) check if all elements of
[Link]()#sorts the elements in reverse order tuple = (x, y) tuples are defined with () and , or only ,
In the code. A message can be printed that warns of the error.
First letter of the sentence in uppercase. set1 is in set2
First Letter Of Each Word In Uppercase. order saved try:
tuple1 + tuple2 join tuples
[Link]() lower case to upper case or vice versa [Link](set2) check if all elements of
print("[Link]())
[Link]() removes spaces from the beginning and end Remove elements from a list set2 is in set1
tuple(list) create tuples from a list except:
.pop() tuple(dicc) create tuples from the keys of a does not work
[Link]() divides string into list - by spaces
[Link](i) #removes the element at index i and dictionary
by default, or specify another delimiter in ()
returns its value input()
[Link]("phrase", "phrase") replaces the first tuple([Link]()) create tuples of the values
phrase of the string for the other .remove() tuple([Link]()) create tuples of the key:values allows obtaining text typed by the user range()
Join the elements of a list in [Link](x) # removes the first element from the list the text that you want to show to the user
a string with the specified separator in " " with value x len(tupla) returns the number of elements
It can be stored in a variable it returns a list of numbers that by default
list(string) converts a string variable into a check if there is an element increase by one starting from 0
[Link]() #empties the list By default, it is saved as a string.
list [Link](x) returns the index of x range(start:stop:step)
[Link]("substring") finds the index where [Link](x) returns the number of elements with x = int(input("write a number")) to use the variable
delete list You can specify where it starts and the limit (which must
start the substring/'-1' if the substring does not exist value x in the tuple how an integer or float can be converted into the variable
del list[i] # deletes the element at index i be +1 because one stops before the limit we set as
stop)
string[i] returns the element at index i to change the content of a tuple one has to
You can also specify jumps.
string[i:j] returns a range of characters convert it into a list and then to a tuple

# permanent methods (changes the variable, does not return anything)

Functions and Classes; Libraries Regex Modules/Libraries (function packages) XML files MySQL Connector/Python
an abbreviation of 'regular expression' Import and use modules and their functions import the xml library Obtain results from a query
Functions `regex` is a string of text that allows import module to import a module variable_tree = [Link]('path/[Link]') opens the
variable_cursor.fetchone() returns the first result
create patterns that help to match, from module import function just import one function file
locate and manage strings variable_root = variable_tree.getroot() extracts the element variable_cursor.fetchall() returns all the results
Define a function: [Link]() use a function from a module
[Link]() to use a function from a class that wraps everything (the root element) in a list as iterable – each row is a tuple
def function_name(parameter1, parameter2, ...): import it to work with regex
import module as mdassign an alias to a module <root>
return return_value Pandas dataframe with SQL
Common regex operators <child_tag attribute1="value" attribute2=value>
subchild_tag element import pandas as pd
Llamar una funcion: + matches the preceding character one or more times OS library
</child_tag>
function_name(argument1, argument2, ...) times [Link]() returns the path of where we are working; it variable_df = [Link](variable_resultado_fetchall,
</root>
matches the previous character zero or You can store it in a variable e.g. path = [Link]() columns = ['column1', 'column2', ...]) create a
variable_root.tag returns the name of the root tag
return: is optional, but without return it returns None more times or optional [Link]() returns a list of files and folders dataframe with the results of a query in a variable
variable_root.attrib returns the attributes of the file
default parameters: – should always be where we are working
indicate zero or one occurrence of the element variable_df.head(n) returns the first n rows of the df,
last [Link]('folder') returns the contents of another folder
precedent variable_root.find("tag").find("childtag").text returns 5 by default
[Link]('path') changes the folder in which you are.
coincide with any individual character the first time an element's tag matches variable_df = pd.read_sql_query(variable_query,
*args: a tuple of unlimited arguments [Link]('new_folder') create a new folder
with the string variable_cnx)convert the query results into df
dictionaries whose keys are converted into matches the initial position of any [Link]('folder_name', 'new_name') changes the name
variable_root.findall("tag").findall("childtag").text
parameters and their values in the arguments of the string from a folder pd.read_sql(variable_query, variable_cnx)
return all elements whose tag matches
parameters it matches the final position of any [Link]('folder') deletes the folder
variable_df.to_csv("[Link]") save to csv
string
def function_name(parameters, *args, **kwargs, shutil library MySQL Connector/Python variable_df.to_string() format the data as string
Basic regex syntax from shutil import rmtree
default_parameter = value format the data into a string that
arg/kwarg: sin */** inside the function any character of alphabetical type rmtree('folder') deletes the folder and subfolders Connect to a database facilitates the insertion in a latex document
arg[0] any character of numeric type import [Link] to import MySQL Connector
spaces Open and close files Create and alter a database
pip install mysql-connector
Call a function with *args: line breaks First, you need to save the file path:
pip install mysql-connector-Python variable_cursor.execute("CREATE DATABASE database_name")
function_name(argument, argument, argument, ... ) ubicacion_carpeta = [Link]()
Any character that is not a letter connect() to connect to a database: variable_cursor.execute("CREATE TABLE table_name
o nombre_archivo = "[Link]"
Any character that is not a digit ubicacion_archivo = ubicacion_carpeta + "/" + nombre_archivo variable_cnx = [Link](user='root', (column_name TYPE, column_name2 TYPE2)
function_name(*[list_or_tuple_of_args])
Any element that is not a space password='AlumnaAdalab', variable_cursor.execute("ALTER TABLE table_name
f = open(file_location) open a file in variable f host='[Link]', ALTERATIONS
Call a function with **kwargs: Isolates only a part of our pattern of
database='database_name'
search that we want to return [Link]() close a file * IMPORTANT *
function_name(**dictionary) Insert data
with open(file_location) as f: import errors from [Link] import errorcode
includes all the characters we want
code e.g. variable = [Link]() opens the file only for variable_query = "INSERT INTO nombre_tabla (columna1,
that match and even includes ranges such as [Link] can be used in a try/except
execute the indicated code (and then leave it) column2) VALUES (%s, %s)
this: a-z and 0-9
Classes it's like the 'or' operator
[Link]() disconnect from the database
variable_valores = (valor1, valor2)
Encoding Make queries
Define a class: point out a special sequence ( escape from locale import getpreferredencoding variable_cursor.execute(variable_query, variable_values)
special characters) create the cursor object that
class ClassName: getpreferredencoding() to find out what encoding system another method:
Exactly the specified number of allows us to communicate with the database
we are using
variable_query = "UPDATE nombre_tabla SET nombre_columna =
occurrences open a file variable_cursor.close() disconnect the cursor
def __init__(self, attribute1, attribute2): new_value WHERE column_name = 'value'
Exactly n times and read it with the used encoding; save with .read() save a query in a
self.atribute1 = attribute1
self.atribute2 = atribute2 At least n times variable Insert multiple rows into a table
Between n and m times
mode: optional argument when opening a file execute the query
self.atributo_por_defecto = 'valor' variable_values_in_tuples = ((value1column1,
r–read returns a list of tuples (value2column1, value2column2), (value1column2), (value2column1, value2column2))
w–write - overwrite
def function_name1(self, parameters) import datetime get dates in the format YYYY-MM-DD variable_cursor.executemany(variable_query,
Regex Methods x–exclusive creation, only create it if it does not exist yet
variable_values_in_tuples
[Link] += 1 a–appending, adding text to the file without manipulating the text [Link](YYYY, M, D) returns the date format
[Link]("patron", string) searches throughout the
the new value is {[Link]} that had already
string and returns a list with all the dynamic query
we need to add another letter: variable_connection.commit() after executing the
matches in our string t-text-read in text variable_cursor.execute(query, (variable1, variable2))
Define a child class: insertion, so that the changes take effect in the database
b–bytes–read in bytes (cannot be used with encoding) values that go in place of the %s
class ChildClassName(ParentClassName): [Link]("patron", string_original) searches in variable_conexion.rollback() can be used after
the entire string and return an object with the variable_cursor.execute("SHOW DATABASES") show the databases
def __init__(self, attribute1, attribute2): f = open(file_location, mode = "rt") execute y before commit to undo the changes
first match in our string show the tables
super().__init__(inherited_attribute1, ...) print(variable_cursor.rowcount, "message")
from the database indicated in the connection
[Link]("pattern", "original_string") searches in Read files number of rows in which action has been taken
def child_function_name(self, parameters): the first line of the string and returns a variable_cursor.execute("SHOW TABLES")
[Link]() read the contents of a file
object with the first match in our [Link](n) reads the first n characters of a file variable_cursor.execute("SHOW COLUMNS FROM [Link]") Delete records
string variable = [Link]() save the content of the file (or n show the columns of the specified table; one must variable_query = "DROP TABLE nombre_tabla"
Create an object of the class:
characters of a file) in a variable connect to the information_schema database
variable_object = ClassName(attribute_value1, resultado_match.span() returns the reference
value_attribute2) instantiate (create) an object from the positions where he made the "match"
[Link](n) by default returns the first line or n lines
Cursor arguments: Add errors
[Link]() returns a list of all the lines from the import errorcode and use try/except:
variable_object.attribute returns the value of the file (each line is an element); it is used empty without n and variable_cursor = [Link]([arg=value[, arg=value]...])
resultado_match.group() returns the element
attribute saved for that object list_name[x:] to select specific lines try:
resulting from the match coincidence buffered=True returns all rows from the database
variable_object.attribute = new_value_to_change action
the value of the attribute [Link]("patron", "string_original") searches in raw=True the cursor will not perform the conversions except [Link] as err:
Writing to files automatic between data types
variable_object.function_name() call a function take the whole string and return a list with the print(err)
with open(file_location, "w") as f:
elements separated by the pattern dictionary=Truedevuelve las filas como diccionarios print("Error Code:", [Link])
[Link]("Text that goes in the file.") to write
print(help(ClassName) prints information about the print("SQLSTATE", [Link])
[Link]("patron", "new_string", with open(file_location, "a") as f: named_tuple=Truereturns the rows as named tuples
class print("Message", [Link])
"string_original") searches through the entire string and text to be added to the file.
cursor_class an argument that can be used to indicate
returns a string with the element that matches [Link]('list') to add lines of text from a list which subclass do we want to use to instantiate the new cursor
Python: Pandas DataFrames Exploration methods Data types Null values
Crear DataFrames [Link](n) returns the first n lines of the dataframe Data types in Pandas: Identify nulls
Series: structures in one dimension df = [Link](data, index, columns) [Link](n) returns the last n rows of the dataframe object
[Link]() and [Link]() return True or False depending on whether each
data: NumPy Array, diccionario, lista de diccionarios int64
[Link](n) returns n random rows from our float64 value is null or not
Create series index: default index assigned as 0-(n-1), n
dataframe, or one by default - datetime, timedelta[ns] [Link]().sum() [Link]().sum() returns a series with the
series = [Link]() create empty series being the number of rows;
index = [list] to assign 'labels' (names of [Link] returns the number of rows and columns. - category number of null values by columns
series = [Link](array) create series from a boolean
rows) df_%_nulls = (([Link]().sum() / [Link][0] *
array with the default index [Link] returns the data type of each
column: name of the columns; by default 0-(n-1); [Link] returns the data type present 100).reset_index()
series = [Link](array, index = ['a', 'b', 'c'...]) column
columns = [list] to add more names in each column df_%_nulos.columns = ['column', '% nulls'] create a dataframe
create a series with defined index; it must be a list of
the same length of the array [Link] returns the names of the columns df_type = df.select_dtypes(include = 'type') of the percentages of null values
df = [Link](array) create a dataframe from a create a dataframe of columns of the type of
series = [Link](list) create a series from [Link] returns a dataframe with a summary of the
array with default indices and columns specified data Remove nulls
a list main statistics of the numerical columns
create a dataframe from df['column'] = df['column'].astype('type', [Link](inplace = True, axis=b, subset=[list_of_columns],
series = [Link](number, index) create a series to
to start from a scalar with the longitude equal to the number from a dictionary - the keys are the names of the [Link]() returns a summary about the number of columns, copy = True, errors = 'ignore') converts a how to remove nulls
columns column names, number of non-null values, and the column in the specified data type
of indices how = 'any' | 'all' by default 'any': if there is any NA value,
data types df["column_name"].unique() copy = Truedevuelve una copia
series = [Link](dictionary) create a series a the row or column is removed; all: if all values are NA,
df.nombre_columna.unique() returns an array with the copy = False*warning: changes in the
starting from a dictionary the row or column is deleted
DataFrames: data loading unique values of the column values can be propagated to other objects
subset a column or list of columns
pandas*
Access information about a series df['column_name'].value_counts() errors = ignore omit exceptions; in case of Types of nulls
[Link] returns the indices Data loading df.nombre_columna.value_counts() returns a series with error returns the original object
[Link] returns the values df = pd.read_csv("path/file_name.csv") create a the count of unique values in descending order [Link] means 'not a number'; it is a numeric type
errors = raise allows for the generation
[Link] returns the shape (number of rows) dataframe from a Comma Separated Values file No null values in string type columns
[Link]().sum() returns the number of rows exceptions NaT null values of datetime type
[Link] returns the size df = pd.read_csv("path/filename", sep= ";") create a
dataframe from a csv if the separator is ; duplicates values text: "n/a", "NaN", "nan", "null" strings that
[Link] returns the data type
they are usually automatically converted to [Link]
df = pd.read_csv("path/file_name", index_col= 0) Remove duplicate rows
serie[i] returns the value of the element at index i create a dataframe from a csv if the file already has one 99999o00000 integers that can be converted to nulls
df.drop_duplicates(inplace = True, ignore_index=True) [Link].max_columns = Noneexecute
series[[i,j]] returns the value of the two elements index column before [Link]() to be able to see all the Replace nulls
serie[i:m] returns the value of a range remove duplicate rows; ignore_index to avoid having the
columns
df = pd.read_excel("path/[Link]") create a index in account df = pd.read_csv('[Link]', na_values = ['n/a'])
series['label'] returns the value of the elements dataframe from an Excel file .fillna([Link]) replaces the strings 'n/a' with [Link]
pd.set_option("[Link]", 2)
in indices i and j - if you get 'ImportError:... openpyxl...', in the terminal: Statistical methods load the dataframe
pip3 install openpyxlopip install openpyxl
Operations with series
Outliers [Link](df[value=n, axis=b, inplace=True) replace all
the NaN of the dataframe with the value we specify
df['columna'].mean() | mode() | median() | var() | std()
series1 +-*/ series2 add/subtract/multiply/divide them df = pd.read_json("path/[Link]") create a
calculate the mean/mode/median/variation/standard deviation Calculate three standard deviations: df['column'].fillna(df['column'].median, axis=b,
rows with common indices between the two series dataframe from a JavaScript Object Notation file
standard of the values of a column media = [Link]() replace the nulls in a column with the
[Link](serie2, fill_value=número) sums the rows (raw format)
desviacion = [Link]() median of that column
with common indices, and add the fill value to the convert the dataframe of df['column1'].corr(df['column2'] calculates the
value by default is NaN; it is the value for which we want
values without common index JSON in a readable format correlation between two variables
lcb = media - deviation * 3 replace the null values that can be a scalar,
[Link](series2, fill_value = number) subtract the correlation_matrix = [Link]() creates a matrix showing dictionary, series or dataframe
create a dataframe from ucb = mean + standard deviation * 3
rows of series2 of series1 when they have indices the correlations between all variables default axis 0 (rows)
data in the form of a dataframe in the clipboard; the separator
common, and subtract the fill value from the other indices of df_crosstab = [Link](df['column1'], [Link](null_value, new_value, inplace=True, regex=False)
series1 could be ; , etc.
df['column2'], normalize = True, margins = True
Remove Outliers replace the nulls with the new value
[Link](serie2, fill_value = number) multiplies the outlier_step = 1.5 * IQR calculate outlier step
with open('path/[Link]', 'wb') as f: normalize shows the values in percentages (by one)
rows with common indices and multiply the fill value outliers_data = df[(df['columna'] < Q1– Null imputation
with the others *use 1 to keep the value* [Link](df, f) puts the data of a dataframe in margins show the totals and subtotals outlier_step) | (df['column'] > Q3 + from [Link] import SimpleImputer
[Link](series2, fill_value = number) divide the the pkl file weighted_average = [Link](df['column'], weights = w) identify outlier data
range from maximum to minimum imputer = SimpleImputer(strategy='mean', missing_values =
rows of series1 between those of series2 when calculate the weighted average according to the weights
pd.read_pickle('path/[Link]').head(n) read n outliers_index_list = list(outliers_data. [Link]) initiates the instance of the method, specifying that
they have common indices, and divide the others by the fill percentil_n = [Link](df['column'], n) remove the
rows and 5 columns of the pickle file index) create a list of the indices of the we want to replace the nulls with the mean
value value in the nth percentile
[Link](serie2, fill_value = number) returns the rows with outliers imputer = [Link](df['column1']) we apply the imputer
modulo (division without subtraction) pd.read_parquet('path/[Link]') read a q3, q1 = [Link](df['column'], [75, 25]) take out the
df['media_columna1'] = [Link](df[['price']]) fills
[Link](serie2, fill_value = number) calculates the parquet file third and first quartiles if outliers_data.shape[0] > 0: the null values according to how we have specified
exponential dicc_indices[key] =
from [Link] import enable_iterative_imputer
Data storage (list(outliers_data.index)) create a
[Link](series2) compares if series1 is greater than
series2 and returns True or False df.to_csv('path/file_name.csv') save dataframe Side table: data frequencies dictionary of the row indices with
from [Link] import IterativeImputer
[Link](serie2) compares if serie1 is less than as csv file nulls; it can be done by iterating through columns imputer = IterativeImputer(n_nearest_features=n,
df.to_excel('path/file_name.xlsx') save dataframe [Link](['column']) returns a dataframe with create the instance
series2 and returns True or False
as an Excel file information about the frequency of occurrence of each values = dicc_indices.values() take all n_nearest_features defaults to None; number of columns to
df.to_json('path/[Link]') save dataframe category of a categorical variable the values e.g. all the indices use to estimate the null values
Boolean filtering parameters:
as a JSON file values = {index for sublist in values for imputation order by default ascending; the order of imputation
series < > >= <= == value returns True or False depending on thresh = limits the displayed values to the most
df.to_parquet('path/[Link]') save index in sublist} set comprehension for
if each condition meets the requirement frequent up to a threshold of n% cumulative and grouping we apply the imputer
dataframe as parquet file remove duplicates
series1[series1 < > >= <= == value] returns only the the remaining ones under the label 'other' df_datos_trans = [Link]([Link](df_numericas),
df.to_pickle('path/file_name.pkl') save dataframe df_sin_outliers = [Link]([Link][list
values that meet the condition other_label = 'etiqueta'cambia la etiqueta 'other' columns = df_numericas.columns) create a dataframe from the data
as a pickle file (values)]) create a new dataframe without outliers
[Link] create null value (NaN) value = 'columna'ordena los resultados por la columna transformed; we put these columns into the original dataframe
[Link]() returns True or False depending on whether the
specified
values exist or are null ('' does not count as null) ExcelWriter Replace Outliers from [Link] import KNNImputer
with [Link]("path/[Link]") as writer: [Link](['column1', 'column2']) combines two
.notnull() returns True or False depending on whether the series for k, v in dicc_indices.items(): imputerKNN = KNNImputer(n_neighbors=5) creates the instance
df.to_Excel(writer, sheet_name = 'name') columns and return the frequencies of the subcategories
values exist or are null ("" does not count as null) media = df[k].mean() [Link](df_numericas)
save a dataframe in an Excel sheet for i in v:
[Link](['column']) returns information about the df_knn_imp = [Link]([Link](df_numericas),
[Link][i,k] = media replace
frequency of null data Create a dataframe of the data
outliers by the mean
transformed; we put these columns into the original dataframe
Pandas Subsets: loc and iloc Data filtering Create columns Change values
Pandas filtering methods Creation of ratios Replace values based on indices and conditions:
Union of data [Link]['row_label', 'column_label']
returns the content of a field in a column df_filtered = df['column?ratio'] = [Link](lambda df: df['column1'] filtered_indices = [Link][df['column'] == 'value']
of a row df[df['column_name'].isin(iterable)] extracts the / df["columna2"], axis = 1)
.concat()join dataframes with common columns rows whose values in the column named are in
for index in filtered_indices:
df_union = [Link]([df1, df2, df3], axis=b, join = [Link]['row_label', :] returns the values the iterable (a list, series, dataframe or Creation of percentages df['column_name'].iloc[index] = 'new_value'
'inner/outer', ignore_index = True/False of all the columns of a row dictionary) def percentage(column1, column2): Replace values based on NumPy methods:
parameters:
[Link][:,'etiqueta_columna'] returns the return (column1 * 100) / column2
axis = 0 a column-wise - the dataframes go one on top of the other [Link](to_replace = value, value = new_value,
values of all the rows of a column filtered_df = df[df["column_name"].[Link]
another; the columns must be of compatible formats inplace = True) replaces a certain value with another that
(pattern, regex = True, na = False)]extracts the rows df["column_%"] = [Link](lambda df:
axis = 1 means by rows – the dataframes go side by side; [Link][row_index, column_index] returns the we specify
whose values in the column named contain the percentage(df["column1"], data["column2"]), axis = 1)
the data must be related in order to make sense
content of a field in a column of a row regex pattern df["new_column"] = [Link](df["column_name"] > n, df["column_name"].replace(to_replace = value,
join = 'inner' only elements that appear in all are retained
dataframes create a new value = new_value, inplace = True) replaces certain
[Link][row_index, :] returns the values of filtered_df = df[df['column_name'].[Link]
join = 'outer' keeps all the data from all the dataframes column based on a condition value in one column by another that we specify
all the columns of a row
ignore_index = True/False by default is False; if it is True, it does not use extract
df[["column1", "column2"]] = df[["column1",
the indices for the union (for example for union along axis 0) [Link][:,column_index] returns the content the rows whose values in the column named df["new_column"] = [Link](list_of_conditions, columna2]].replace(r"string", "string", regex=True)
from a field in a column of a row they contain the substring, being case insensitive list_of_options) create a new column with the values
.merge()merge the columns of one dataframe with another replace a pattern/string with another in multiple
based on multiple conditions columns
df_nuevo = [Link](df2, on = 'column') inner merge [Link][[list_row_labels], filtered_df = df[df["column_name"].[Link] df["new_column"] = [Link](x = df["column_name"],
df_new = [Link](left = df1, right = df2, how='left', left_on = [list_labels_columns]]returns the extract df['column_name'] = df['column_name'] + x
bins = [n, m, l..], labels = ['a', 'b', 'c']) separate the
'columna_df1', right_on = 'columna_df2') left merge content of several rows / several columns the rows whose values in the column named replace the values of the column with the value + x
elements of a dataframe in different intervals (n-m, (or another value that we indicate)
parameters: they contain the substring, being case insensitive
[Link][[row_index_list], m-l, etc), creating a new column that indicates in which
how = 'left' | 'right' | 'outer' | 'inner' | 'cross'
[list_column_indices]] returns the content interval falls the value; with labels you can assign a
on = column|[column1, column2, etc] if the columns are named
the same in both dataframes multiple rows / multiple columns
df[[Link](df["nombre_columna"])]: returns the string to each interval datetime
rows that do not have null values in the column
left_on = column_df1 | right_on = column_df2 to specify You can use the indices/ranges of the lists specified Create columns import datetime
where to do the merge within loc/iloc df["nueva_columna"] = (df["etiqueta_columna"] + x)crea [Link]() and [Link]() return the date
suffixes = ['left', 'right'] by default nothing, the suffix that a new column based on another actual
will appear in duplicate columns [Link][[Link] > x] select data based Change columns df = [Link](new_column= df['label_column'] + x) timedelta(n) represents a duration, the difference
in a condition using comparative operators create a new one based on another
.join()merge dataframes by the indices between two instances; n is a number of days but it
column_list = [Link].to_list() creates a list df = [Link](new_column= [list_values]) create a can specify days, seconds, or microseconds
df_nuevo = [Link](df2, on = 'column', how = 'left') inner merge [Link][([Link] > x) & ([Link] == y)] of the column names of the dataframe new column of a list of values *must be from
parametros: select data that must meet both yesterday = [Link]() - timedelta(1)
the same length as the number of rows of the dataframe*
how = 'left' | 'right' | 'outer' | 'inner'por defecto left conditions (and) yesterday = [Link](yesterday, '%Y-%m-%d')
[Link](new_column_index, "column_name",
on = column, the column or index by which we want to do the df.set_index(["column_name"], inplace = True)
[Link][([Link] > x) | ([Link] == y)] (values) create a new column at the indicated index df["fecha"] = ayercrea una columna con la fecha de
union; they must have the same name in both dataframes set the index using one or more columns;
select data that must comply allow_duplicates = True parameter when we want yesterday
lsuffix = 'string' | rsuffix = 'string' by default nothing, the can replace or extend an existing index
one of the two conditions (or) allow duplicate columns (default is False)
suffix that will appear in duplicate columns inplace = True changes overwrite the df
strftime() allows us to create more readable formats of
[Link][list([Link] > x), :] iloc does not accept when a column is changed to an index, it is no longer Apply datetime type data
Group By a boolean series; it needs to be converted into a list column *
[Link](variable_fecha, '%Y-%m-%d') formats
df.reset_index(inplace = True) remove a column apply() takes a function as an argument and applies it to the
variable_df.head(n) returns the first n the date to the indicated format
df_groupby = [Link]("category_column") creates an object length of an axis of the DataFrame
rows of the df, or 5 by default as an index to become a column again; create a
DataFrameGroupBy; groups the values according to the categories of the
dataframe of a series df['new_column'] = df['col_1'].apply(function) strftime() syntax
values of the indicated column (or multiple columns in a list) abbreviated day of the week (Sun)
create a new column with the values from another column
df_groupby.ngroups returns the number of groups Rename columns transformed according to the indicated function %Adía de la semana (Sunday)
df_groupby.groups returns a dictionary where the keys are the Data filtered [Link](columns = {"nombre_columna": df['new_column'] = df['col_1'].apply(lambda x: %day of the week from 0 (Sunday) to 6 (Saturday)
categories and the values are lists of the indices of each "new_name"}, inplace = True changes the names [Link]() if x > 1 abbreviated PMS (Sep)
element in the category Filtered by a column with operators of from one or more columns
create a new column with the values of another column September
df_grupo1 = df_groupby.get_group("grupo1") returns a dataframe comparison example of dict comprehension to create a dictionary transformed according to the indicated lambda %mmes with a zero (09)
with the results of a group (the category indicated as group1) about the existing columns of a dataframe:
filtered_df = df[df['column_name'] == value] df['new_column'] = [Link](lambda name: %-mmes as float
Calculations with groupby:
extract the rows where the value of the column diccionario = {col : [Link]() for col in function(name['column1'], name['column2']), axis =
Day of the month with a zero (08)
df_nuevo = [Link]("category_column").mean() returns a equal to the given value [Link] b) create a new column using a function that takes two
dataframe with the average of all columns of numerical values, parameters (column 1 and column 2) %-day of the month as float (8)
by category [Link](columns=dictionary, inplace=True) Year without century (99)
Filtering by multiple columns with operators change the names of the columns according to the [Link](function, na_action=None, **kwargs) accepts and
df_new = [Link]("category_column") ["column1"].mean() logical dictionary returns a scalar to each element of a dataframe; it %Year with century (1999)
returns a dataframe with the average of the specified column it has to apply to the entire DataFrame Time zone (UTC)
filtered_df = df[(df['column1'] == value) & Delete columns
count() number of observations (df["column2"] == value) & (df["column3"] > n df['column'] = df['column'].map(map, na_action = %pAM or PM
no nulls extracts the rows where the values of [Link](columns = ["column1", "column2"], axis = replace column values according to the map, %date and time
the columns meet the conditions in b, inplace=True) delete one or more columns or rows it can be a dictionary or a series; it can only be
summary of the %date
parenthesis according to what we specified to apply to a particular column.
main statistics
%Xhora (07:06:05)
sum() sum of all values Reorder columns apply() with datetime
filtered_df = df[(df['column1'] == value) | Hour with zero (07)
mean() average of the values df['date_column'] = df['date_column']
(df['columna1'] == value) extracts the rows where df = [Link](columns = reordered_list) %-Hour as float (7)
order of the columns of the dataframe according to the order of .apply(pd.to_datetime) changes a column of data type
df_new = [Link]('category_column', dropna=False) the values of the columns meet a 0 minutes (06)
date in datetime format
["columna_valores"].agg([nombre_columna= ‘estadistico1’, condition or another the reordered list
def extract_year(x): %-Mminute as float (6)
column_name2 = 'statistic2']) add columns with the calculations
of the specified statistics df_filtered = ~(df[df["column1"] == value]) return [Link]("%Y") %Seconds with zero (05)
dropna = False to take NaNs into account in the calculations (for extract the rows where the values of the df['year_column'] = (df['date_column'].apply %-Seconds as float (5)
the defect is True columns DO not meet the condition (extract_year) create a new column for the year using only a
datetime library method; (%B) for months
Matplotlib and Seaborn Seaborn graphs Multigraphs Customization Personalization
Line plot fig, ax = [Link](number_of_rows, number_of_columns) Titles Colors
Matplotlib fig = [Link](x = 'column1', y = 'column2', data = create a figure with multiple graphs; fig is the
df, ci = None) creates a line graph where the axes are: figure and ax is an array with subplots as elements [Link](label = 'title') assign a title to color = "color"establece el color de la grafica
the graph facecolor = "color"establece el color del relleno
Graphs columna1–x, columna2–y
ci = None so that it does not show the confidence interval of
it is established how each graph is with the indices:
edgecolor = "color"establece el color de los bordes
ax[index].chart_type(chart details)
the data Colors in Scatter Plots:
import [Link] as plt hue = optional column; shows lines in different
ax[index].set_title('title') Axes
ax[index].set_xlabel('xlabel') c = df['column'].map(dictionary)
colors by categories according to a variable [Link]("x_axis_label") assign name to the x axis
[Link]["[Link]"] = (10,8) ax[index].set_ylabel('ylabel') diccionario = {"valor1": "color1", "valor1": "color1"}
Scatter plot [Link]("y_axis_label") assign name to the y axis
[Link](figsize = (n,m)) starts a plot list of colors
fig = [Link](x = 'column1', y = 'column2', data ax[index].set_xlim(min, max [Link]([n,m] sets the range of the x-axis; where n is
drawing the frame of the figure; n is the width and
m is the height, in inches Create a scatter plot with df, hue = 'column'. ax[index].set_ylim(min, max) the minimum and m is the maximum Seaborn Palettes:
[Link]() displays the figure Swarm plot ax[indice].set_xticklabels(labels = df['column'], [Link]([n,m]) sets the range of the y-axis; where n ["Accent","Accent_r","Blues","Blues_r","BrBG","BrBG_r"]
fig = [Link](x = 'column1', y = 'column2', data = rotation = n) to change the names and/or the rotation m is the maximum and m is the minimum 'BuGn', 'BuGn_r', 'BuPu', 'BuPu_r', 'CMRmap', 'CMRmap_r',
df, hue = 'column') creates a scatter plot where of the labels of the values on the axes 'Dark2', 'Dark2_r', 'GnBu', 'GnBu_r', 'Greens',
Basic graphs 'Greens_r', 'Greys', 'Greys_r', 'OrRd', 'OrRd_r',
the markers do not overlap Create subplots in a for loop [Link](xlabel = 'x_axis_label', ylabel =
Bar plot Count plot
{"Oranges":"Oranges","Oranges_r":"Oranges_r","PRGn":"PRGn","PRGn_r":"PRGn_r","Paired":"Paired"}
fig, axes = [Link](number_of_rows, number_of_columns, assign name to the axes
[Link](df['columna1'], df['columna2']) creates a ["Paired_r","Pastel1","Pastel1_r","Pastel2"]
fig = [Link](x = 'column1', data = df, hue = figsize = (n, m)) fig.set_title('title') assign a title to the graph {"Pastel2_r":"Pastel2 reversed","PiYG":"Pigment Yellow Green","PiYG_r":"Pigment Yellow Green reversed","PuBu":"Purple Blue","PuBuGn":"Purple Blue Green"}
bar chart where the axes are: column1– 'column') create a bar chart with the count of a
x, column2–y axes = [Link]() 'PuBuGn_r', 'PuBu_r', 'PuOr', 'PuOr_r', 'PuRd', 'PuRd_r',
categorical variable; only one can be specified
Horizontal bar plot for col in [Link]: fig.set_xlabel(xlabel = "etiqueta_eje_x", fontsize = n) 'Purples', 'Purples_r', 'RdBu', 'RdBu_r', 'RdGy',
variable on the x or y axis, plus one optional variable with
[Link](df['columna1'], df['columna2']) creates a fig = [Link](x=col, data=df, ax=axes[i] 'RdGy_r', 'RdPu', 'RdPu_r', 'RdYlBu', 'RdYlBu_r',
hue fig.set_ylabel(ylabel = "etiqueta_eje_y", fontsize = n)
horizontal bar chart where the axes 'RdYlGn', 'RdYlGn_r', 'Reds', 'Reds_r', 'Set1', 'Set1_r',
Histogram Create subplots in a for loop
["Set2","Set2_r","Set3","Set3_r","Spectral"]
son: columna1–x, columna2–y fig = [Link](x = 'columna1', data = df, hue = fig, axes = [Link](number_of_rows, number_of_columns, [Link](xticks = [1, 2, 3]) 'Spectral_r', 'Wistia', 'Wistia_r', 'YlGn', 'YlGnBu',
Stacked bar plot creates a histogram that figsize = (n, m))
[Link](yticks = [1, 2, 3, 4, 5]) 'YlGnBu_r', 'YlGn_r', 'YlOrBr', 'YlOrBr_r', 'YlOrRd',
[Link](x, y, label = 'label') shows the frequencies of a data distribution; where 'YlOrRd_r', 'afmhot', 'afmhot_r', 'autumn', 'autumn_r',
[Link](x2, y2, bottom = y, label = 'label2') [Link](xticklabels = ['0%','20%', '40%', '60%', '80%',
x is the variable of interest and n is the number of bars 'binary', 'binary_r', 'bone', 'bone_r', 'brg', 'brg_r',
100%
create a stacked bar chart for kde = True shows a curve of the distribution Uses of types of graphs [Link](yticklabels = ['cat1', 'cat2', 'cat3'])
{"bwr":"bwr","bwr_r":"bwr_r","cividis":"cividis","cividis_r":"cividis_r","cool":"cool","cool_r":"cool_r"}
visualize two variables together; and indicate the bar ["coolwarm","coolwarm_r","copper","copper_r","crest"]
Box Plot
for reference Categorical data 'crest_r', 'cubehelix', 'cubehelix_r', 'flag', 'flag_r',
fig = [Link](x = 'column1', data = df, hue =
Scatter plot column') creates a box plot; x is the variable of fig.set_xticklabels(labels = [0, 500, 1000, 1500], 'flare', 'flare_r', 'gist_earth', 'gist_earth_r',
Bars ["gist_gray","gist_gray_r","gist_heat","gist_heat_r"]
[Link](df['column1'], df['column2']) creates interest; by default it is displayed with horizontal orientation size=n)
a scatter plot where the axes are: -use the y-axis for vertical orientation shows the relationship between a numeric variable and ["gist_ncar","gist_ncar_r","gist_rainbow"]
fig.set_yticklabels(labels = fig.get_yticklabels(),
columna1–x, columna2–y categorical 'gist_rainbow_r', 'gist_stern', 'gist_stern_r',
Catplot size=n)
barplot if you have a numerical variable 'gist_yarg', 'gist_yarg_r', 'gnuplot', 'gnuplot2',
fig = [Link](x = 'column1', y = 'column2', data = 'gnuplot2_r', 'gnuplot_r', 'gray', 'gray_r', 'hot',
Statistical graphs df, hue = 'column', kind = 'type') creates a graph that - countplot to count records/rows by category
To put labels on top of the bars ["hot_r","hsv","hsv_r","icefire","icefire_r"]
Histogram show the relationship between a categorical variable and one Pie chart/slices {"inferno":"inferno","inferno_r":"inferno_r","jet":"jet","jet_r":"jet_r","magma":"magma"}
for index, value in enumerate(df['col']):
[Link](x = df['columna1'], bins = n) creates a numerical variable ["magma_r","mako","mako_r","nipy_spectral"]
determination of frequencies [Link](value+1, index, value,
histogram that shows the frequencies of a kind = 'box' | 'bar' | 'violín' | 'boxen' | 'point' por ["nipy_spectral_r","ocean","ocean_r","pink","pink_r"]
data distribution; where x is the variable of the default is strip plot Numerical data horizontalalignment='left', fontsize= 16
'plasma', 'plasma_r', 'prism', 'prism_r', 'rainbow',
interest and n is the number of bars Pairplot Lines ["rainbow_r","rocket","rocket_r","seismic","seismic_r"]
Box Plot fig = [Link](data = df, hue = 'column', kind = trends/evolution of one or more numerical variables order = df.sort_values('columnay', ascending=False) 'spring', 'spring_r', 'summer', 'summer_r', 'tab10',
[Link](x = df['columna1']) creates a diagram of create the histograms and scatter plots of (normally over a period of time) columnax ["tab10_r","tab20","tab20_r","tab20b","tab20b_r"]
boxes to study the characteristics of a all the numerical variables available in the [Link](font_scale=2) 'tab20c', 'tab20c_r', 'terrain', 'terrain_r', 'turbo',
numeric variable; x is the variable of interest dataset we are working with; hue is optional Histogram ["turbo_r","twilight","twilight_r","twilight_shifted"]
font size
the minimum is the same as Q1 - 1.5 * IQR kind = 'scatter' | 'kde' | 'hist' | 'reg' | 'point' por distribution of a numerical variable ["twilight_shifted_r","viridis","viridis_r","vlag"]
general
The maximum is the same as Q3 + 1.5 * IQR the default is scatter 'vlag_r', 'winter', 'winter_r
Boxplot
Heatmap representation of the most commonly used measures of position: palette='light:nombre_paleta'|'dark:nombre_paleta'
[Link]([Link](), cmap = 'color_palette', annot = Legends
mediana, IQR, outliers
True, vmin = -1, vmax = 1 creates a heatmap with a scale [Link](labels = ['label1', 'label2', etc]) shows Markers
in colors that reflect the values of correlation Scatterplot the legend when we show the figure marker = 'tipo'establece el tipo de marcador; se usa con
annot = True so that the values appear shows the relationship between two numerical variables [Link](bbox_to_anchor = (1, 1) places the legend in [Link] and [Link]
vmin/vmax set the color scale Regplot relationship with the axes
scatter plot with a regression line . P More (filler)
Regplot Pixel * Star
fig = [Link](x = 'column1', y = 'column2', data = Swarmplot Remove borders the Circle "h" Hexagon 1
df, scatter_kws = {'color':'blue'}, line_kws = {'color'; type of scatter plot to represent [Link]["top", "right"].set_visible(False) v Triangle down "H" Hexagon 2
create a scatterplot plus the regression line; we categorical variables; prevents them from overlapping Up triangle + More
allows to find the best line function that allows markers Left triangle < "x" x
predict the value of a variable knowing the values of Line of three standard deviations: Right triangle "X" x (filling)
Pie Chart another variable Violin plot [Link](x=value, c='color', label='value') 8 Octagon "D" Diamond
[Link](x, labels = categories, radius = n) creates a to visualize the distribution of the data and its [Link](x=value, c='color', label='value')
Jointplot "s" Square d Fine diamond
pie chart where x is the variable of probability density
[Link](x = 'column1', y = 'column2', data = df, p Pentagon
interest (must be grouped by categories); n is Grid
the size color = 'blue', kind = 'type') create a scatterplot or Pairplot
regplot with histograms attached on the sides for each to represent multiple relationships between two [Link]() creates a grid in the background of the figure;
variable variables take the parameters:
Violin Plot
[Link](x, showmedians = True, showmeans = color = "color"
Heatmap
True) create a violin plot where x is the Export figures linestyle = "solid"|"dashed"|"dashdot"|"dotted"
[Link]('figure_name.extension') evaluate the correlation between the variables in a
variable of interest and shows the median and the mean linewidth = sets the width of the line
correlation matrix
NumPy (Numerical Python) Indices, Subsets, Array Methods Statistical and mathematical operations Set functions Statistics
Array indices Statistical and mathematical operations [Link](array) returns an array with unique values Frequency tables
Create arrays from the sorted array
array[i] returns the index i; the indices of the The axis parameter in two-dimensional arrays: [Link](array, return_index=True) returns an array with Absolute frequencies
Create arrays of lists One-dimensional arrays work the same as lists axis = 0columnas the unique values of the array sorted and an array with the the number of times a number is repeated in a
array = [Link](lista, dtype= tipo) creates an array array[i, j] or array[i][j] returns the element of the axis = 1rows position of the first instance of each value dataset
unidimensional de una lista column j of row i if we specify the axis, the operation returns the [Link](array, return_inverse=True) returns an array with df = [Link]('column').count().reset_index()
array = [Link]([list1, list2]) creates an array array[:, :n] select all rows and columns result for each row or column. the unique values of the array sorted and an array with the
Relative frequencies
bidimensional of two lists up to n-1 For example: positions of each element of each value
the times a number or category is repeated in a
array = [Link]([listoflists1, listoflists2]) create array[h, i, j] or array[h][i][j] returns the element of [Link](array, axis = 0) returns an array with the [Link](array, return_counts=True) returns an array with
dataset regarding the total, in percentages
a two-dimensional array of two lists the column j of the row i of the array h sum of each row the unique values of the array sorted and an array with it
number of times each value appears df_group_sin_str = df_group.drop('columna_str',
array[h][i][j] = change the value of the element in [Link](array, axis = b) returns an array with the axis=1)
Create other types of arrays this position at value n
The axis parameter in multidimensional arrays:
axis = 0 dimension unique ordered values of rows or columns frecuencia_relativa = df_group_sin_str / [Link][0] *
array = [Link](start_value, end_value, steps) creates
axis = 1columnas 100
an array using the format [start:stop:step]
array = [Link](z, y, x) creates an array of all ones Subsets axis = 2rows Functions for one-dimensional arrays columns = df_group_sin_strings.columns
specified form array > returns the shape of the array with True or False if we specify the axis, the operation returns the df_group[columnas] = frecuencia_relativa
np.intersect1d(array1, array2) returns an array with the
array2 = np.ones_like(array1) creates an array of all ones depending on whether the element meets the condition or not result for each dimension, row or column.
unique values of the common elements of two arrays
in the form based on another array
array[array > n] returns a subset: all values
For example:
np.intersect1d(array1, array2, return_indices=True) Contingency tables
array = [Link](z,y,x) creates an array of all zeros of the [Link](array_3D, axis = 0) returns an array of one frequency table that counts all the
that meet the condition in a list within a returns an array with the unique values of the elements
specified form matrix with the sum of all matrices possible combinations of each pair of values
array common elements of two arrays and arrays with the indices of each
array2 = np.zeros_like(array1) creates an array of everything [Link](array_3D, axis = 1) returns an array where the columns we are trying to compare
array[(array > n) & (array < m)] returns a subset: value, by array
zeros of the form based on another array the rows contain the sums of the columns of
np.union1d(array1, array2) returns a sorted array with df_crosstab = [Link](df['column1'],
array = [Link]((z,y,x), type) creates an empty array with all the values that meet the conditions in a each matrix
the elements resulting from joining two arrays (values df['column2'], normalize = True, margins = True
default data of type float list inside an array; you can use | for "or"
unique) normalize shows the values in percentages (by one)
array2 = np.empty_like(array1) creates an empty array with the Operations with axis parameter: np.in1d(array1, array2) returns an array with True or False
form based on another array [Link](array_3D) returns the sum of all the margins shows the totals and subtotals
Array methods for each element of array1 depending on whether the same value appears
array = [Link](z, y, x, k = n) creates an array with ones in elements of the matrices
new_array = [Link]() creates a copy of the array in array2 Pearson correlation coefficient
diagonal starting at position k [Link](array) returns the mean of the entire array
np.setdiff1d(array1, array2) returns a sorted array with it allows us to know the intensity and direction of the
array = [Link](x) creates an identity matrix with [Link](bidimensional_array) changes the rows of the [Link](array) returns the standard deviation of
the unique values that are in array1 but not in array2 relationship between the two variables
zeros in rows and ones on the diagonal, in a square form array to columns and columns to rows everything
np.setxor1d(array1, array2) returns a sorted array with
[Link](multidimensional_array) changes the number [Link](array) returns the variance of values of - coefficient > 0: positive correlation
the unique values that are NOT common between the two arrays
from columns to the number of arrays and vice versa; the number all - coefficient < 0: negative correlation
the rows do not change [Link](array) returns the minimum value of the array
NumPy Random [Link](multidimensional_array, (z, y, x)) makes the
[Link](array) returns the maximum value of the array Statistics coefficient = 1 or -1: total correlation
- coefficient = 0: there is no linear relationship
[Link](array) returns the sum of the elements of the
transposition according to what we specify using the df['column1'].corr(df['column2'] calculates the
[Link](x) sets the random seed of array Measures of dispersion
positions of the tuple (0,1,2) in the original form correlation between two variables
random number generator, for the functions [Link](array) returns an array with the sum
array = [Link](n).reshape((y,x)) creates an array cumulative of the elements throughout the array
Deviation from the mean
random that will always take the same values afterwards correlation_matrix = [Link]() creates a matrix
using reshape to define the shape the absolute difference between each value of the
random [Link](array) returns an array with the showing the correlations between all the variables
array = [Link](array, (z,y,x)) creates an array with data and its arithmetic mean
cumulative multiplication of the elements along [Link]([Link]()[['column1', 'column2']], cmap =
the values of another array using reshape to define from the array diferencias = df['columna'] - df['columna'].mean()
Create arrays with random values the form
'color_palette', annot = True, vmin = -1, vmax = 1
desviación_media = [Link](diferencias)
array = [Link](start, end, matrix_shape) create a heatmap graph of the correlation matrix
create an array of random numbers between two values; array = [Link](array, position, position) Operations without axis parameter: Variance
swap two axes of a matrix using the [Link](array) returns an array with the square root
forma_matriz: (z,y,x) measure of dispersion; the variability with respect to the mean Skewness
z: number of arrays positions (z=0,y=1,x=2) of the original shape non-negative square of each element of the array measure of the skewness of the distribution of the
[Link](array) returns an array with the exponential df['column'].var()
y: number of rows values of a variable around its mean value
x: number of columns of each element of the array Standard deviation or typical deviation positive skew value: skewed to the right
Other operations [Link](array1, array2) returns an array with the the square root of the variance; the greater it is, the greater
array = [Link](start, end) returns a negative skew value: skewed to the left
[Link](array) returns an array with the values of remainder of the division between two arrays it will be the dispersion or variability in our data
random number in the range - skewness value equal to 0: symmetric values
each row sorted in ascending order by default [Link](array1, n) returns an array with the remainder of
array = [Link](z, y, x) creates an array of floats df['column'].std() [Link](df['column'], kde = True) creates a
[Link](array, axis=0) returns an array with the the division between the array and the value of n
random ones in the shape that we specify; by default histogram that shows the distribution of values
values of each column sorted in ascending order [Link](array) returns an array with the cosine of Robustness
generate random numbers between 0-1 import [Link] as skew
each element of the array the more data there is, the more robust
array = [Link].random_sample((z,y,x)) creates an array of [Link](-array) returns an array with the values of skew(df['column'] shows the value of the bias of a
random floats with the shape that we specify; [Link](array) returns an array with the sine of each 1/n where n is the number of records variable
each row sorted in descending order
element of the array
defect generates random numbers between 0-0.9999999... [Link](array, decimals = x) returns an array with the Coefficient of variation
array = [Link].z,y,x=None) returns a number [Link](array) returns an array with the tangent of the quotient between the standard deviation and the mean; how much
Confidence intervals
array values rounded to x decimals each element of the array describe the variability between the obtained measurement in
random between 0 and 0.999999999999... the greater the sea, the greater the dispersion in our data
[Link]([Link](z,y,x), n) create array with floats [Link](array, decimals = x) returns an array with the a study and the actual measurement of the population (the value
of n decimals array values rounded to x decimals Comparison operations on arrays df['column'].std() / df['column'].mean() real
[Link](n,m, size = (z,y,x)) generates samples [Link](array > x) returns the indices of the values bidimensional import [Link] as st
Percentiles
random from a uniform distribution in the interval that meet the condition, by row and column [Link](array > n) returns True or False depending on whether
[Link](alpha = n, df = len(df['column']-1, loc
divide datos ordenados de menor a mayor en cien partes;
between n and m any value of the array meets the condition = [Link](df['columna']), scale =
shows the proportion of data below its value
[Link](n, m, size = (z, y, x)) generates samples [Link](array > n, axis = b) returns an array with [Link](df['column'])
with a binomial distribution; n is the total number of
Operations with arrays True or False for each column or row depending on whether any percentil_n = [Link](df['column'], n) gets the value
in the n percentile returns the range of values for which there is an n% of
tests; m is the probability of success [Link](array1, array2) sum of two arrays value of the row or column meets the condition probability that a real value falls within that range
[Link](loc = n, scale = m, size = (z,y,x)) [Link](array1, array2) subtracts array2 from array1 [Link](array > n) returns True or False depending on whether
generate random numbers from a normal distribution all values of the array meet the condition Interquartile ranges alpha: porcentaje de confianza ([Link]. 90%, 95%, o 99%)
[Link](array1, array2) multiplies two arrays measure of dispersion: difference between the 75th and 25th quartiles the data
(bell curve); loc is the mean; scale is the [Link](array > n, axis = b) returns an array with
[Link](array1, array2) divides array1 by it True or False for each column or row depending on whether all q3, q1 = [Link](df['column'], [75, 25]) get the the average
standard deviation
array2 the values of the row or column meet the third and first quartiles
[Link](array) returns an array with the scale: the standard deviation
same values randomly mixed array + n, n * array, etc. - algebraic operators condition rango_intercuartílico = q3 - q1
EDAy ETL ETL: Extract, Transform, Load Machine Learning: Preparation Statistical tests Normalization
Extraction Independence between predictive variables Manual method
EDA: Exploratory Data Analysis Null Hypothesis and Type I and Type II Errors
obtain raw data and store it we take the value we want to normalize and
the predictor variables must be independent in order to
SQL or NoSQL database tables
Exploratory Data Analysis refers to
Plain text files
Null hypothesis (H0) to create a linear regression model we subtract the mean of the column, and divide the
process of conducting a series of investigations In general, it is the opposite statement to the one we want to prove. result by the maximum subtracted by the minimum of the
initials about the data we have to be able to Emails Numerical variables: Correlations column
discover patterns, detect anomalies, test Website information Alternative hypothesis (H1) pairplot
Spreadsheets df["col_norm"] = (df["col_VR"] -
hypothesis and verify assumptions with the help of In general, the statement we want to verify [Link](df)
Files obtained from APIs df['col_VR'].mean()) / (df['col_VR'].max() -
statistics and graphical representations. df["col_VR"].min()
p-value covariance
Transformation measure of the probability that a null hypothesis is true df_numerics.cov() Logarithmic method
1. Understand the variables process the data, unify it, clean it, value between 0 and 1 Pearson correlation (linear relationship) it cannot be done if any value is 0
validate them, filter them, etc. If *p-value* < 0.05❌We reject the null hypothesis. df_numerics.corr()
What variables do we have? Format dates df["col_norm"] = df["col_VR"].apply(lambda x:
if *p-value* > 0.05 we accept the null hypothesis. Spearman correlation (non-linear relationship) [Link](x) if x > 0 else 0
.head(), .tail(), .describe(), .info(), .shape Reorder rows or columns
df_numeric.corr(method = 'spearman')
what types of data Join or separate data Type I Error: Square root method
.dtypes(), .info() - Combine the data sources reject the null hypothesis when it is true Kendall correlation (numerical data but
if we have nulls or duplicates Clean and standardize the data categorical and ordinal import math
.isnull().sum() Verify and validate the data Type II Error: df_numerical.corr(method = 'kendall') df['col_norm'] = df['col_VR'].apply(lambda x:
.duplicated().sum() Remove duplicates or erroneous data accept the null hypothesis when it is false [Link](x)
what unique values do we have Filtering, performing calculations or groupings
.unique(), .value_counts() Categorical variables: Chi-square [Link]() method
Load
Statistical tests V-Cramer: varies between 0 and 1
apply a logarithmic transformation to the
side table library closer to 1 more dependent
load the data into its destination format, the type positive and exponential values for values
[Link]() returns the value_counts of variables
which will depend on the nature, size and
Normality -result < 0.7 to perform ML negatives of our column
categorical, but the percentage, cumulative count and
cumulative percentage
data complexity. The most common systems the response variable must have a normal distribution in order to import researchpy as rp from scipy import stats
they tend to be: create a linear regression model
[Link]() null account table and the crosstab, test_results, expected = [Link] df['col_norm'], adjusted lambda =
CSV files [Link](df["col_VR"])
percentage of the total
JSON files Visually: df["col1"], df["col2"], test= "chi-square"
expected_freqs= True, prop= "cell")
Databases histogram or distribution MinMaxScaler Method
2. Clean the dataset Data Warehouses theoretical quantile plot (Q-Q) test_results returns the test results in a
from [Link] import MinMaxScaler
Data Lakes the more aligned the points are around the line, the more normal dataframe
model = MinMaxScaler(feature_range=(0,1),
remove duplicates (rows or columns) they will be our data Homoscedasticity (homogeneity of variances) copy=True)
change column names import [Link] as sm [Link](df['col_VR'])
change data type of columns the predictor variables must have homogeneity of
[Link](data, line='45') datos_normalizados = [Link](df["col_VR"])
sort columns variances in comparison with the response variable
- split column into two with [Link]() APIs Analytical methods:
Visually: df_datos_norm = [Link](normalized_data,
- create intervals with [Link]() Asymmetry columns = ['col_norm'])
violin plot
create percentages or ratios import requests library to make HTTP requests df['col_norm'] = df_datos_norm
- positive skewed distributions: mean > median and mode boxplot
decide how to treat outliers: keep them, to a URL, for web scraping
- negative skewed distributions: mean < median and mode - regplot (numerical columns vs response variable)
eliminate them, or replace them with the average,
median or mode; or apply an imputation
url = 'enlace'el enlace de la que queremos extraer
data from [Link] import skew
ANOVA
Analytical methods:
header = {}opcional; contiene informacion sobre las skew(normal_data) method of scipy that calculates the skewness import [Link] as sm
decide how to handle nulls: requests made (file types, credentials) Levene's test (more robust against lack of
df['column'].skew() pandas method that calculates the skewness normality) or Bartlett from [Link] import ols
remove rows or columns with nulls [Link]() response = [Link](url=url, header = header)
impute missing values: we ask the API to give us the data Kurtosis from scipy import stats lm = ols('col_VR ~ col_VP1 + col_VP2 + col_VP3',
replace them with the mean, median, or mode variables = {'parametro1':'valor1', leptokurtosis: kurtosis value greater than 0 (high peak) from [Link] import levene data=df).fit()
using .fillna() or .replace() 'parametro2':'valor2'} Categorical variables: returns a dataframe of the results:
-mesocurtosis: kurtosis value equal to 0 (medium peak)
imputer with machine learning methods response = [Link](url=url, params=variables) df (degrees of freedom): for categorical variables
we ask the API to give us the data with the -platykurtosis: kurtosis value less than 0 (flat) we need to create a dataframe for each unique value of
using the sklearn library: Simple-Imputer, it will be the number of unique values minus 1; for
parameters according to the parameters dictionary that you from [Link] import kurtosistest the categorical columns
Iterative Imputer, or KNN Imputer numeric variables will always be 1
we passed kurtosistest(data) returns a p-value df_valor1 = df[df['col1'] == 'value1']['col_VR']
sum_sq: medida de variación/desviación de la media
response.status_code returns the status of the request p-value of the test > 0.05: normal data df_valor2 = df[df['col1'] == 'valor2']['col_VR']
[Link] returns the reason for the status code mean_sq: es el resultado de dividir la suma de
3. Analyze relationships between variables [Link] returns the data in string format p-value of the test < 0.05: non-normal data levene_test = [Link](df_valor1, df_valor2,
center='median'
squares between the number of degrees of freedom.
[Link]() returns the data in json format Shapiro-Wilk test A test that is used to assess the ability
Analyze relationships between the variables bartlett_test = [Link](df_valor1, df_valor2,
df = pd.json_normalize([Link]) returns the for samples < 5000 explanatory that the predictor variable has on
to find patterns, relationships, or anomalies center='median'
data in a dataframe the variation of the response variable
Relationships between two numerical variables: null hypothesis: normal distribution Numeric variables:
scatterplot PR(>F): if the p-value < 0.05 is a variable
HTTP response codes from scipy import stats a dataframe of the numeric columns must be created significant; that can affect the VR
- regplot - scatterplot with regression line [Link](df['data']) without the response variable
correlation matrix and heatmap p-value of the test > 0.05: normal data [Link]() returns a summary of the
1XX informs of a 4XX error during request for col in df_numericas.columns:
- joinplot - allows pairing two graphs - one resultados:
correct answer 401 bad request - p-value of the test < 0.05: non-normal data statistic, p_val = levene(df[col], df['col_VR']
histogram with scatter or regression plot for example coef: represents the average changes in the VR for
2XX success code 402 without authorization
Relationships between two categorical variables: Kolmogorov-Smirnov test center='median' a unit of change in the VP while being maintained
200 OK 403 forbidden
countplot for samples > 5000 resultados[col] = p_val constants the rest of the VP; the signs us
201 created 404 not found
Relationships between numerical and categorical variables: indicate whether this relationship is positive or negative
202 accepted 5XX server error - null hypothesis: normal distribution returns the p-values in a dictionary
swarmplot the smaller the standard error, the more
204 without content 501 internal server error -p-value of the test > 0.05: equal variances,
violin plot from scipy import kstest the estimation will need to be precise
3XX redirect 503 service unavailable homoscedasticity
pointplot kstest(df['data'], 'norm')
it is the result of dividing the coefficient by
boxplot p-value of the test > 0.05: normal data - p-value of the test < 0.05: different variances,
its standard error
- p-value of the test < p-value (alpha) 0.05: data NOT normal heteroscedasticity
Machine Learning Linear Regression: Metrics Logistic Regression: Metrics Balancing for Logistic Regression GridSearch and best_estimator_
from [Link] import r2_score, mean_squared_error, Confusion matrix Downsampling After making the predictions of a model
Standardization mean absolute error
adjust the amount of data of the majority category to
Decision Tree, we examine the metrics of the
Prediction resultados:
R2 represents the proportion of variance that can be Matrix of the minority
change the values of our columns so that explained by the VP of the model; higher R2 = better model if we have overfitting we need to reduce it
the standard deviation of the distribution is equal to 1 and r2_score(y_train,y_predict_train)
confusion Positive Negative Manual method model depth
the mean equal to 0; for the VPs to be comparable df_minoritaria = df[df['col'] == valor_min]
r2_score(y_test, y_predict_test) if we have underfitting we need to increase the
Manual method MAE (Mean Absolute Error): measure of the difference between the Positive True False negativedf_sample = df[df['col'] == max_value].sample model depth
predicted values vs the real ones; lower MAE = better model positive (num_minoritarios, random_state = 42) max_features = [Link](len(x_train.columns))
df['col_esta'] = (df['col_VR'] - df['col_VR'].mean()) /
mean_absolute_error(y_train,y_predict_train) Reality df_balanced = [Link]([df_minoritary,
(df["col_VR"].std()) we can calculate the value of max_features being the
False Negative True df_sample], axis = 0)
mean_absolute_error(y_test, y_predict_test) square root of the number of predictor variables
Sklearn StandardScaler positive negative RandomUnderSample Method
MSE (Mean Squared Error): measures the average of the arbol.tree_.max_depth shows us the max depth
from [Link] import StandardScaler squared errors; lower MSE=better model import imblearn used by default, in order to adjust it; we should
scaler = StandardScaler() mean_squared_error(y_train,y_predict_train) to create a heatmap of a confusion matrix: X = [Link]('col_VR', axis=1) use half at most
[Link](df_num_sin_VR) mean_squared_error(y_test,y_predict_test) from [Link] import confusion_matrix y = df['col_VR'] GridSearch executes all possible
datos_estandarizados = [Link] (df_num_sin_VR) RMSE (Root Mean Squared Error): average distance between the mat_lr = confusion_matrix(y_test, y_pred_test) down_sampler = RandomUnderSampler() combinations of hyperparameters that we give with
df_datos_esta = [Link](standardized_data, columns predicted values and the actual ones; lower RMSE=better model [Link](figsize = (n,m)) X_down, y_down = down_sampler.fit_resample(X,y) the parameter 'param' and best_estimator_ returns the
= df_num_without_VR.columns major combination found
[Link](mean_squared_error(y_train,y_predict_train)) [Link](mat_lr, square=True, annot=True= df_balanceado = [Link]([X_down, y_down], axis = 1)
We define a dictionary of hyperparameters
Sklearn RobustScaler [Link](mean_squared_error(y_test,y_predict_test)) predicted value
Tomek Method
from [Link] import RobustScaler actual value param = {"max_depth": [n,m,l], "max_features":
x = [Link]('col_VR', axis=1) [a,b,c,d], "min_samples_split": [x,y,z],
scaler = RobustScaler() Linear Regression: Model [Link]()
y = df['col_VR'] "min_samples_leaf": [r,s,t]} from
[Link](df_num_sin_VR) Métricas x_train, x_test, y_train, y_test = train_test_split(x, y, sklearn.model_selection import GridSearchCV
datos_estandarizados = [Link] (df_num_sin_VR) 1. separate the data from the predictor variables (x) from the test_size = 0.2, random_state = 42)
from [Link] import confusion_matrix, 2. We start the model with GridSearch
df_data_this = [Link](standardized_data, columns variable response (y) tomek_sampler = SMOTETomek()
accuracy_score, precision_score, recall_score,
= df_num_sin_VR.columns X = [Link]('col_VR', axis=1) gs = GridSearchCV(estimator =
f1_score , cohen_kappa_score, X_train_res, y_train_res =
DecisionTreeRegressor(), param_grid = param, cv=10,
y = df['col_VR'] roc_curve,roc_auc_score tomek_sampler.fit_resample(X_train, y_train)
verbose=-1, return_train_score = True, scoring =
Encoding 2. we split the data into training data and test data Accuracy (exactitud):porcentaje de los valores
those that are foretold are well foretold Upsampling neg_mean_squared_error
test with train_test_split()
Categorical variables adjust the amount of data in the minority category to 3. We adjust the model in the GridSearch
from sklearn.model_selection import train_test_split accuracy_score(y_train, y_predict_train)
the majority [Link](x_train, y_train)
Ordinary: does not require numbers but does consist of an order x_train, x_test, y_train, y_test = train_test_split(X, y, accuracy_score(y_test, y_predict_test)
or a position; median differences between categories test_size = 0.2, random_state = 42) Recall: percentage of captured positive cases Manual method 4. We apply the best_estimator_ method
Nominal: a variable that is not represented by numbers, not 3. We adjust the model If we prefer FP, we want high recall. df_majority = df[df['col'] == valor_may] mejor_modelo = gs.best_estimator_
has some kind of order, and therefore is from sklearn.linear_model import LinearRegression df_sample = df[df['col'] == min_value].sample
recall_score(y_train, y_predict_train) returns the best combination of hyperparameters
mathematically less precise; there will be no big (majority_num, random_state = 42)
lr = LinearRegression(n_jobs=-1) recall_score(y_test, y_predict_test)
differences of medians between categories 5. We bring the predictions back out.
[Link](x_train, y_train) Precision (sensitivity): percentage of df_balanceado = [Link]([df_mayoritaria,
Binary: two possibilities; it can be ordered or not df_sample], axis = 0) y_pred_test_dt2 = mejor_modelo.predict(x_test)
4. We make the predictions correct positive predictions
Unordered variables:we create a new column by value If we prefer FN, we want high accuracy RandomOverSample Method y_pred_train_dt2 = best_model.predict(x_train)
y_predict_train = [Link](x_train)
unique, assigning ones and zeros
y_predict_test = [Link](x_test) precision_score(y_train, y_predict_train) import imblearn Importance of predictors
One-Hot Encoding X = [Link]('col_VR', axis=1)
We save the results in dataframes and concatenate them. precision_score(y_test, y_predict_test) importancia_predictores = [Link](
from [Link] import OneHotEncoder y = df['col_VR']
train_df = [Link]({'Real': y_train, 'Predicted': Specificity: percentage of negative cases {'predictor': x_train.columns, 'importancia':
oh = OneHotEncoder() captured down_sampler = RandomUnderSampler()
y_predict_train, 'Set': ['Train']*len(y_train)}) best_model.feature_importances_
transformed_df = oh.fit_transform(df[['column']]) F1: the average of precision and recall X_down, y_down = down_sampler.fit_resample(X,y)
test_df = [Link]({'Real': y_test, 'Predicted': importance_predictors.sort_values(by=["importance
oh_df = [Link](df_transformados.toarray()) y_predict_test, 'Set': ['Test']*len(y_test)}) balanced_df = [Link]([X_down, y_down], axis = 1)
f1_score(y_train,y_predict_train) creates a
oh_df.columns = oh.get_feature_names_out() resultados = [Link]([train_df,test_df], axis = 0) f1_score(y_test, y_predict_test) dataframe with the relative importance of each VP
df_final = [Link]([df, oh_df], axis=1) 6. we created a column of the waste: the difference between kappa: a measure of agreement that is based on Logistic Regression: Model
get_dummies the observed values and the predicted ones compare the concordance observed in a set for nominal categorical variables to the
df_dum = pd.get_dummies(df['col'], prefix='prefix', resultados['residuos'] = resultados['Real'] - of data, regarding what could happen by mere follow the same steps as for Linear Regression but which have been encoded, they must be added
dtype=int) resultados['Predicted'] chance conLogisticRegression() results of the split columns:
df[df_dum.columns] = [Link] No agreement from sklearn.linear_model import LogisticRegression
Cross-validation df_sum = importancia_predictores_esta.iloc[[n, m]]
[Link]('col', axis=1, inplace=True) 0.0-0.2 Insignificant
from sklearn.model_selection import cross_val_score importance_predictors_this.drop(df_sum.index,
Ordered variables: from sklearn.model_selection import cross_validate
0.2-0.4 Low Decision Tree: Model inplace = True)
0.4-0.6 Moderate
Label Encoding assigns a number to each unique value of a cv_scores = cross_val_score(estimator = LinearRegression(), from sklearn.model_selection import train_test_split importance_predictors_this.loc[n] =
variable X = X, y = y, scoring = 'neg_root_mean_squared_error', cv = - 0.6-0.8 Good
["nombre_col", df_sum["importancia"].sum()]
10) 0.8-1.0 Very good from [Link] import DecisionTreeRegressor
from [Link] import LabelEncoder
from sklearn import tree
le = LabelEncoder() cv_scores.mean() cohen_kappa_score(y_train, y_predict_train)
follow the same steps as for Linear Regression but
Random Forest: Model
df['col_VR_le'] = le.fit_transform(df[col_VR']) calculate the average of the CV results of a metric cohen_kappa_score(y_test, y_predict_test) conDecisionTreeRegressor() or DecisionTreeClassifier()
map() assigns the value we want according to the map that cv_scores = cross_validate(estimator = LinearRegression(), X ROC curve: graphical representation of kappa; the follow the same steps as for the Decision Tree
= X, y = y, scoring ='r2', 'neg_root_mean_squared_error', cv sensitivity vs. specificity arbol = DecisionTreeRegressor(random_state=42) but with RandomForestRegressor() or
we create
= 10) AUC (area under curve): the area under the ROC curve; To draw the tree: Random Forest Classifier()
df['col_VR_map'] = df[col_VR'].map(dictionary)
cv_scores['test_r2'].mean() the closer to 1, the better our model will be fig = [Link](figsize = (10,6)) from [Link] import RandomForestRegressor
Ordinal encoding assigns labels based on an order or
cv_scores['test_neg_root_mean_squared_error'].mean() calculates classifying the VPs tree.plot_tree(tree, feature_names = x_train.columns,
hierarchy the same hyperparameters can be used
the averages of the multiple validation results filled = True best_estimator_ or rerun the GridSearch
from [Link] import OrdinalEncoder
metrics [Link]()