Function

Normal Functions
1. col(col)

Description:
Returns a Column based on the given column name.

Parameters:

 col (String): The name of the column to retrieve.

Examples:

# Example 1: Retrieve the 'age' column

age_column = col('age')

# Example 2: Access the 'salary' column

salary_column = col('salary')

2. column(col)

Description:
Returns a Column based on the given column name.

Parameters:

 col (String): The name of the column to retrieve.

Examples:

# Example 1: Retrieve the 'age' column

age_column = column('age')

# Example 2: Access the 'salary' column

salary_column = column('salary')

3. lit(col)

Description:
Creates a Column of literal value.

Parameters:
  col (Any): The literal value to convert into a column.

Examples:

# Example 1: Create a column with a literal value of 10

literal_value_column = lit(10)

# Example 2: Create a column with a literal string value

literal_string_column = lit("Hello")

4. broadcast(df)

Description:
Marks a DataFrame as small enough for use in broadcast joins.

Parameters:

 df (DataFrame): The DataFrame to be broadcasted.

Examples:

# Example 1: Mark a DataFrame for broadcast join

broadcasted_df = broadcast(small_df)

# Example 2: Use broadcast for a join operation

joined_df = [Link](broadcast(large_df), [Link] == large_df.id)

5. coalesce(*cols)

Description:
Returns the first column that is not null from the given columns.

Parameters:

 *cols (Column): Columns to evaluate.

Examples:

# Example 1: Coalesce columns and return the first non-null value

result_column = coalesce(df.col1, df.col2)

# Example 2: Coalesce more columns and return the first non-null value
result_column = coalesce(df.col1, df.col2, df.col3)

6. input_file_name()
Description:
Creates a string column for the file name of the current Spark task.

Parameters:

 None.

Examples:

# Example 1: Get the file name of the current task

file_name_column = input_file_name()

# Example 2: Add file name to a DataFrame

df_with_file_name = [Link]("file_name", input_file_name())

7. isnan(col)

Description:
An expression that returns true if the column is NaN.

Parameters:

 col (Column): The column to check.

Examples:

# Example 1: Check if the 'age' column has NaN values

is_nan_column = isnan([Link])

# Example 2: Filter rows where the 'salary' column is NaN

filtered_df = [Link](isnan([Link]))

8. isnull(col)

Description:
An expression that returns true if the column is null.

Parameters:

 col (Column): The column to check.

Examples:

# Example 1: Check if the 'salary' column is null

is_null_column = isnull([Link])
# Example 2: Filter rows where the 'age' column is null
filtered_df = [Link](isnull([Link]))

9. monotonically_increasing_id()

Description:
A column that generates monotonically increasing 64-bit integers.

Parameters:

 None.

Examples:

# Example 1: Add a monotonically increasing ID column to a DataFrame

df_with_id = [Link]("id", monotonically_increasing_id())

# Example 2: Use monotonically increasing ID to partition the DataFrame

df_partitioned = [Link]("id",
monotonically_increasing_id()).repartition(4)

10. named_struct(*cols)

Description:
Creates a struct with the given field names and values.

Parameters:

 *cols (List of Column names and values): The names and corresponding values to create
the struct.

Examples:

# Example 1: Create a struct from 'name' and 'age' columns

struct_column = named_struct('name', [Link], 'age', [Link])

# Example 2: Create a struct with more fields

struct_column = named_struct('name', [Link], 'age', [Link], 'salary',
[Link])

11. nanvl(col1, col2)

Description:
Returns col1 if it is not NaN, or col2 if col1 is NaN.
Parameters:

 col1 (Column): First column to check for NaN.

 col2 (Column): Second column to return if col1 is NaN.

Examples:

# Example 1: Use 'col1' if not NaN, otherwise use 'col2'

result_column = nanvl(df.col1, df.col2)

# Example 2: Coalesce 'col1' and 'col2' if 'col1' is NaN

result_column = nanvl(df.col1, df.col2)

12. rand([seed])

Description:
Generates a random column with independent and identically distributed (i.i.d.) samples
uniformly distributed in [0.0, 1.0).

Parameters:

 seed (Optional, Integer): Random seed for reproducibility.

Examples:

# Example 1: Generate a random column

random_column = rand()

# Example 2: Generate a random column with a specific seed

random_column_with_seed = rand(123)

13. randn([seed])

Description:
Generates a column with independent and identically distributed (i.i.d.) samples from the
standard normal distribution.

Parameters:

 seed (Optional, Integer): Random seed for reproducibility.

Examples:

# Example 1: Generate a random normal column

random_normal_column = randn()
# Example 2: Generate a random normal column with a specific seed
random_normal_column_with_seed = randn(42)

14. spark_partition_id()

Description:
A column for partition ID.

Parameters:

 None.

Examples:

# Example 1: Get the partition ID of the current row

partition_id_column = spark_partition_id()

# Example 2: Use partition ID for processing

df_with_partition_id = [Link]("partition_id", spark_partition_id())

15. when(condition, value)

Description:
Evaluates a list of conditions and returns one of multiple possible result expressions.

Parameters:

 condition (Column): The condition to evaluate.

 value (Any): The value to return when the condition is true.

Examples:

# Example 1: Apply a conditional expression

result_column = when([Link] > 30, "Senior").otherwise("Junior")

# Example 2: Use when with multiple conditions

result_column = when([Link] > 50000, "High").when([Link] < 20000,
"Low").otherwise("Medium")

16. bitwise_not(col)

Description:
Computes bitwise not.

Parameters:
  col (Column): The column to apply the bitwise not operation on.

Examples:

# Example 1: Apply bitwise not operation

not_column = bitwise_not(df.number_column)

# Example 2: Apply bitwise not to a different column

not_column2 = bitwise_not(df.another_column)

17. bitwiseNOT(col)

Description:
Computes bitwise not.

Parameters:

 col (Column): The column to apply the bitwise not operation on.

Examples:

# Example 1: Apply bitwise NOT operation

not_column = bitwiseNOT(df.number_column)

# Example 2: Apply bitwise NOT to a different column

not_column2 = bitwiseNOT(df.another_column)

18. expr(str)

Description:
Parses the expression string into the column that it represents.

Parameters:

 str (String): The expression to parse.

Examples:

# Example 1: Use an expression string to create a column

expression_column = expr("age + salary")

# Example 2: Parse a string expression with conditional logic

conditional_column = expr("CASE WHEN age > 30 THEN 'Senior' ELSE 'Junior'
END")

19. greatest(*cols)
Description:
Returns the greatest value of the list of column names, skipping null values.

Parameters:

 *cols (List of Columns): Columns to compare.

Examples:

# Example 1: Find the greatest value across columns

greatest_column = greatest(df.col1, df.col2, df.col3)

# Example 2: Find the greatest value, skipping nulls

greatest_column = greatest(df.col1, df.col2)

20. least(*cols)

Description:
Returns the least value of the list of column names, skipping null values.

Parameters:

 *cols (List of Columns): Columns to compare.

Examples:

# Example 1: Find the least value across columns

least_column = least(df.col1, df.col2, df.col3)

# Example 2: Find the least value, skipping nulls

least_column = least(df.col1, df.col2)

21. sqrt(col)

Description:
Computes the square root of the specified float value.

Parameters:

 col (Column): The column whose square root is to be calculated.

Examples:

# Example 1: Calculate the square root of 'age' column

sqrt_column = sqrt([Link])

# Example 2: Calculate the square root of a constant value

sqrt_constant = sqrt(lit(16))
22. abs(col)

Description:
Computes the absolute value of the specified column.

Parameters:

 col (Column): The column whose absolute value is to be calculated.

Examples:

# Example 1: Calculate the absolute value of 'salary' column

abs_column = abs([Link])

# Example 2: Calculate the absolute value of 'debt' column

abs_debt_column = abs([Link])

23. acos(col)

Description:
Computes the inverse cosine (arccosine) of the input column.

Parameters:

 col (Column): The column whose inverse cosine is to be calculated.

Examples:

# Example 1: Compute the inverse cosine of the 'value' column

acos_column = acos([Link])

# Example 2: Compute inverse cosine of a literal value

acos_literal = acos(lit(0.5))

24. acosh(col)

Description:
Computes inverse hyperbolic cosine of the input column.

Parameters:

 col (Column): The column whose inverse hyperbolic cosine is to be calculated.

Examples:
# Example 1: Compute inverse hyperbolic cosine of 'value'
acosh_column = acosh([Link])

# Example 2: Compute inverse hyperbolic cosine of a literal value

acosh_literal = acosh(lit(2))

25. asin(col)

Description:
Computes inverse sine (arcsine) of the input column.

Parameters:

 col (Column): The column whose inverse sine is to be calculated.

Examples:

# Example 1: Compute the inverse sine of 'angle' column

asin_column = asin([Link])

# Example 2: Compute inverse sine of a literal value

asin_literal = asin(lit(0.5))

26. asinh(col)

Description:
Computes inverse hyperbolic sine of the input column.

Parameters:

 col (Column): The column whose inverse hyperbolic sine is to be calculated.

Examples:

# Example 1: Compute inverse hyperbolic sine of 'value' column

asinh_column = asinh([Link])

# Example 2: Compute inverse hyperbolic sine of a literal value

asinh_literal = asinh(lit(2))

27. atan(col)

Description:
Computes inverse tangent (arctangent) of the input column.

Parameters:
  col (Column): The column whose inverse tangent is to be calculated.

Examples:

# Example 1: Compute inverse tangent of 'angle' column

atan_column = atan([Link])

# Example 2: Compute inverse tangent of a literal value

atan_literal = atan(lit(1))

28. atanh(col)

Description:
Computes inverse hyperbolic tangent of the input column.

Parameters:

 col (Column): The column whose inverse hyperbolic tangent is to be calculated.

Examples:

# Example 1: Compute inverse hyperbolic tangent of 'value' column

atanh_column = atanh([Link])

# Example 2: Compute inverse hyperbolic tangent of a literal value

atanh_literal = atanh(lit(0.5))

29. atan2(col1, col2) (New in version 1.4.0)

Description:
Computes the inverse tangent of the two columns col1 and col2. This returns the angle formed
by the point (col1, col2) in a two-dimensional plane.

Parameters:

 col1 (Column): The first column, representing the x-coordinate.

 col2 (Column): The second column, representing the y-coordinate.

Examples:

# Example 1: Compute the inverse tangent of 'x' and 'y' columns

atan2_column = atan2(df.x, df.y)

# Example 2: Compute inverse tangent of literal values

atan2_literal = atan2(lit(1), lit(1))
30. bin(col)

Description:
Returns the string representation of the binary value of the given column.

Parameters:

 col (Column): The column whose binary value is to be calculated.

Examples:

# Example 1: Convert the 'value' column to binary

bin_column = bin([Link])

# Example 2: Convert a constant integer value to binary

bin_literal = bin(lit(10))

31. cbrt(col)

Description:
Computes the cube-root of the given value.

Parameters:

 col (Column): The column whose cube root is to be calculated.

Examples:

# Example 1: Calculate the cube root of 'value' column

cbrt_column = cbrt([Link])

# Example 2: Calculate the cube root of a constant value

cbrt_literal = cbrt(lit(27))

32. ceil(col)

Description:
Computes the ceiling of the given value, which rounds the value up to the nearest integer.

Parameters:

 col (Column): The column whose ceiling is to be calculated.

Examples:

# Example 1: Calculate the ceiling of 'salary' column

ceil_column = ceil([Link])

# Example 2: Calculate the ceiling of a literal value

ceil_literal = ceil(lit(3.4))

33. ceiling(col)

Description:
Computes the ceiling of the given value, which rounds the value up to the nearest integer.

Parameters:

 col (Column): The column whose ceiling is to be calculated.

Examples:

# Example 1: Calculate the ceiling of 'salary' column

ceiling_column = ceiling([Link])

# Example 2: Calculate the ceiling of a literal value

ceiling_literal = ceiling(lit(3.4))

34. conv(col, fromBase, toBase)

Description:
Converts a number in a string column from one base to another.

Parameters:

 col (Column): The column containing the number to be converted.

 fromBase (Integer): The base of the input number.
 toBase (Integer): The base to convert to.

Examples:

# Example 1: Convert a number from binary to decimal

conv_column = conv(df.number_column, 2, 10)

# Example 2: Convert a number from hexadecimal to decimal

conv_column_hex = conv(df.hex_column, 16, 10)

35. cos(col)

Description:
Computes the cosine of the input column.
Parameters:

 col (Column): The column whose cosine is to be calculated.

Examples:

# Example 1: Calculate the cosine of 'angle' column

cos_column = cos([Link])

# Example 2: Calculate the cosine of a constant value

cos_constant = cos(lit(0.5))

36. cosh(col)

Description:
Computes the hyperbolic cosine of the input column.

Parameters:

 col (Column): The column whose hyperbolic cosine is to be calculated.

Examples:

# Example 1: Calculate the hyperbolic cosine of 'value' column

cosh_column = cosh([Link])

# Example 2: Calculate the hyperbolic cosine of a constant value

cosh_constant = cosh(lit(0.5))

37. cot(col)

Description:
Computes the cotangent of the input column.

Parameters:

 col (Column): The column whose cotangent is to be calculated.

Examples:

# Example 1: Calculate the cotangent of 'angle' column

cot_column = cot([Link])

# Example 2: Calculate the cotangent of a constant value

cot_constant = cot(lit(1))
38. csc(col)

Description:
Computes the cosecant of the input column.

Parameters:

 col (Column): The column whose cosecant is to be calculated.

Examples:

# Example 1: Calculate the cosecant of 'angle' column

csc_column = csc([Link])

# Example 2: Calculate the cosecant of a constant value

csc_constant = csc(lit(1))

39. e()

Description:
Returns Euler’s number (the base of natural logarithms).

Parameters:

 No parameters.

Examples:

# Example 1: Get Euler's number

e_value = e()

40. exp(col)

Description:
Computes the exponential of the given value (e^x).

Parameters:

 col (Column): The column whose exponential is to be calculated.

Examples:

# Example 1: Calculate the exponential of 'value' column

exp_column = exp([Link])

# Example 2: Calculate the exponential of a constant value

exp_constant = exp(lit(1))

41. log(arg1[, arg2])

Description:
Returns the logarithm of the first argument to the base of the second argument. If the second
argument is not provided, it computes the natural logarithm.

Parameters:

 arg1 (Column or Literal): The value for which the logarithm is calculated.
 arg2 (Column or Literal, optional): The base for the logarithm. If not provided, the
natural logarithm is calculated.

Examples:

# Example 1: Compute the logarithm of 'value' to base 10

log_column = log([Link], 10)

# Example 2: Compute the natural logarithm of 'value' column

log_natural_column = log([Link])

42. log10(col)

Description:
Computes the logarithm of the given value in base 10.

Parameters:

 col (Column): The column whose logarithm to base 10 is to be calculated.

Examples:

# Example 1: Compute the logarithm (base 10) of 'value' column

log10_column = log10([Link])

# Example 2: Compute the logarithm (base 10) of a literal value

log10_literal = log10(lit(100))

43. log1p(col)

Description:
Computes the natural logarithm of the given value plus one, i.e., log(1 + col).

Parameters:
  col (Column): The column whose log1p is to be calculated.

Examples:

# Example 1: Compute log(1 + 'value') column

log1p_column = log1p([Link])

# Example 2: Compute log(1 + literal value)

log1p_literal = log1p(lit(5))

44. log2(col)

Description:
Returns the logarithm of the given column in base 2.

Parameters:

 col (Column): The column whose logarithm to base 2 is to be calculated.

Examples:

# Example 1: Compute the base-2 logarithm of 'value' column

log2_column = log2([Link])

# Example 2: Compute the base-2 logarithm of a literal value

log2_literal = log2(lit(8))

45. negate(col)

Description:
Returns the negative value of the specified column.

Parameters:

 col (Column): The column whose negative value is to be calculated.

Examples:

# Example 1: Negate the 'salary' column

negate_column = negate([Link])

# Example 2: Negate a literal value

negate_literal = negate(lit(5))

46. negative(col)
Description:
Returns the negative value of the specified column.

Parameters:

 col (Column): The column whose negative value is to be calculated.

Examples:

# Example 1: Negative of the 'value' column

negative_column = negative([Link])

# Example 2: Negative of a constant value

negative_literal = negative(lit(10))

47. pi()

Description:
Returns the constant value of Pi (π).

Parameters:

 No parameters.

Examples:

# Example 1: Get the value of Pi

pi_value = pi()

48. pmod(dividend, divisor)

Description:
Returns the positive value of dividend mod divisor, which is the remainder of the division of the
two values.

Parameters:

 dividend (Column): The number to be divided.

 divisor (Column): The number by which the dividend is divided.

Examples:

# Example 1: Compute positive modulo of 'value' and 3

pmod_column = pmod([Link], lit(3))

# Example 2: Compute positive modulo of 10 and 3

pmod_literal = pmod(lit(10), lit(3))

49. positive(col)

Description:
Returns the value of the specified column (identical to the input).

Parameters:

 col (Column): The column whose value is to be returned.

Examples:

# Example 1: Return the 'value' column as is

positive_column = positive([Link])

# Example 2: Return a literal value as is

positive_literal = positive(lit(5))

50. pow(col1, col2)

Description:
Returns the value of the first argument raised to the power of the second argument (col1^col2).

Parameters:

 col1 (Column): The base value.

 col2 (Column): The exponent value.

Examples:

# Example 1: Raise 'value' column to the power of 2

pow_column = pow([Link], lit(2))

# Example 2: Raise 'base' column to the power of 'exponent' column

pow_column2 = pow([Link], [Link])

51. power(col1, col2)

Description:
Returns the value of the first argument raised to the power of the second argument (col1^col2).
This is equivalent to pow.

Parameters:
  col1 (Column): The base value.
 col2 (Column): The exponent value.

Examples:

# Example 1: Raise 'value' column to the power of 2

power_column = power([Link], lit(2))

# Example 2: Raise 'base' column to the power of 'exponent' column

power_column2 = power([Link], [Link])

52. rint(col)

Description:
Returns the double value that is closest in value to the argument and is equal to a mathematical
integer (rounds to the nearest integer).

Parameters:

 col (Column): The column whose value is to be rounded.

Examples:

# Example 1: Round 'value' column to the nearest integer

rint_column = rint([Link])

# Example 2: Round a constant value to the nearest integer

rint_literal = rint(lit(3.6))

53. round(col[, scale])

Description:
Rounds the given value to a specified number of decimal places. If scale >= 0, it rounds the
value to that many decimal places using HALF_UP rounding. If scale < 0, it rounds to the
integral part.

Parameters:

 col (Column): The column whose value is to be rounded.

 scale (Integer, optional): The number of decimal places to round to. Defaults to 0 if not
provided.

Examples:

# Example 1: Round the 'value' column to 2 decimal places

round_column = round([Link], 2)
# Example 2: Round the 'value' column to the nearest integer
round_column2 = round([Link])

54. bround(col[, scale])

Description:
Rounds the given value to the specified number of decimal places using HALF_EVEN rounding
mode if scale >= 0 or at the integral part when scale < 0.

Parameters:

 col (Column): The column whose value is to be rounded.

 scale (Integer, optional): The number of decimal places to round to. Defaults to 0 if not
provided.

Examples:

# Example 1: Round 'value' column to 2 decimal places using HALF_EVEN rounding

bround_column = bround([Link], 2)

# Example 2: Round 'value' column to the nearest integer using HALF_EVEN

rounding
bround_column2 = bround([Link])

55. sec(col)

Description:
Computes the secant (1/cosine) of the input column.

Parameters:

 col (Column): The column whose secant is to be calculated.

Examples:

# Example 1: Compute the secant of 'angle' column

sec_column = sec([Link])

# Example 2: Compute the secant of a constant value (e.g., 1)

sec_literal = sec(lit(1))

56. shiftleft(col, numBits)

Description:
Shift the given value in the specified column to the left by the specified number of bits.

Parameters:

 col (Column): The column whose value is to be shifted.

 numBits (Integer): The number of bits to shift.

Examples:

# Example 1: Shift the 'value' column left by 3 bits

shiftleft_column = shiftleft([Link], 3)

# Example 2: Shift a constant value (e.g., 5) left by 2 bits

shiftleft_literal = shiftleft(lit(5), 2)

57. shiftright(col, numBits)

Description:
(Signed) shift the given value in the specified column to the right by the specified number of bits.

Parameters:

 col (Column): The column whose value is to be shifted.

 numBits (Integer): The number of bits to shift.

Examples:

# Example 1: Shift the 'value' column right by 3 bits

shiftright_column = shiftright([Link], 3)

# Example 2: Shift a constant value (e.g., 16) right by 2 bits

shiftright_literal = shiftright(lit(16), 2)

58. shiftrightunsigned(col, numBits)

Description:
Unsigned shift the given value in the specified column to the right by the specified number of
bits. This means no sign extension, unlike shiftright.

Parameters:

 col (Column): The column whose value is to be shifted.

 numBits (Integer): The number of bits to shift.

Examples:
# Example 1: Unsigned shift the 'value' column right by 3 bits
shiftrightunsigned_column = shiftrightunsigned([Link], 3)

# Example 2: Unsigned shift a constant value (e.g., 16) right by 2 bits

shiftrightunsigned_literal = shiftrightunsigned(lit(16), 2)

59. sign(col)

Description:
Computes the signum (sign) of the given value, returning 1 for positive, 0 for zero, and -1 for
negative values.

Parameters:

 col (Column): The column whose signum is to be computed.

Examples:

# Example 1: Get the sign of the 'value' column

sign_column = sign([Link])

# Example 2: Get the sign of a constant value (e.g., -5)

sign_literal = sign(lit(-5))

60. signum(col)

Description:
Computes the signum (sign) of the given value, equivalent to sign. It returns 1 for positive, 0 for
zero, and -1 for negative values.

Parameters:

 col (Column): The column whose signum is to be computed.

Examples:

# Example 1: Get the signum of the 'value' column

signum_column = signum([Link])

# Example 2: Get the signum of a constant value (e.g., 10)

signum_literal = signum(lit(10))

61. sin(col)

Description:
Computes the sine of the input column.
Parameters:

 col (Column): The column whose sine is to be calculated.

Examples:

# Example 1: Compute the sine of 'angle' column

sin_column = sin([Link])

# Example 2: Compute the sine of a constant value (e.g., π/2)

sin_literal = sin(lit(3.14159 / 2))

62. sinh(col)

Description:
Computes the hyperbolic sine of the input column.

Parameters:

 col (Column): The column whose hyperbolic sine is to be calculated.

Examples:

# Example 1: Compute the hyperbolic sine of 'value' column

sinh_column = sinh([Link])

# Example 2: Compute the hyperbolic sine of a constant value

sinh_literal = sinh(lit(1))

63. tan(col)

Description:
Computes the tangent of the input column.

Parameters:

 col (Column): The column whose tangent is to be calculated.

Examples:

# Example 1: Compute the tangent of 'angle' column

tan_column = tan([Link])

# Example 2: Compute the tangent of a constant value (e.g., π/4)

tan_literal = tan(lit(3.14159 / 4))
64. tanh(col)

Description:
Computes the hyperbolic tangent of the input column.

Parameters:

 col (Column): The column whose hyperbolic tangent is to be calculated.

Examples:

# Example 1: Compute the hyperbolic tangent of 'value' column

tanh_column = tanh([Link])

# Example 2: Compute the hyperbolic tangent of a constant value

tanh_literal = tanh(lit(1))

65. toDegrees(col)

Description:
Converts an angle measured in radians to an approximately equivalent angle measured in
degrees. Introduced in version 1.4.0.

Parameters:

 col (Column): The column containing values in radians to be converted to degrees.

Examples:

# Example 1: Convert 'angle' column from radians to degrees

toDegrees_column = toDegrees([Link])

# Example 2: Convert a constant value (e.g., π/2 radians) to degrees

toDegrees_literal = toDegrees(lit(3.14159 / 2))

66. try_add(left, right)

Description:
Returns the sum of left and right. If there is an overflow, the result is null.

Parameters:

 left (Column): The left operand of the addition.

 right (Column): The right operand of the addition.

Examples:
# Example 1: Add 'value1' and 'value2' columns safely
try_add_column = try_add(df.value1, df.value2)

# Example 2: Add two constant values safely

try_add_literal = try_add(lit(100), lit(200))

67. try_avg(col)

Description:
Returns the mean calculated from values of a group, and the result is null on overflow.

Parameters:

 col (Column): The column to compute the average of.

Examples:

# Example 1: Calculate the average of the 'value' column safely

try_avg_column = try_avg([Link])

# Example 2: Calculate the average of a constant value (e.g., 100)

try_avg_literal = try_avg(lit(100))

69. try_divide(left, right)

Description:
Returns the result of left / right. If there is an overflow or division by zero, the result is null.

Parameters:

 left (Column): The numerator in the division.

 right (Column): The denominator in the division.

Examples:

# Example 1: Divide 'value1' by 'value2' safely

try_divide_column = try_divide(df.value1, df.value2)

# Example 2: Divide two constant values safely

try_divide_literal = try_divide(lit(100), lit(0))

70. try_multiply(left, right)

Description:
Returns the result of left * right. If there is an overflow, the result is null.
Parameters:

 left (Column): The first operand for multiplication.

 right (Column): The second operand for multiplication.

Examples:

# Example 1: Multiply 'value1' and 'value2' safely

try_multiply_column = try_multiply(df.value1, df.value2)

# Example 2: Multiply two constant values safely

try_multiply_literal = try_multiply(lit(100), lit(200))

71. try_subtract(left, right)

Description:
Returns the result of left - right. If there is an overflow, the result is null.

Parameters:

 left (Column): The value to subtract from.

 right (Column): The value to subtract.

Examples:

# Example 1: Subtract 'value2' from 'value1' safely

try_subtract_column = try_subtract(df.value1, df.value2)

# Example 2: Subtract two constant values safely

try_subtract_literal = try_subtract(lit(200), lit(100))

72. try_sum(col)

Description:
Returns the sum calculated from values of a group, and the result is null on overflow.

Parameters:

 col (Column): The column to compute the sum of.

Examples:

# Example 1: Calculate the sum of the 'value' column safely

try_sum_column = try_sum([Link])

# Example 2: Calculate the sum of a constant value (e.g., 100)

try_sum_literal = try_sum(lit(100))
73. try_to_binary(col[, format])

Description:
This is a special version of to_binary that performs the same operation, but returns a NULL
value instead of raising an error if the conversion cannot be performed.

Parameters:

 col (Column): The column to convert to binary.

 format (Optional, String): The format to use for the conversion (default is null).

Examples:

# Example 1: Convert the 'value' column to binary safely

try_to_binary_column = try_to_binary([Link])

# Example 2: Convert a constant value (e.g., 'text') to binary safely

try_to_binary_literal = try_to_binary(lit("text"))

74. try_to_number(col, format)

Description:
Convert string col to a number based on the specified string format. If the conversion fails, the
result is null.

Parameters:

 col (Column): The string column to convert to a number.

 format (String): The format to use for the conversion.

Examples:

# Example 1: Convert the 'value' column to a number safely

try_to_number_column = try_to_number([Link], "####")

# Example 2: Convert a string literal to a number safely

try_to_number_literal = try_to_number(lit("1234"), "####")

75. degrees(col)

Description:
Converts an angle measured in radians to an approximately equivalent angle measured in
degrees.
Parameters:

 col (Column): The column containing values in radians to be converted to degrees.

Examples:

# Example 1: Convert 'angle' column from radians to degrees

degrees_column = degrees([Link])

# Example 2: Convert a constant value (e.g., π/2 radians) to degrees

degrees_literal = degrees(lit(3.14159 / 2))

76. toRadians(col)

Description:
Converts an angle measured in degrees to an approximately equivalent angle measured in
radians.

Parameters:

 col (Column): The column containing values in degrees to be converted to radians.

Examples:

# Example 1: Convert 'angle' column from degrees to radians

toRadians_column = toRadians([Link])

# Example 2: Convert a constant value (e.g., 90 degrees) to radians

toRadians_literal = toRadians(lit(90))

77. radians(col)

Description:
Converts an angle measured in degrees to an approximately equivalent angle measured in
radians.

Parameters:

 col (Column): The column containing values in degrees to be converted to radians.

Examples:

# Example 1: Convert 'angle' column from degrees to radians

radians_column = radians([Link])

# Example 2: Convert a constant value (e.g., 180 degrees) to radians

radians_literal = radians(lit(180))
78. width_bucket(v, min, max, numBucket)

Description:
Returns the bucket number into which the value of this expression would fall after being
evaluated.

Parameters:

 v (Column): The column to be evaluated.

 min (Numeric): The lower bound of the bucket.
 max (Numeric): The upper bound of the bucket.
 numBucket (Int): The number of buckets to divide the range into.

Examples:

# Example 1: Get the bucket number for values in 'score' column

bucket_column = width_bucket([Link], lit(0), lit(100), lit(5))

# Example 2: Get the bucket number for a constant value (e.g., 50)
bucket_literal = width_bucket(lit(50), lit(0), lit(100), lit(10))

Datetime Functions:
79. add_months(start, months)

Description:
Returns the date that is months months after the start date.

Parameters:

 start (Column): The starting date.

 months (Int): The number of months to add (can be negative to subtract).

Examples:

# Example 1: Add 3 months to the 'start_date' column

new_date_column = add_months(df.start_date, lit(3))

# Example 2: Subtract 2 months from a given date literal

new_date_literal = add_months(lit("2023-01-01"), lit(-2))

80. convert_timezone(sourceTz, targetTz, sourceTs)

Description:
Converts the timestamp sourceTs from the sourceTz time zone to the targetTz time zone.

Parameters:

 sourceTz (String): The source time zone.

 targetTz (String): The target time zone.
 sourceTs (Column): The timestamp to be converted.

Examples:

# Example 1: Convert 'timestamp' from 'UTC' to 'PST'

converted_time_column = convert_timezone(lit("UTC"), lit("PST"), [Link])

# Example 2: Convert a specific timestamp literal from 'CET' to 'UTC'

converted_time_literal = convert_timezone(lit("CET"), lit("UTC"), lit("2023-
01-01 12:00:00"))

81. curdate()

Description:
Returns the current date at the start of query evaluation as a DateType column.

Parameters:
None.

Examples:

# Example 1: Get the current date

current_date_column = curdate()

# Example 2: Use the current date in a conditional statement

date_check = df.date_column == curdate()

82. current_date()

Description:
Returns the current date at the start of query evaluation as a DateType column.

Parameters:
None.

Examples:

# Example 1: Get the current date

current_date_column = current_date()
# Example 2: Compare a date column to the current date
is_today = df.date_column == current_date()

83. current_timestamp()

Description:
Returns the current timestamp at the start of query evaluation as a TimestampType column.

Parameters:
None.

Examples:

# Example 1: Get the current timestamp

current_timestamp_column = current_timestamp()

# Example 2: Compare a timestamp column to the current timestamp

is_now = df.timestamp_column == current_timestamp()

84. current_timezone()

Description:
Returns the current session local timezone.

Parameters:
None.

Examples:

# Example 1: Get the current session local timezone

timezone_column = current_timezone()

# Example 2: Compare a timestamp to the current session timezone

timezone_check = df.timestamp_column == current_timezone()

85. date_add(start, days)

Description:
Returns the date that is days days after the start date.

Parameters:

 start (Column): The starting date.

 days (Int): The number of days to add (can be negative to subtract).
Examples:

# Example 1: Add 5 days to the 'start_date' column

new_date_column = date_add(df.start_date, lit(5))

# Example 2: Subtract 3 days from a given date literal

new_date_literal = date_add(lit("2023-01-01"), lit(-3))

86. date_diff(end, start)

Description:
Returns the number of days from start to end.

Parameters:

 end (Column): The ending date.

 start (Column): The starting date.

Examples:

# Example 1: Calculate the number of days between 'end_date' and 'start_date'

date_diff_column = date_diff(df.end_date, df.start_date)

# Example 2: Calculate the difference between two specific dates

date_diff_literal = date_diff(lit("2023-01-10"), lit("2023-01-01"))

87. date_format(date, format)

Description:
Converts a date, timestamp, or string to a value of string in the format specified by the date
format given by the second argument.

Parameters:

 date (Column): The date/timestamp/string to be formatted.

 format (String): The format to apply.

Examples:

# Example 1: Format the 'date_column' to a string in 'yyyy-MM-dd' format

formatted_date_column = date_format(df.date_column, lit("yyyy-MM-dd"))

# Example 2: Format a specific date literal to 'dd/MM/yyyy'

formatted_date_literal = date_format(lit("2023-01-01"), lit("dd/MM/yyyy"))
88. date_from_unix_date(days)

Description:
Create a date from the number of days since 1970-01-01.

Parameters:

 days (Int): The number of days since 1970-01-01.

Examples:

# Example 1: Create a date from the number of days (e.g., 10000 days since
epoch)
date_column = date_from_unix_date(lit(10000))

# Example 2: Convert a constant value (e.g., 20000 days) to a date

date_literal = date_from_unix_date(lit(20000))

89. date_sub(start, days)

Description:
Returns the date that is days days before the start date.

Parameters:

 start (Column): The starting date.

 days (Int): The number of days to subtract.

Examples:

# Example 1: Subtract 5 days from the 'start_date' column

new_date_column = date_sub(df.start_date, lit(5))

# Example 2: Subtract 10 days from a given date literal

new_date_literal = date_sub(lit("2023-01-01"), lit(10))

90. date_trunc(format, timestamp)

Description:
Returns the timestamp truncated to the unit specified by the format.

Parameters:

 format (String): The unit to truncate (e.g., 'year', 'month', 'day').

 timestamp (Column): The timestamp to truncate.
Examples:

# Example 1: Truncate the 'timestamp_column' to the beginning of the year

truncated_timestamp_column = date_trunc(lit("year"), df.timestamp_column)

# Example 2: Truncate a specific timestamp to the beginning of the month

truncated_timestamp_literal = date_trunc(lit("month"), lit("2023-01-15
14:45:00"))

91. dateadd(start, days)

Description:
Returns the date that is days days after the start date. (Synonym for date_add.)

Parameters:

 start (Column): The starting date.

 days (Int): The number of days to add.

Examples:

# Example 1: Add 10 days to the 'start_date' column

new_date_column = dateadd(df.start_date, lit(10))

# Example 2: Add 7 days to a specific date literal

new_date_literal = dateadd(lit("2023-01-01"), lit(7))

92. datediff(end, start)

Description:
Returns the number of days from start to end. (Synonym for date_diff.)

Parameters:

 end (Column): The ending date.

 start (Column): The starting date.

Examples:

# Example 1: Calculate the number of days between 'end_date' and 'start_date'

date_diff_column = datediff(df.end_date, df.start_date)

# Example 2: Calculate the difference between two specific dates

date_diff_literal = datediff(lit("2023-01-10"), lit("2023-01-01"))

93. day(col)
Description:
Extracts the day of the month from a given date or timestamp column as an integer.

Parameters:

 col (Column): The column containing a date or timestamp.

Examples:

# Example 1: Extract the day of the month from 'timestamp_column'

day_column = day(df.timestamp_column)

# Example 2: Extract the day from a specific date literal

day_literal = day(lit("2023-01-15"))

date_part(field, source)

Description:
Extracts a part of the date, timestamp, or interval source based on the field.

Parameters:

 field (String): The part of the date/timestamp to extract (e.g., 'year', 'month', 'day').
 source (Column): The column containing the date or timestamp to extract from.

Examples:

# Example 1: Extract the year from the 'timestamp_column'

year_column = date_part(lit("year"), df.timestamp_column)

# Example 2: Extract the month from a specific date literal

month_literal = date_part(lit("month"), lit("2023-01-01"))

95. datepart(field, source)

Description:
Synonym for date_part. Extracts a part of the date, timestamp, or interval source.

Parameters:

 field (String): The part of the date/timestamp to extract (e.g., 'year', 'month', 'day').
 source (Column): The column containing the date or timestamp to extract from.

Examples:

# Example 1: Extract the day from the 'timestamp_column'

day_column = datepart(lit("day"), df.timestamp_column)
# Example 2: Extract the hour from a specific timestamp literal
hour_literal = datepart(lit("hour"), lit("2023-01-01 14:30:00"))

96. dayofmonth(col)

Description:
Extract the day of the month from the given date or timestamp column as an integer.

Parameters:

 col(Column): The column containing the date or timestamp from which the day of the
month is extracted.

Examples:

# Example 1: Extract the day of the month from 'timestamp_column'

day_of_month_column = dayofmonth(df.timestamp_column)

# Example 2: Extract the day of the month from a specific date literal
day_of_month_literal = dayofmonth(lit("2023-01-15"))

97. dayofweek(col)

Description:
Extract the day of the week from the given date or timestamp column as an integer (1 =
Sunday, 7 = Saturday).

Parameters:

 col(Column): The column containing the date or timestamp from which the day of the
week is extracted.

Examples:

# Example 1: Extract the day of the week from 'timestamp_column'

day_of_week_column = dayofweek(df.timestamp_column)

# Example 2: Extract the day of the week from a specific date literal
day_of_week_literal = dayofweek(lit("2023-01-01"))

98. dayofyear(col)

Description:
Extract the day of the year from the given date or timestamp column as an integer.
Parameters:

 col (Column): The column containing the date or timestamp from which the day of the
year is extracted.

Examples:

# Example 1: Extract the day of the year from 'timestamp_column'

day_of_year_column = dayofyear(df.timestamp_column)

# Example 2: Extract the day of the year from a specific date literal
day_of_year_literal = dayofyear(lit("2023-01-01"))

99. extract(field, source)

Description:
Extracts a part of the date, timestamp, or interval source based on the field.

Parameters:

 field (String): The part of the date/timestamp to extract (e.g., 'year', 'month', 'day').
 source (Column): The column containing the date/timestamp from which the part is
extracted.

Examples:

# Example 1: Extract the year from 'timestamp_column'

year_column = extract(lit("year"), df.timestamp_column)

# Example 2: Extract the hour from a specific timestamp literal

hour_literal = extract(lit("hour"), lit("2023-01-01 14:30:00"))

100. second(col)

Description:
Extract the seconds from a given date or timestamp column as an integer.

Parameters:

 col (Column): The column containing the date or timestamp from which the seconds
are extracted.

Examples:

# Example 1: Extract the seconds from 'timestamp_column'

seconds_column = second(df.timestamp_column)
# Example 2: Extract the seconds from a specific timestamp literal
seconds_literal = second(lit("2023-01-01 14:30:45"))

101. weekofyear(col)

Description:
Extract the week number of the year from the given date or timestamp column.

Parameters:

 col (Column): The column containing the date or timestamp from which the week of
the year is extracted.

Examples:

# Example 1: Extract the week number from 'timestamp_column'

week_of_year_column = weekofyear(df.timestamp_column)

# Example 2: Extract the week number from a specific date literal

week_of_year_literal = weekofyear(lit("2023-01-01"))

102. year(col)

Description:
Extract the year from the given date or timestamp column.

Parameters:

 col (Column): The column containing the date or timestamp from which the year is
extracted.

Examples:

# Example 1: Extract the year from 'timestamp_column'

year_column = year(df.timestamp_column)

# Example 2: Extract the year from a specific date literal

year_literal = year(lit("2023-01-01"))

103. quarter(col)

Description:
Extract the quarter of the given date or timestamp column as an integer (1 to 4).
Parameters:

 col (Column): The column containing the date or timestamp from which the quarter is
extracted.

Examples:

# Example 1: Extract the quarter from 'timestamp_column'

quarter_column = quarter(df.timestamp_column)

# Example 2: Extract the quarter from a specific date literal

quarter_literal = quarter(lit("2023-01-01"))

104. month(col)

Description:
Extract the month from the given date or timestamp column as an integer (1 = January, 12 =
December).

Parameters:

 col (Column): The column containing the date or timestamp from which the month is
extracted.

Examples:

# Example 1: Extract the month from 'timestamp_column'

month_column = month(df.timestamp_column)

# Example 2: Extract the month from a specific date literal

month_literal = month(lit("2023-01-01"))

105. last_day(date)

Description:
Returns the last day of the month which the given date belongs to.

Parameters:

 date (Column): The column containing the date or timestamp.

Examples:

# Example 1: Get the last day of the month from 'date_column'

last_day_column = last_day(df.date_column)
# Example 2: Get the last day of the month from a specific date literal
last_day_literal = last_day(lit("2023-01-15"))

106. localtimestamp()

Description:
Returns the current timestamp without time zone at the start of query evaluation as a timestamp
without time zone column.

Parameters:
None.

Examples:

# Example 1: Get the current timestamp without time zone

current_local_timestamp_column = localtimestamp()

# Example 2: Use the current timestamp in a conditional statement

is_current = df.timestamp_column == localtimestamp()

107. make_dt_interval([days, hours, mins, secs])

Description:
Makes a DayTimeIntervalType duration from the specified days, hours, minutes, and seconds.

Parameters:

 days (Int): The number of days.

 hours (Int): The number of hours.
 mins (Int): The number of minutes.
 secs (Int): The number of seconds.

Examples:

# Example 1: Create a DayTimeIntervalType from 2 days, 3 hours, 5 minutes, and

30 seconds
interval_column = make_dt_interval(lit(2), lit(3), lit(5), lit(30))

# Example 2: Create a DayTimeIntervalType from 1 day and 2 hours

interval_literal = make_dt_interval(lit(1), lit(2), lit(0), lit(0))

108. make_interval([years, months, weeks, days, …])

Description:
Creates an interval from the specified years, months, weeks, days, hours, minutes, and seconds.
Parameters:

 years (Int): The number of years.

 months (Int): The number of months.
 weeks (Int): The number of weeks.
 days (Int): The number of days.
 hours (Int): The number of hours.
 mins (Int): The number of minutes.
 secs (Int): The number of seconds.

Examples:

# Example 1: Create an interval of 2 years, 3 months, 1 week, and 5 days

interval_column = make_interval(lit(2), lit(3), lit(1), lit(5), lit(0),
lit(0), lit(0))

# Example 2: Create an interval with only months and days

interval_literal = make_interval(lit(0), lit(4), lit(0), lit(10), lit(0),
lit(0), lit(0))

109. make_timestamp(years, months, days, hours, …)

Description:
Creates a timestamp from the specified years, months, days, hours, minutes, seconds, and
timezone fields.

Parameters:

 years (Int): The year component.

 months (Int): The month component.
 days (Int): The day component.
 hours (Int): The hour component.
 mins (Int): The minute component.
 secs (Int): The second component.
 timezone (String): The time zone.

Examples:

# Example 1: Create a timestamp from specific year, month, day, hour, and
minute
timestamp_column = make_timestamp(lit(2023), lit(5), lit(15), lit(10),
lit(30), lit(0), lit("UTC"))

# Example 2: Create a timestamp from date and time with specific timezone
timestamp_literal = make_timestamp(lit(2023), lit(5), lit(15), lit(14),
lit(0), lit(0), lit("America/New_York"))
110. make_timestamp_ltz(years, months, days, …)

Description:
Creates the current timestamp with a local time zone from the specified years, months, days,
hours, minutes, seconds, and timezone fields.

Parameters:

 years (Int): The year component.

 months (Int): The month component.
 days (Int): The day component.
 hours (Int): The hour component.
 mins (Int): The minute component.
 secs (Int): The second component.
 timezone (String): The time zone.

Examples:

# Example 1: Create a timestamp with local time zone from the provided date
and time
local_timestamp_column = make_timestamp_ltz(lit(2023), lit(5), lit(15),
lit(10), lit(30), lit(0), lit("America/Los_Angeles"))

# Example 2: Create a timestamp with local time zone from the provided
timestamp literal
local_timestamp_literal = make_timestamp_ltz(lit(2023), lit(5), lit(15),
lit(14), lit(0), lit(0), lit("Asia/Kolkata"))

111. make_timestamp_ntz(years, months, days, …)

Description:
Creates a local date-time from the specified years, months, days, hours, minutes, and seconds
fields, without time zone.

Parameters:

 years (Int): The year component.

 months (Int): The month component.
 days (Int): The day component.
 hours (Int): The hour component.
 mins (Int): The minute component.
 secs (Int): The second component.

Examples:

# Example 1: Create a local date-time from the provided date and time fields
local_date_time_column = make_timestamp_ntz(lit(2023), lit(5), lit(15),
lit(10), lit(30), lit(0))

# Example 2: Create a local date-time from a specific date and time

local_date_time_literal = make_timestamp_ntz(lit(2023), lit(5), lit(15),
lit(14), lit(0), lit(0))

112. make_ym_interval([years, months])

Description:
Makes a year-month interval from the specified years and months.

Parameters:

 years (Int): The number of years.

 months (Int): The number of months.

Examples:

# Example 1: Create a year-month interval from 2 years and 3 months

ym_interval_column = make_ym_interval(lit(2), lit(3))

# Example 2: Create a year-month interval from 0 years and 5 months

ym_interval_literal = make_ym_interval(lit(0), lit(5))

113. minute(col)

Description:
Extract the minutes from the given timestamp column as an integer.

Parameters:

 col (Column): The column containing the timestamp from which the minutes are
extracted.

Examples:

# Example 1: Extract the minutes from 'timestamp_column'

minute_column = minute(df.timestamp_column)

# Example 2: Extract the minutes from a specific timestamp literal

minute_literal = minute(lit("2023-01-01 14:30:45"))

114. months_between(date1, date2[, roundOff])

Description:
Returns the number of months between the two given dates (date1 and date2). Optionally, you
can round off the result.

Parameters:

 date1 (Column): The first date or timestamp column.

 date2 (Column): The second date or timestamp column.
 roundOff (Optional, Boolean): Whether to round off the result. Default is False.

Examples:

# Example 1: Calculate the number of months between two dates

months_between_column = months_between(df.date1, df.date2)

# Example 2: Calculate the number of months between two specific dates

months_between_literal = months_between(lit("2023-01-01"), lit("2024-01-01"))

115. next_day(date, dayOfWeek)

Description:
Returns the first date that is later than the value of the date column based on the specified
dayOfWeek argument (e.g., 'Monday', 'Tuesday').

Parameters:

 date (Column): The column containing the date.

 dayOfWeek (String): The day of the week to find, such as 'Monday', 'Tuesday', etc.

Examples:

# Example 1: Get the first date after the given date that falls on a Monday
next_monday_column = next_day(df.date_column, lit("Monday"))

# Example 2: Get the first date after the given date that falls on a Sunday
next_sunday_literal = next_day(lit("2023-01-01"), lit("Sunday"))

116. hour(col)

Description:
Extract the hour from the given timestamp column as an integer.

Parameters:

 col (Column): The column containing the timestamp from which the hour is extracted.
Examples:

# Example 1: Extract the hour from 'timestamp_column'

hour_column = hour(df.timestamp_column)

# Example 2: Extract the hour from a specific timestamp literal

hour_literal = hour(lit("2023-01-01 14:30:00"))

117. make_date(year, month, day)

Description:
Returns a column with a date built from the specified year, month, and day columns.

Parameters:

 year (Int): The year component.

 month (Int): The month component.
 day (Int): The day component.

Examples:

# Example 1: Create a date from specific year, month, and day

date_column = make_date(lit(2023), lit(5), lit(15))

# Example 2: Create a date from specific year, month, and day literals
date_literal = make_date(lit(2023), lit(1), lit(1))

118. now()

Description:
Returns the current timestamp at the start of query evaluation.

Parameters:
None.

Examples:

# Example 1: Get the current timestamp

current_timestamp_column = now()

# Example 2: Use the current timestamp in a conditional statement

is_current = df.timestamp_column == now()

119. from_unixtime(timestamp[, format])

Description:
Converts the number of seconds from the Unix epoch (1970-01-01 00:00:00 UTC) to a string
representing the timestamp of that moment in the current system time zone in the given format.

Parameters:

 timestamp (Column): The timestamp in seconds since the Unix epoch.

 format (Optional, String): The format of the resulting timestamp string (default is 'yyyy-
MM-dd HH:mm:ss').

Examples:

# Example 1: Convert a timestamp to a formatted date string

formatted_timestamp_column = from_unixtime(df.timestamp_column, "yyyy-MM-dd
HH:mm:ss")

# Example 2: Convert a Unix timestamp to string in a default format

formatted_timestamp_literal = from_unixtime(lit(1609459200))

120. unix_timestamp([timestamp, format])

Description:
Converts a time string with a given pattern (yyyy-MM-dd HH:mm:ss by default) to a Unix
timestamp (in seconds), using the default timezone and locale. Returns null if conversion fails.

Parameters:

 timestamp (Column or String, Optional): The time string.

 format (Optional, String): The format of the time string (default is yyyy-MM-dd
HH:mm:ss).

Examples:

# Example 1: Convert a time string to Unix timestamp

unix_timestamp_column = unix_timestamp(df.time_column, "yyyy-MM-dd HH:mm:ss")

# Example 2: Convert a time string literal to Unix timestamp

unix_timestamp_literal = unix_timestamp(lit("2023-01-01 12:00:00"))

121. to_unix_timestamp(timestamp[, format])

Description:
Returns the Unix timestamp (in seconds) for a given timestamp, with an optional format.

Parameters:
  timestamp (Column): The column with the timestamp.
 format (Optional, String): The format of the timestamp string.

Examples:

# Example 1: Convert a timestamp column to Unix timestamp

unix_timestamp_col = to_unix_timestamp(df.timestamp_column)

# Example 2: Convert a timestamp string to Unix timestamp with specific format

unix_timestamp_literal = to_unix_timestamp(lit("2023-01-01 12:00:00"), "yyyy-
MM-dd HH:mm:ss")

122. to_timestamp(col[, format])

Description:
Converts a column into a TimestampType using the optionally specified format.

Parameters:

 col (Column): The column to be converted to a timestamp.

 format (Optional, String): The format of the timestamp string.

Examples:

# Example 1: Convert a column into a timestamp

timestamp_column = to_timestamp(df.date_column)

# Example 2: Convert a column with a custom format to a timestamp

timestamp_column_custom = to_timestamp(df.date_column, "yyyy-MM-dd HH:mm:ss")

123. to_timestamp_ltz(timestamp[, format])

Description:
Parses the timestamp with the format to a TimestampType without the time zone.

Parameters:

 timestamp (Column): The column containing the timestamp.

 format (Optional, String): The format of the timestamp string.

Examples:

# Example 1: Convert timestamp to timestamp without time zone

timestamp_ltz_column = to_timestamp_ltz(df.timestamp_column)

# Example 2: Convert timestamp string to timestamp without time zone

timestamp_ltz_literal = to_timestamp_ltz(lit("2023-01-01 12:00:00"), "yyyy-MM-
dd HH:mm:ss")

124. to_timestamp_ntz(timestamp[, format])

Description:
Parses the timestamp with the format to a timestamp without time zone.

Parameters:

 timestamp (Column): The column containing the timestamp.

 format (Optional, String): The format of the timestamp string.

Examples:

# Example 1: Convert timestamp to timestamp without time zone

timestamp_ntz_column = to_timestamp_ntz(df.timestamp_column)

# Example 2: Convert timestamp string to timestamp without time zone

timestamp_ntz_literal = to_timestamp_ntz(lit("2023-01-01 12:00:00"), "yyyy-MM-
dd HH:mm:ss")

125. to_date(col[, format])

Description:
Converts a column into DateType using the optionally specified format.

Parameters:

 col (Column): The column to be converted to a date.

 format (Optional, String): The format of the date string.

Examples:

# Example 1: Convert a column into a date

date_column = to_date(df.date_column)

# Example 2: Convert a column with a specific format into a date

date_column_custom = to_date(df.date_column, "yyyy-MM-dd")

126. trunc(date, format)

Description:
Returns the date truncated to the unit specified by the format.
Parameters:

 date (Column): The column containing the date.

 format (String): The format to truncate the date (e.g., "YYYY", "MM", "DD").

Examples:

# Example 1: Truncate a date to the year

truncated_date_year = trunc(df.date_column, "YYYY")

# Example 2: Truncate a date to the month

truncated_date_month = trunc(df.date_column, "MM")

127. from_utc_timestamp(timestamp, tz)

Description:
Converts a UTC timestamp to a given time zone.

Parameters:

 timestamp (Column): The column containing the timestamp in UTC.

 tz (String): The time zone to which the timestamp should be converted.

Examples:

# Example 1: Convert UTC timestamp to Eastern Time

timestamp_et = from_utc_timestamp(df.utc_timestamp_column, "America/New_York")

# Example 2: Convert UTC timestamp to Pacific Time

timestamp_pt = from_utc_timestamp(df.utc_timestamp_column,
"America/Los_Angeles")

128. to_utc_timestamp(timestamp, tz)

Description:
Converts a timestamp from a given time zone to UTC.

Parameters:

 timestamp (Column): The column containing the timestamp.

 tz (String): The time zone from which the timestamp should be converted.

Examples:

# Example 1: Convert timestamp from Eastern Time to UTC

timestamp_utc = to_utc_timestamp(df.timestamp_column, "America/New_York")
# Example 2: Convert timestamp from Pacific Time to UTC
timestamp_utc_pt = to_utc_timestamp(df.timestamp_column,
"America/Los_Angeles")

129. weekday(col)

Description:
Returns the day of the week for a date/timestamp (0 = Monday, 1 = Tuesday, …, 6 = Sunday).

Parameters:

 col (Column): The column containing the date or timestamp.

Examples:

# Example 1: Get the weekday from the date column

weekday_column = weekday(df.date_column)

# Example 2: Get the weekday from a specific date literal

weekday_literal = weekday(lit("2023-01-01"))

130. window(timeColumn, windowDuration[, …])

Description:
Bucketizes rows into one or more time windows, given a timestamp specifying column.

Parameters:

 timeColumn (Column): The column containing the timestamp.

 windowDuration (String): The duration of the time window (e.g., "1 hour", "30
minutes").
 slideDuration (Optional, String): The sliding window duration.

Examples:

# Example 1: Create time windows based on timestamp column

windowed_column = window(df.timestamp_column, "1 hour")

# Example 2: Create sliding time windows with a slide duration

sliding_window_column = window(df.timestamp_column, "1 hour", "15 minutes")

131. session_window(timeColumn, gapDuration)

Description:
Generates a session window given a timestamp specifying column. A session window defines a
session gap between events, where events that occur within the gap are grouped together.

Parameters:

 timeColumn (Column): The column containing the timestamp.

 gapDuration (String): The duration of the session gap (e.g., "10 minutes", "30 seconds").

Examples:

# Example 1: Create a session window based on timestamp column with a 10-

minute gap
session_window_column = session_window(df.timestamp_column, "10 minutes")

# Example 2: Create a session window with a 15-second gap

session_window_15_sec = session_window(df.timestamp_column, "15 seconds")

132. timestamp_micros(col)

Description:
Creates a timestamp from the number of microseconds since the UTC epoch (1970-01-01
00:00:00 UTC).

Parameters:

 col (Column): The column containing the number of microseconds.

Examples:

# Example 1: Convert microseconds column to timestamp

timestamp_from_micros = timestamp_micros(df.microseconds_column)

# Example 2: Convert a literal number of microseconds to timestamp

timestamp_from_micros_literal = timestamp_micros(lit(1610100000000000))

133. timestamp_millis(col)

Description:
Creates a timestamp from the number of milliseconds since the UTC epoch (1970-01-01
00:00:00 UTC).

Parameters:

 col (Column): The column containing the number of milliseconds.

Examples:

# Example 1: Convert milliseconds column to timestamp

timestamp_from_millis = timestamp_millis(df.milliseconds_column)

# Example 2: Convert a literal number of milliseconds to timestamp

timestamp_from_millis_literal = timestamp_millis(lit(1610100000000))

134. timestamp_seconds(col)

Description:
Converts the number of seconds from the Unix epoch (1970-01-01 00:00:00 UTC) to a
timestamp.

Parameters:

 col (Column): The column containing the number of seconds.

Examples:

# Example 1: Convert seconds column to timestamp

timestamp_from_seconds = timestamp_seconds(df.seconds_column)

# Example 2: Convert a literal number of seconds to timestamp

timestamp_from_seconds_literal = timestamp_seconds(lit(1609459200)) # Unix
timestamp for 2021-01-01

135. try_to_timestamp(col[, format])

Description:
Parses the column with the specified format to a timestamp, but if the conversion fails, it returns
null instead of throwing an error.

Parameters:

 col (Column): The column to be parsed to a timestamp.

 format (Optional, String): The format of the timestamp string.

Examples:

# Example 1: Try to parse a column as a timestamp

timestamp_try_parse = try_to_timestamp(df.date_column, "yyyy-MM-dd HH:mm:ss")

# Example 2: Try to parse a timestamp string literal

timestamp_try_parse_literal = try_to_timestamp(lit("2023-01-01 12:00:00"),
"yyyy-MM-dd HH:mm:ss")
136. unix_date(col)

Description:
Returns the number of days since the Unix epoch (1970-01-01) for a given date or timestamp
column.

Parameters:

 col (Column): The column containing the date or timestamp.

Examples:

# Example 1: Convert a date column to number of days since Unix epoch

days_since_unix_epoch = unix_date(df.date_column)

# Example 2: Convert a timestamp column to number of days since Unix epoch

days_since_unix_epoch_ts = unix_date(df.timestamp_column)

137. unix_micros(col)

Description:
Returns the number of microseconds since the Unix epoch (1970-01-01 00:00:00 UTC) for a
given timestamp column.

Parameters:

 col (Column): The column containing the timestamp.

Examples:

# Example 1: Convert a timestamp column to microseconds since Unix epoch

microseconds_since_unix = unix_micros(df.timestamp_column)

# Example 2: Convert a literal timestamp to microseconds since Unix epoch

microseconds_since_unix_literal = unix_micros(lit("2023-01-01 12:00:00"))

138. unix_millis(col)

Description:
Returns the number of milliseconds since the Unix epoch (1970-01-01 00:00:00 UTC) for a
given timestamp column.

Parameters:

 col (Column): The column containing the timestamp.

Examples:

# Example 1: Convert a timestamp column to milliseconds since Unix epoch

milliseconds_since_unix = unix_millis(df.timestamp_column)

# Example 2: Convert a literal timestamp to milliseconds since Unix epoch

milliseconds_since_unix_literal = unix_millis(lit("2023-01-01 12:00:00"))

139. unix_seconds(col)

Description:
Returns the number of seconds since the Unix epoch (1970-01-01 00:00:00 UTC) for a given
timestamp column.

Parameters:

 col (Column): The column containing the timestamp.

Examples:

# Example 1: Convert a timestamp column to seconds since Unix epoch

seconds_since_unix = unix_seconds(df.timestamp_column)

# Example 2: Convert a literal timestamp to seconds since Unix epoch

seconds_since_unix_literal = unix_seconds(lit("2023-01-01 12:00:00"))

140. window_time(windowColumn)

Description:
Computes the event time from a window column.

Parameters:

 windowColumn (Column): The column containing the window information.

Examples:

# Example 1: Extract event time from a window column

event_time_from_window = window_time(df.window_column)

141. array(*cols)

Description:
Creates a new array column from the provided columns.
Parameters:

 *cols (Columns): Columns to be included in the array.

Examples:

# Example 1: Create an array column from existing columns

array_column = array(df.col1, df.col2, df.col3)

# Example 2: Create an array from literal values

array_literal = array(lit(1), lit(2), lit(3))

142. array_contains(col, value)

Description:
Returns true if the given array column contains the specified value, otherwise returns false. If the
array is null, returns null.

Parameters:

 col (Column): The column containing the array.

 value (Any): The value to check within the array.

Examples:

# Example 1: Check if the array column contains the value

contains_value = array_contains(df.array_column, lit(5))

# Example 2: Check if the array contains a string value

contains_value_str = array_contains(df.array_column, lit("apple"))

143. arrays_overlap(a1, a2)

Description:
Returns true if the arrays a1 and a2 contain any common non-null element; if not, returns null if
both arrays are non-empty and any of them contains a null element; returns false otherwise.

Parameters:

 a1 (Column): The first array column.

 a2 (Column): The second array column.

Examples:

# Example 1: Check if two arrays contain any common elements

overlap_result = arrays_overlap(df.array_column1, df.array_column2)

# Example 2: Check overlap with literal arrays

overlap_literal = arrays_overlap(array(lit(1), lit(2), lit(3)), array(lit(3),
lit(4), lit(5)))

144. array_join(col, delimiter[, null_replacement])

Description:
Concatenates the elements of the array column col using the specified delimiter. If
null_replacement is provided, null values will be replaced with the specified string.

Parameters:

 col (Column): The array column to be concatenated.

 delimiter (String): The string to separate array elements.
 null_replacement (Optional, String): The string to replace null values in the array.

Examples:

# Example 1: Join array elements with a comma delimiter

joined_array = array_join(df.array_column, ", ")

# Example 2: Join array elements, replacing null values with "NULL"

joined_array_with_nulls = array_join(df.array_column, ", ", "NULL")

145. create_map(*cols)

Description:
Creates a new map column from key-value pairs. Each key and value is passed as alternating
arguments.

Parameters:

 *cols (Column): Alternating columns of keys and values.

Examples:

# Example 1: Create a map column with string keys and integer values
map_column = create_map(lit("key1"), lit(1), lit("key2"), lit(2))

# Example 2: Create a map from two columns

map_column_from_cols = create_map(df.key_column, df.value_column)

146. slice(x, start, length)

Description:
Returns a new array containing all the elements in the array x starting from index start with the
specified length. Array indices start at 1, and negative indices refer to positions from the end of
the array.

Parameters:

 x (Column): The array column.

 start (Integer): The starting index (1-based).
 length (Integer): The number of elements to return.

Examples:

# Example 1: Slice the first 3 elements of the array column

array_slice = slice(df.array_column, 1, 3)

# Example 2: Slice last 2 elements of the array (using negative indexing)

array_slice_negative = slice(df.array_column, -2, 2)

147. concat(*cols)

Description:
Concatenates multiple input columns into a single column. It can handle various data types, such
as strings, arrays, etc.

Parameters:

 *cols (Columns): Columns to concatenate.

Examples:

# Example 1: Concatenate two string columns

concatenated_strings = concat(df.col1, df.col2)

# Example 2: Concatenate three columns with arrays

concatenated_arrays = concat(df.array1, df.array2, df.array3)

148. array_position(col, value)

Description:
Returns the position of the first occurrence of the specified value in the array column col. If the
value does not exist, returns null.

Parameters:

 col (Column): The array column.

  value (Any): The value to find in the array.

Examples:

# Example 1: Find the position of value 3 in the array column

position = array_position(df.array_column, lit(3))

# Example 2: Find the position of a string "apple" in the array

position_str = array_position(df.array_column, lit("apple"))

149. element_at(col, extraction)

Description:
Returns the element at the specified index extraction in the array column col. The index is 1-
based, and if the index is out of bounds, it returns null.

Parameters:

 col (Column): The array column.

 extraction (Integer): The 1-based index of the element to extract.

Examples:

# Example 1: Extract the first element of the array column

first_element = element_at(df.array_column, 1)

# Example 2: Extract the third element of the array column

third_element = element_at(df.array_column, 3)

150. array_append(col, value)

Description:
Returns an array that is the result of appending the given value to the end of the array column
col.

Parameters:

 col (Column): The array column.

 value (Any): The value to append to the array.

Examples:

# Example 1: Append value 5 to the array column

array_with_append = array_append(df.array_column, lit(5))

# Example 2: Append string "apple" to the array column

array_with_append_str = array_append(df.array_column, lit("apple"))

151. array_size(col)

Description:
Returns the total number of elements in the array column col.

Parameters:

 col (Column): The array column.

Examples:

# Example 1: Get the size of the array column

array_size_result = array_size(df.array_column)

# Example 2: Get the size of an array with literal values

array_size_literal = array_size(array(lit(1), lit(2), lit(3)))

152. array_sort(col[, comparator])

Description:
Sorts the elements of the array column col in ascending order. An optional comparator can be
provided to customize the sorting behavior.

Parameters:

 col (Column): The array column.

 comparator (Optional, Column): A custom comparator to sort the array elements.

Examples:

# Example 1: Sort the elements of the array column in ascending order

sorted_array = array_sort(df.array_column)

# Example 2: Sort the array using a custom comparator (descending order)

sorted_array_desc = array_sort(df.array_column, desc(df.array_column))

153. array_insert(arr, pos, value)

Description:
Adds an item into the specified array arr at the index position pos.

Parameters:
  arr (Column): The array column.
 pos (Integer): The index position to insert the value at.
 value (Any): The value to insert into the array.

Examples:

# Example 1: Insert value 10 at the 2nd position of the array

array_with_insert = array_insert(df.array_column, 2, lit(10))

# Example 2: Insert "banana" at the 1st position of the array

array_with_insert_str = array_insert(df.array_column, 1, lit("banana"))

154. array_remove(col, element)

Description:
Removes all occurrences of the specified element from the given array column.

Parameters:

 col (Column): The array column to operate on.

 element (Any): The element to remove from the array.

Example:

# Remove element '5' from the array column

[Link](array_remove(df.array_column, lit(5)))

155. array_prepend(col, value)

Description:
Adds the specified value to the beginning of the array in col.

Parameters:

 col (Column): The array column.

 value (Any): The value to prepend.

Example:

# Prepend value '1' to the array column

[Link](array_prepend(df.array_column, lit(1)))

156. array_distinct(col)
Description:
Removes duplicate values from the array column col.

Parameters:

 col (Column): The array column.

Example:

# Remove duplicates from the array column

[Link](array_distinct(df.array_column))

157. array_intersect(col1, col2)

Description:
Returns an array containing the elements that are common to both col1 and col2 arrays, without
duplicates.

Parameters:

 col1 (Column): The first array column.

 col2 (Column): The second array column.

Example:

# Find the intersection between two array columns

[Link](array_intersect(df.array_column1, df.array_column2))

158. array_union(col1, col2)

Description:
Returns an array of the elements that are in the union of col1 and col2 arrays, without
duplicates.

Parameters:

 col1 (Column): The first array column.

 col2 (Column): The second array column.

Example:

# Find the union of two array columns

[Link](array_union(df.array_column1, df.array_column2))
159. array_except(col1, col2)

Description:
Returns an array of the elements that are in col1 but not in col2, without duplicates.

Parameters:

 col1 (Column): The first array column.

 col2 (Column): The second array column.

Example:

# Find the elements in the first array but not in the second array
[Link](array_except(df.array_column1, df.array_column2))

160. array_compact(col)

Description:
Removes null values from the array column col.

Parameters:

 col (Column): The array column.

Example:

# Remove null values from the array column

[Link](array_compact(df.array_column))

161. transform(col, f)

Description:
Applies a transformation function f to each element in the array col and returns a new array
with the transformed elements.

Parameters:

 col (Column): The array column.

 f (Function): The function to apply to each element in the array.

Example:

# Multiply each element in the array by 2

[Link](transform(df.array_column, lambda x: x * 2))
162. exists(col, f)

Description:
Checks if the predicate f holds for one or more elements in the array col. Returns true if at least
one element satisfies the predicate.

Parameters:

 col (Column): The array column.

 f (Function): The predicate function to check for each element.

Example:

# Check if the array contains any element greater than 10

[Link](exists(df.array_column, lambda x: x > 10))

163. forall(col, f)

Description:
Checks if the predicate f holds for every element in the array col. Returns true if all elements
satisfy the predicate.

Parameters:

 col (Column): The array column.

 f (Function): The predicate function to apply to each element.

Example:

# Check if all elements in the array are greater than 10

[Link](forall(df.array_column, lambda x: x > 10))

164. filter(col, f)

Description:
Returns an array of elements from the array col where the predicate function f holds.

Parameters:

 col (Column): The array column.

 f (Function): The predicate function to apply to each element.

Example:
# Filter the array to keep only elements greater than 10
[Link](filter(df.array_column, lambda x: x > 10))

165. aggregate(col, initialValue, merge[, finish])

Description:
Applies a binary operator to an initial state and all elements in the array, and reduces this to a
single result.

Parameters:

 col (Column): The array column.

 initialValue (Any): The initial state for the aggregation.
 merge (Function): The binary operator that merges the state with each element.
 finish (Optional, Function): The function to apply after all elements are merged.

Example:

# Sum up all elements in the array

[Link](aggregate(df.array_column, lit(0), lambda acc, x: acc + x))

166. zip_with(left, right, f)

Description:
Merges two arrays, left and right, element-wise into a single array using the function f.

Parameters:

 left (Column): The first array column.

 right (Column): The second array column.
 f (Function): The function to apply to each pair of elements from both arrays.

Example:

# Combine two arrays element-wise by adding corresponding elements

[Link](zip_with(df.array_column1, df.array_column2, lambda x, y: x + y))

167. transform_keys(col, f)

Description:
Applies a function f to every key in a map column col and returns a new map with the results as
the new keys.

Parameters:
  col (Column): The map column.
 f (Function): The function to apply to each key.

Example:

# Transform the keys of a map by converting them to uppercase

[Link](transform_keys(df.map_column, lambda k: [Link]()))

168. transform_values(col, f)

Description:
Applies a function f to every value in a map column col and returns a new map with the results
as the new values.

Parameters:

 col (Column): The map column.

 f (Function): The function to apply to each value.

Example:

# Transform the values of a map by multiplying each value by 2

[Link](transform_values(df.map_column, lambda v: v * 2))

169. map_filter(col, f)

Description:
Returns a map containing key-value pairs that satisfy the predicate function f.

Parameters:

 col (Column): The map column.

 f (Function): The predicate function to apply to each key-value pair.

Example:

# Filter a map to keep only key-value pairs where the value is greater than 10
[Link](map_filter(df.map_column, lambda k, v: v > 10))

170. map_from_arrays(col1, col2)

Description:
Creates a new map from two array columns, one for keys and one for values.
Parameters:

 col1 (Column): The array column of keys.

 col2 (Column): The array column of values.

Example:

# Create a map from two array columns

[Link](map_from_arrays(df.keys_array, df.values_array))

171. map_zip_with(col1, col2, f)

Description:
Merges two maps, col1 and col2, key-wise into a single map using a function f.

Parameters:

 col1 (Column): The first map column.

 col2 (Column): The second map column.
 f (Function): A function to combine key-value pairs from both maps.

Example:

# Merge two maps by adding corresponding values

[Link](map_zip_with(df.map_column1, df.map_column2, lambda k, v1, v2: v1 +
v2))

172. explode(col)

Description:
Returns a new row for each element in the given array or map col.

Parameters:

 col (Column): The array or map column.

Example:

# Explode the array column into multiple rows

[Link](explode(df.array_column))

173. explode_outer(col)
Description:
Returns a new row for each element in the given array or map column, including null values. If
an array or map is null, it will return a row with null values.

Parameters:

 col (Column): The array or map column.

Example:

# Explode the array column, including nulls

[Link](explode_outer(df.array_column))

174. posexplode(col)

Description:
Returns a new row for each element in the given array or map, but also includes the position of
the element within the array or map.

Parameters:

 col (Column): The array or map column.

Example:

# Explode the array column with the position of each element

[Link](posexplode(df.array_column))

175. posexplode_outer(col)

Description:
Returns a new row for each element with the position in the given array or map, including null
values. If an array or map is null, it will return a row with null values and the position null.

Parameters:

 col (Column): The array or map column.

Example:

# Explode the array column with positions, including nulls

[Link](posexplode_outer(df.array_column))

176. inline(col)
Description:
Explodes an array of structs into a table, where each element in the array becomes a row in the
table.

Parameters:

 col (Column): The array of structs column.

Example:

# Explode an array of structs into a table

[Link](inline(df.array_of_structs))

177. inline_outer(col)

Description:
Similar to inline(col), this function explodes an array of structs into a table, but it also
includes null values from the array.

Parameters:

 col (Column): The array of structs column.

Example:

# Explode an array of structs into a table, including nulls

[Link](inline_outer(df.array_of_structs))

178. get(col, index)

Description:
Returns the element at the specified index from the given array col. The index is 0-based.

Parameters:

 col (Column): The array column.

 index (Int): The 0-based index of the element to retrieve.

Example:

# Get the element at index 2 from the array column

[Link](get(df.array_column, lit(2)))

179. get_json_object(col, path)

Description:
Extracts a JSON object from a JSON string column col based on the specified JSON path path,
and returns the extracted JSON object as a string.

Parameters:

 col (Column): The column containing the JSON string.

 path (String): The JSON path to extract the object from.

Example:

# Extract the "name" field from the JSON column

[Link](get_json_object(df.json_column, '$.name'))

180. json_tuple(col, *fields)

Description:
Creates a new row for a JSON column by extracting specific fields (*fields) from the JSON
object in col.

Parameters:

 col (Column): The column containing the JSON string.

 *fields (String): The field names to extract from the JSON object.

Example:

# Extract the "name" and "age" fields from the JSON column
[Link](json_tuple(df.json_column, 'name', 'age'))

181. from_json(col, schema[, options])

Description:
Parses a JSON string in col into a MapType with StringType as keys, or a StructType or
ArrayType based on the provided schema.

Parameters:

 col (Column): The column containing the JSON string.

 schema (StructType or ArrayType): The schema to use for parsing the JSON.
 options (Optional): JSON options like dateFormat and timestampFormat.

Example:

# Parse the JSON string into a struct with the provided schema
[Link](from_json(df.json_column, schema))

182. schema_of_json(json[, options])

Description:
Infers the schema of a JSON string in json and returns it in DDL format.

Parameters:

 json (String): The JSON string to infer the schema from.

 options (Optional): JSON options.

Example:

# Get the schema of a JSON string

[Link](schema_of_json(df.json_column))

183. to_json(col[, options])

Description:
Converts a column containing a StructType, ArrayType, or MapType into a JSON string.

Parameters:

 col (Column): The column to convert into a JSON string.

 options (Optional): JSON options like dateFormat and timestampFormat.

Example:

# Convert the struct column into a JSON string

[Link](to_json(df.struct_column))

184. json_array_length(col)

Description:
Returns the number of elements in the outermost JSON array in col.

Parameters:

 col (Column): The column containing the JSON array.

Example:

# Get the number of elements in the JSON array

[Link](json_array_length(df.json_column))

185. json_object_keys(col)

Description:
Returns an array of all the keys in the outermost JSON object in col.

Parameters:

 col (Column): The column containing the JSON object.

Example:

# Get the keys of the JSON object

[Link](json_object_keys(df.json_column))

186. size(col)

Description:
Returns the length (number of elements) of the array or map stored in the column col.

Parameters:

 col (Column): The array or map column.

Example:

# Get the length of the array column

[Link](size(df.array_column))

187. cardinality(col)

Description:
Returns the length (number of elements) of the array or map stored in col.

Parameters:

 col (Column): The array or map column.

Example:

# Get the cardinality (size) of the map column

[Link](cardinality(df.map_column))
188. struct(*cols)

Description:
Creates a new struct column by combining the specified columns into a single struct.

Parameters:

 *cols (Column): The columns to combine into a struct.

Example:

# Create a new struct from multiple columns

[Link](struct(df.col1, df.col2).alias("new_struct"))

189. sort_array(col[, asc])

Description:
Sorts the input array col in ascending or descending order based on the natural ordering of the
array elements. By default, it sorts in ascending order.

Parameters:

 col (Column): The array column.

 asc (Boolean, optional): If True (default), the array is sorted in ascending order;
otherwise, it is sorted in descending order.

Example:

# Sort the array column in descending order

[Link](sort_array(df.array_column, lit(False)))

190. array_max(col)

Description:
Returns the maximum value from the array column col.

Parameters:

 col (Column): The array column.

Example:

# Get the maximum value from the array column

[Link](array_max(df.array_column))
191. array_min(col)

Description:
Returns the minimum value from the array column col.

Parameters:

 col (Column): The array column.

Example:

# Get the minimum value from the array column

[Link](array_min(df.array_column))

192. shuffle(col)

Description:
Generates a random permutation of the given array column col.

Parameters:

 col (Column): The array column.

Example:

# Shuffle the array column randomly

[Link](shuffle(df.array_column))

193. reverse(col)

Description:
Reverses the elements in the array or string column col.

Parameters:

 col (Column): The array or string column.

Example:

# Reverse the elements in the array column

[Link](reverse(df.array_column))

194. flatten(col)
Description:
Creates a single array from an array of arrays. This is useful when you have nested arrays and
want to convert them into a single flat array.

Parameters:

 col (Column): The array of arrays column.

Example:

# Flatten an array of arrays into a single array

[Link](flatten(df.array_of_arrays))

195. sequence(start, stop[, step])

Description:
Generates a sequence of integers from start to stop, incrementing by step. The step is
optional and defaults to 1.

Parameters:

 start (Int): The starting value of the sequence.

 stop (Int): The end value of the sequence.
 step (Int, optional): The step size, default is 1.

Example:

# Generate a sequence from 1 to 10 with step size of 2

[Link](sequence(lit(1), lit(10), lit(2)))

196. array_repeat(col, count)

Description:
Creates an array where the elements from col are repeated count times.

Parameters:

 col (Column): The column containing the element to repeat.

 count (Int): The number of times the element should be repeated.

Example:

# Repeat the array column 3 times

[Link](array_repeat(df.array_column, lit(3)))
197. map_contains_key(col, value)

Description:
Returns True if the map contains the specified key value. If the key is found, it returns True,
otherwise False.

Parameters:

 col (Column): The map column.

 value (Any): The key to check for in the map.

Example:

# Check if the map column contains a specific key

[Link](map_contains_key(df.map_column, lit("key_name")))

198. map_keys(col)

Description:
Returns an unordered array containing the keys of the map in col.

Parameters:

 col (Column): The map column.

Example:

# Get the keys of the map column

[Link](map_keys(df.map_column))

199. map_values(col)

Description:
Returns an unordered array containing the values of the map in col.

Parameters:

 col (Column): The map column.

Example:

# Get the values of the map column

[Link](map_values(df.map_column))
200. map_entries(col)

Description:
Returns an unordered array of all entries (key-value pairs) in the map col. The array consists of
structs, where each struct contains a key and value.

Parameters:

 col (Column): The map column.

Example:

# Get all key-value entries in the map column

[Link](map_entries(df.map_column))

201. map_from_entries(col)

Description:
Converts an array of entries (key-value struct types) into a map. The col is expected to contain
an array of structs, each having a key and a value.

Parameters:

 col (Column): The array of key-value pair structs column.

Example:

# Convert an array of key-value structs into a map

[Link](map_from_entries(df.array_of_key_value_structs))

202. arrays_zip(*cols)

Description:
Returns a merged array of structs, where each struct contains the N-th values from the input
arrays. If the input arrays have different lengths, the result will have the length of the shortest
array.

Parameters:

 *cols (Columns): The arrays to zip together.

Example:

# Zip together two arrays into an array of structs

[Link](arrays_zip(df.array1, df.array2))
203. map_concat(*cols)

Description:
Returns the union of all the maps from the provided columns. If multiple maps have the same
key, the last value encountered for that key is kept.

Parameters:

 *cols (Columns): The map columns to concatenate.

Example:

# Concatenate two map columns into one

[Link](map_concat(df.map1, df.map2))

204. from_csv(col, schema[, options])

Description:
Parses a column containing a CSV string and converts it to a row with the specified schema. It
can handle CSV formatted strings with custom delimiters and other options.

Parameters:

 col (Column): The column containing the CSV string.

 schema (StructType): The schema to use for parsing the CSV.
 options (Optional): Additional options, such as delimiter and quote.

Example:

# Parse a CSV string with a custom schema

[Link](from_csv(df.csv_column, schema))

205. schema_of_csv(csv[, options])

Description:
Parses a CSV string and infers its schema in DDL (Data Definition Language) format. This can
be used when the schema is not known in advance.

Parameters:

 csv (String): The CSV string.

 options (Optional): Options such as delimiter and quote.
Example:

# Infer the schema from a CSV string

[Link](schema_of_csv(df.csv_column))

206. str_to_map(text[, pairDelim, keyValueDelim])

Description:
Converts a string into a map by splitting the text into key-value pairs using delimiters.
pairDelim separates the key-value pairs, and keyValueDelim separates the key and value within
each pair.

Parameters:

 text (String): The text containing key-value pairs.

 pairDelim (String, optional): The delimiter that separates key-value pairs (default is ,).
 keyValueDelim (String, optional): The delimiter that separates keys and values (default
is =).

Example:

# Convert a string into a map

[Link](str_to_map(df.text_column, lit(','), lit('=')))

207. to_csv(col[, options])

Description:
Converts a column containing a StructType into a CSV string. You can specify options like
delimiter and quote character.

Parameters:

 col (Column): The column to convert to CSV.

 options (Optional): Options such as delimiter and quote.

Example:

# Convert a struct column into a CSV string

[Link](to_csv(df.struct_column))

208. try_element_at(col, extraction)

Description:
Returns the element of the array at the given 1-based index extraction. If the index is out of
bounds, it returns null.

Parameters:

 col (Column): The array column.

 extraction (Int): The 1-based index to extract the element.

Example:

# Safely extract the element at the 3rd position of an array

[Link](try_element_at(df.array_column, lit(3)))