Function Application

Pandas - Applying Custom Functions

Pandas allows you to apply custom functions or functions from other libraries to its objects (Series, DataFrame). Understanding how to apply functions effectively is key to leveraging Pandas for complex data manipulation. There are three main methods for function application:

• DataFrame-wise: pipe() - For chaining functions that expect a DataFrame or Series.
• Row/Column-wise: apply() - For applying a function along an axis (rows or columns).
• Element-wise: applymap() (DataFrame) / map() (Series) - For applying a Python function to each element individually.

Performance Note: Vectorized operations (using built-in Pandas/NumPy functions directly on Series/DataFrames) are almost always faster than applying custom Python functions using these methods. Use apply, applymap, or map when vectorized solutions are not available or are overly complex.

DataFrame-wise Application: .pipe()

The .pipe() method allows you to chain custom functions in a more readable way, especially functions that expect a DataFrame or Series as the first argument. It passes the DataFrame (or Series) it’s called on as the first argument to the provided function, along with any other specified arguments.

Consider a scenario where you have a function that processes a DataFrame, perhaps adding a constant value.

Example: adder function via pipe()

Let’s define a simple function add_value that adds a specified number to all elements of a DataFrame.

import pandas as pd
import numpy as np

def add_value(input_df, value_to_add):
    """Adds a constant value to all elements of a DataFrame."""
    return input_df + value_to_add

rng = np.random.default_rng(42)
df = pd.DataFrame(rng.standard_normal((5, 3)), columns=['col1', 'col2', 'col3'])

print("Original DataFrame:")
print(df)

# Apply the function using pipe
df_added = df.pipe(add_value, value_to_add=2)

print("\nDataFrame after df.pipe(add_value, 2):")
print(df_added)

# Without pipe (less readable for complex chains):
# df_added_alt = add_value(df, 2)
# print("\nDataFrame using add_value(df, 2):")
# print(df_added_alt)

Example output:

Original DataFrame:
       col1      col2      col3
0  0.496714 -0.138264  0.647689
1  1.523030 -0.234153 -0.234137
2  1.579213  0.767435 -0.469474
3  0.542560 -0.463418 -0.465730
4 -0.465729  1.104638 -0.122903

DataFrame after df.pipe(add_value, 2):
       col1      col2      col3
0  2.496714  1.861736  2.647689
1  3.523030  1.765847  1.765863
2  3.579213  2.767435  1.530526
3  2.542560  1.536582  1.534270
4  1.534271  3.104638  1.877097

pipe is especially useful when you have multiple custom processing steps to apply sequentially.

Row or Column-wise Application: .apply()

The .apply() method applies a function along an axis of a DataFrame. The function passed to apply receives a Series (representing a row or column) as input.

• axis=0 (default): Apply function to each column (function receives each column Series).
• axis=1: Apply function to each row (function receives each row Series).

Caution: Using apply(..., axis=1) can be slow, as it often involves iterating row by row internally. Always prefer vectorized operations if possible.

Example 1: Column-wise Mean (axis=0)

import pandas as pd
import numpy as np

rng = np.random.default_rng(43)
df = pd.DataFrame(rng.standard_normal((5, 3)), columns=['col1', 'col2', 'col3'])

print("DataFrame:")
print(df)

# Calculate the mean of each column
# The function np.mean receives each column Series
column_means = df.apply(np.mean, axis=0)
print("\nColumn Means (df.apply(np.mean, axis=0)):")
print(column_means)

# This is equivalent to the more direct and usually preferred method:
# column_means_direct = df.mean(axis=0)
# print("\nColumn Means (df.mean(axis=0)):")
# print(column_means_direct)

Example output:

DataFrame:
       col1      col2      col3
0 -0.535952 -0.875490 -0.269001
1  0.125819  0.747178 -1.255743
2 -1.034384 -0.977989 -0.375991
3  0.417561 -0.166471  0.674515
4 -0.563337 -0.508814 -0.494112

Column Means (df.apply(np.mean, axis=0)):
col1   -0.318059
col2   -0.356317
col3   -0.344066
dtype: float64

Example 2: Row-wise Mean (axis=1)

# Using df from Example 1

# Calculate the mean of each row
# The function np.mean receives each row Series
row_means = df.apply(np.mean, axis=1)
print("\nRow Means (df.apply(np.mean, axis=1)):")
print(row_means)

# Again, equivalent to the more direct method:
# row_means_direct = df.mean(axis=1)
# print("\nRow Means (df.mean(axis=1)):")
# print(row_means_direct)

Example output:

Row Means (df.apply(np.mean, axis=1)):
0   -0.560148
1   -0.127582
2   -0.796121
3   -0.021465
4   -0.522088
dtype: float64

Example 3: Row-wise Range (Max - Min) using Lambda

You can use lambda functions for concise custom operations.

# Using df from Example 1

# Calculate the range (max - min) for each row
row_range = df.apply(lambda s: s.max() - s.min(), axis=1)
print("\nRow Range (df.apply(lambda s: s.max() - s.min(), axis=1)):")
print(row_range)

Example output:

Row Range (df.apply(lambda s: s.max() - s.min(), axis=1)):
0    0.606489
1    2.002921
2    0.658393
3    1.854339
4    0.069225
dtype: float64

Element-wise Application: .applymap() (DataFrame) & .map() (Series)

When you need to apply a Python function that operates on a single value to every element of a DataFrame or Series, use applymap() or map() respectively.

• .applymap(func): Applies func to each element of a DataFrame.
• .map(func): Applies func to each element of a Series.

Performance: These methods iterate element by element at the Python level and can be significantly slower than vectorized operations or even apply. Use them when you need a pure Python function applied element-wise and vectorization isn’t feasible.

Example 1: Using .map() on a Series

Let’s format numbers in a Series as currency strings.

import pandas as pd
import numpy as np

rng = np.random.default_rng(44)
s = pd.Series(rng.random(5) * 100)

print("Original Series:")
print(s)

# Apply a formatting function using map
formatted_series = s.map(lambda x: f'${x:,.2f}') # Format as $xxx.xx
print("\nFormatted Series (s.map(lambda x: f'${x:,.2f}')):")
print(formatted_series)

Example output:

Original Series:
0    67.841675
1    24.097500
2    94.916909
3    23.539393
4    26.238116
dtype: float64

Formatted Series (s.map(lambda x: f'${x:,.2f}')):
0    $67.84
1    $24.10
2    $94.92
3    $23.54
4    $26.24
dtype: object

Example 2: Using .applymap() on a DataFrame

Let’s apply the same currency formatting function to all elements of a DataFrame.

# Using df from Example 1

print("Original DataFrame:")
print(df)

# Apply the formatting function using applymap
formatted_df = df.applymap(lambda x: f'{x:.2f}') # Format to 2 decimal places
print("\nFormatted DataFrame (df.applymap(lambda x: f'{x:.2f}')):")
print(formatted_df)

Example output:

Original DataFrame:
       col1      col2      col3
0 -0.535952 -0.875490 -0.269001
1  0.125819  0.747178 -1.255743
2 -1.034384 -0.977989 -0.375991
3  0.417561 -0.166471  0.674515
4 -0.563337 -0.508814 -0.494112

Formatted DataFrame (df.applymap(lambda x: f'{x:.2f}')):
    col1   col2   col3
0  -0.54  -0.88  -0.27
1   0.13   0.75  -1.26
2  -1.03  -0.98  -0.38
3   0.42  -0.17   0.67
4  -0.56  -0.51  -0.49