Statistical Functions

Pandas - Statistical Functions

Pandas objects (Series, DataFrame) come equipped with a wide range of built-in statistical methods essential for data analysis and understanding data behavior. These methods typically exclude missing data (NaN) by default.

Percentage Change (`.pct_change()`)

The pct_change() function calculates the percentage change between the current element and a prior element (by default, the immediately preceding element). This is very useful for analyzing time series data or sequential measurements.

import pandas as pd
import numpy as np

# Example with a Series
s = pd.Series([1, 2, 3, 4, 5, 4, 5, 6])
print("Original Series:")
print(s)
print("\nPercentage Change in Series:")
print(s.pct_change())

# Example with a DataFrame
df = pd.DataFrame(np.random.randn(5, 2), columns=['A', 'B'])
print("\nOriginal DataFrame:")
print(df)
print("\nPercentage Change in DataFrame (column-wise default):")
print(df.pct_change()) # Default periods=1

Output (random values in DataFrame will differ):

Original Series:
0    1
1    2
2    3
3    4
4    5
5    4
6    5
7    6
dtype: int64

Percentage Change in Series:
0         NaN
1    1.000000  # (2-1)/1
2    0.500000  # (3-2)/2
3    0.333333  # (4-3)/3
4    0.250000  # (5-4)/4
5   -0.200000  # (4-5)/5
6    0.250000  # (5-4)/4
7    0.200000  # (6-5)/5
dtype: float64

Original DataFrame:
          A         B
0  0.123456 -0.987654
1 -0.321098  1.098765
2  0.543210 -0.109876
3 -1.876543  0.765432
4  0.112233 -0.445566

Percentage Change in DataFrame (column-wise default):
          A         B
0       NaN       NaN
1 -3.601135 -2.112388
2 -2.691861 -1.100000
3 -4.454545 -7.966022
4 -1.059806 -1.581996

The first value is always NaN as there’s no prior element to compare with. You can change the comparison period using the periods argument (e.g., periods=2 compares with the element two steps back) and compute row-wise changes using axis=1.

Covariance (`.cov()`)

Covariance measures the joint variability of two random variables (or two Series in Pandas). A positive covariance indicates that the variables tend to increase or decrease together, while a negative covariance indicates they move in opposite directions.

The .cov() method can be applied to a Series to compute its covariance with another Series, or directly to a DataFrame to compute the pairwise covariance between all columns.

Covariance between two Series

import pandas as pd
import numpy as np

s1 = pd.Series(np.random.randn(10))
s2 = pd.Series(np.random.randn(10))

print(f"Covariance between s1 and s2: {s1.cov(s2)}")

Output (a single float value, will vary):

Covariance between s1 and s2: -0.12978405324

Covariance Matrix for a DataFrame

import pandas as pd
import numpy as np

frame = pd.DataFrame(np.random.randn(10, 5), columns=['a', 'b', 'c', 'd', 'e'])

# Covariance between specific columns 'a' and 'b'
print(f"Covariance between column 'a' and 'b': {frame['a'].cov(frame['b'])}")

# Compute the full covariance matrix for the DataFrame
print("\nCovariance matrix for the DataFrame:")
print(frame.cov())

Output (values will vary):

Covariance between column 'a' and 'b': -0.5831292115274144

Covariance matrix for the DataFrame:
          a         b         c         d         e
a  1.780628 -0.583129 -0.185575  0.003679 -0.136558
b -0.583129  1.297011  0.136530 -0.523719  0.251064
c -0.185575  0.136530  0.915227 -0.053881 -0.058926
d  0.003679 -0.523719 -0.053881  1.521426 -0.487694
e -0.136558  0.251064 -0.058926 -0.487694  0.960761

The resulting DataFrame shows the covariance between each pair of columns. The diagonal elements represent the variance of each column (cov(X, X) = var(X)). Note that the covariance value calculated between columns ‘a’ and ‘b’ individually matches the corresponding entry in the matrix.

Correlation (`.corr()`)

Correlation is a standardized version of covariance, indicating the strength and direction of a linear relationship between two variables. Correlation values range from -1 (perfect negative linear relationship) to +1 (perfect positive linear relationship), with 0 indicating no linear relationship.

Pandas provides the .corr() method, which works similarly to .cov(). It can compute the correlation between two Series or the pairwise correlation matrix for a DataFrame. Several correlation methods are available:

'pearson' (Default): Standard correlation coefficient.
'kendall': Kendall Tau correlation coefficient (rank-based).
'spearman': Spearman rank correlation coefficient (rank-based).

import pandas as pd
import numpy as np

frame = pd.DataFrame(np.random.randn(10, 5), columns=['a', 'b', 'c', 'd', 'e'])

# Correlation between specific columns 'a' and 'b'
print(f"Pearson Correlation between 'a' and 'b': {frame['a'].corr(frame['b'])}")

# Compute the full Pearson correlation matrix for the DataFrame
print("\nPearson Correlation matrix:")
print(frame.corr(method='pearson')) # 'pearson' is default

# Compute the Spearman rank correlation matrix
print("\nSpearman Correlation matrix:")
print(frame.corr(method='spearman'))

Output (values will vary):

Pearson Correlation between 'a' and 'b': -0.383712785514

Pearson Correlation matrix:
          a         b         c         d         e
a  1.000000 -0.383713 -0.145368  0.002235 -0.104405
b -0.383713  1.000000  0.125311 -0.372821  0.224908
c -0.145368  0.125311  1.000000 -0.045661 -0.062840
d  0.002235 -0.372821 -0.045661  1.000000 -0.403380
e -0.104405  0.224908 -0.062840 -0.403380  1.000000

Spearman Correlation matrix:
          a         b         c         d         e
a  1.000000 -0.393939 -0.139394  0.078788 -0.163636
b -0.393939  1.000000  0.163636 -0.333333  0.163636
c -0.139394  0.163636  1.000000 -0.066667 -0.103030
d  0.078788 -0.333333 -0.066667  1.000000 -0.406061
e -0.163636  0.163636 -0.103030 -0.406061  1.000000

Like .cov(), non-numeric columns are automatically excluded from the correlation calculation. The diagonal elements are always 1 (correlation of a variable with itself).

Data Ranking (`.rank()`)

Ranking assigns a rank (e.g., 1st, 2nd, 3rd) to each element in a Series or along an axis in a DataFrame based on their magnitude. It’s useful for understanding relative order or as a basis for non-parametric statistics.

The .rank() method handles ties (equal values) using different strategies specified by the method parameter:

'average' (Default): Assigns the average rank to tied elements.
'min': Assigns the minimum rank in the group to all tied elements.
'max': Assigns the maximum rank in the group to all tied elements.
'first': Assigns ranks sequentially based on the order elements appear in the data.
'dense': Like ‘min’, but ranks increase only by 1 between groups (no gaps).

By default, ranks are assigned in ascending order (smaller values get lower ranks). Use ascending=False for descending ranks.

import pandas as pd
import numpy as np

# Create a Series with a tie
# Fix typo: np.random.np.random.randn -> np.random.randn
s = pd.Series(np.random.randn(5), index=list('abcde'))
s['d'] = s['b']  # Introduce a tie between 'b' and 'd'

print("Original Series with a tie:")
print(s)

# Calculate ranks (default: average method, ascending)
print("\nRanks (method='average', ascending=True):")
print(s.rank())

# Calculate ranks using 'first' method
print("\nRanks (method='first', ascending=True):")
print(s.rank(method='first'))

# Calculate ranks in descending order
print("\nRanks (method='average', ascending=False):")
print(s.rank(ascending=False))

Output (random values will differ, affecting ranks, but tie handling logic remains):

Original Series with a tie:
a   -0.500000
b    1.200000  # Tied value
c    0.300000
d    1.200000  # Tied value
e    1.800000
dtype: float64

Ranks (method='average', ascending=True):
a    1.0  # Smallest value
b    3.5  # Average of ranks 3 and 4
c    2.0
d    3.5  # Average of ranks 3 and 4
e    5.0  # Largest value
dtype: float64

Ranks (method='first', ascending=True):
a    1.0
b    3.0  # Gets rank 3 first
c    2.0
d    4.0  # Gets rank 4 next
e    5.0
dtype: float64

Ranks (method='average', ascending=False):
a    5.0  # Largest rank (smallest value)
b    2.5  # Average of ranks 2 and 3
c    4.0
d    2.5  # Average of ranks 2 and 3
e    1.0  # Smallest rank (largest value)
dtype: float64

These statistical functions are building blocks for more complex data analysis workflows in Pandas.

Statistical Functions

Pandas - Statistical Functions

Percentage Change (.pct_change())

Covariance (.cov())