Python Pandas - Panel

Handling 3D+ Data with Pandas (Replacing Panel)

Historically, Pandas provided the Panel object as a primary container for 3-dimensional data. However, Panel has been deprecated since Pandas version 0.25.0 and subsequently removed.

The modern and recommended approach for handling data with 3 or more dimensions in Pandas is to use a DataFrame with a MultiIndex on its rows and/or columns. This approach is more flexible, consistent with other Pandas features, and generally more memory efficient.

While the term ‘Panel data’ (often used in econometrics for multi-entity time series) persists, the data structure used in Pandas is now typically a MultiIndexed DataFrame. Let’s explore how to represent and work with such data:

Understanding MultiIndex DataFrames for 3D+ Data

A MultiIndex (or Hierarchical Index) allows you to have multiple levels of indexing on an axis. To represent 3D data, you can conceptually think of it as:

• Items (like original Panel axis 0): Represented by the outer level of the row or column MultiIndex.
• Major Axis (like original Panel axis 1): Represented by the inner level(s) of the row MultiIndex (often representing time or observations).
• Minor Axis (like original Panel axis 2): Represented by the DataFrame’s columns (which could also be a MultiIndex for 4D+ data).

The key advantage is flexibility: you can structure these dimensions across row and column indices as best suits your analysis.

Creating MultiIndex DataFrames for 3D Data

You can create MultiIndex DataFrames in several ways:

From Multiple DataFrames using pd.concat

This is analogous to creating a Panel from a dictionary of DataFrames. Using pd.concat with the keys argument automatically creates a MultiIndex.

import pandas as pd
import numpy as np

# Create sample DataFrames
df1 = pd.DataFrame(np.random.randn(4, 3), columns=['A', 'B', 'C'])
df2 = pd.DataFrame(np.random.randn(4, 3), columns=['A', 'B', 'C'])

# Combine them, creating a MultiIndex on the rows
data_dict = {'Item1': df1, 'Item2': df2}
multi_df_rows = pd.concat(data_dict)

print("MultiIndex on Rows:")
print(multi_df_rows)
print("\nDataFrame Index:")
print(multi_df_rows.index)

# Alternative: Combine them creating a MultiIndex on the columns
multi_df_cols = pd.concat(data_dict, axis=1)

print("\nMultiIndex on Columns:")
print(multi_df_cols)
print("\nDataFrame Columns:")
print(multi_df_cols.columns)

Output (structure and values will vary due to random data):

MultiIndex on Rows:
                 A         B         C
Item1 0  0.123456 -0.567890  1.234567
      1 -0.987654  0.112233 -0.445566
      2  1.505050 -0.202020  0.303030
      3 -0.606060  1.707070 -0.808080
Item2 0  0.909090 -1.111111  0.222222
      1 -0.333333  1.444444 -0.555555
      2  0.666667 -0.777777  1.888888
      3 -0.999999  0.000000 -1.010101

DataFrame Index:
MultiIndex([('Item1', 0),
            ('Item1', 1),
            ('Item1', 2),
            ('Item1', 3),
            ('Item2', 0),
            ('Item2', 1),
            ('Item2', 2),
            ('Item2', 3)],
           )

MultiIndex on Columns:
      Item1                      Item2
          A         B         C         A         B         C
0  0.123456 -0.567890  1.234567  0.909090 -1.111111  0.222222
1 -0.987654  0.112233 -0.445566 -0.333333  1.444444 -0.555555
2  1.505050 -0.202020  0.303030  0.666667 -0.777777  1.888888
3 -0.606060  1.707070 -0.808080 -0.999999  0.000000 -1.010101

DataFrame Columns:
MultiIndex([('Item1', 'A'),
            ('Item1', 'B'),
            ('Item1', 'C'),
            ('Item2', 'A'),
            ('Item2', 'B'),
            ('Item2', 'C')],
           )

Notice how pd.concat automatically creates a MultiIndex object. The keys=[‘Item1’, ‘Item2’] argument specifies the outer level index names.

From NumPy Arrays with Explicit MultiIndex

You can construct a MultiIndex explicitly using methods like MultiIndex.from_product or MultiIndex.from_tuples and then assign it to a DataFrame created from a reshaped NumPy array.

import pandas as pd
import numpy as np

# Simulate 3D data: 2 items, 4 time points, 3 variables
np_data = np.random.randn(2, 4, 3)

# Create the MultiIndex for rows (Items x Time)
row_index = pd.MultiIndex.from_product(
    [['Item1', 'Item2'], pd.to_datetime(['2023-01-01', '2023-01-02', '2023-01-03', '2023-01-04'])],
    names=['Item', 'Time']
)

# Reshape the 3D array to 2D ( (2*4) rows, 3 columns )
reshaped_data = np_data.reshape(-1, 3)

# Create the DataFrame
multi_df = pd.DataFrame(reshaped_data, index=row_index, columns=['VarA', 'VarB', 'VarC'])

print(multi_df)
print("\nDataFrame Index:")
print(multi_df.index)

Output (structure and values will vary):

                           VarA      VarB      VarC
Item  Time
Item1 2023-01-01  -0.123456  0.567890 -1.234567
      2023-01-02   0.987654 -0.112233  0.445566
      2023-01-03  -1.505050  0.202020 -0.303030
      2023-01-04   0.606060 -1.707070  0.808080
Item2 2023-01-01  -0.909090  1.111111 -0.222222
      2023-01-02   0.333333 -1.444444  0.555555
      2023-01-03  -0.666667  0.777777 -1.888888
      2023-01-04   0.999999  0.000000  1.010101

DataFrame Index:
MultiIndex([('Item1', '2023-01-01'),
            ('Item1', '2023-01-02'),
            ('Item1', '2023-01-03'),
            ('Item1', '2023-01-04'),
            ('Item2', '2023-01-01'),
            ('Item2', '2023-01-02'),
            ('Item2', '2023-01-03'),
            ('Item2', '2023-01-04')],
           names=['Item', 'Time'])

Creating an Empty MultiIndex DataFrame

You can create an empty DataFrame and later assign a MultiIndex, or create one with a predefined MultiIndex but no data.

import pandas as pd

# Empty DataFrame
empty_df = pd.DataFrame()
print("Empty DataFrame:")
print(empty_df)

# Define a MultiIndex
index = pd.MultiIndex.from_product([['A', 'B'], [1, 2]], names=['Level1', 'Level2'])

# DataFrame with MultiIndex but no data
empty_multi_df = pd.DataFrame(index=index, columns=['DataCol1', 'DataCol2'])
print("\nEmpty DataFrame with MultiIndex:")
print(empty_multi_df)

Output:

Empty DataFrame:
Empty DataFrame
Columns: []
Index: []

Empty DataFrame with MultiIndex:
             DataCol1 DataCol2
Level1 Level2
A      1          NaN      NaN
       2          NaN      NaN
B      1          NaN      NaN
       2          NaN      NaN

Selecting Data from MultiIndex DataFrames

Selection works using standard Pandas indexing methods like .loc, .iloc, and [], but with tuples or slices to access different index levels.

Let’s use the multi_df (MultiIndex on rows) from the NumPy example:

# Recreate the DataFrame for selection examples
np_data = np.arange(24).reshape(2, 4, 3) # Use predictable data
row_index = pd.MultiIndex.from_product(
    [['Item1', 'Item2'], pd.to_datetime(['2023-01-01', '2023-01-02', '2023-01-03', '2023-01-04'])],
    names=['Item', 'Time']
)
reshaped_data = np_data.reshape(-1, 3)
multi_df = pd.DataFrame(reshaped_data, index=row_index, columns=['VarA', 'VarB', 'VarC'])

print("Original DataFrame:")
print(multi_df)

Selecting by Outer Index Level (like Panel ‘Items’)

Use .loc with the name of the outer index level.

# Select all data for 'Item1'
item1_data = multi_df.loc['Item1']
print("\nData for Item1:")
print(item1_data)

Output:

Data for Item1:
            VarA  VarB  VarC
Time
2023-01-01     0     1     2
2023-01-02     3     4     5
2023-01-03     6     7     8
2023-01-04     9    10    11

The result is a DataFrame where the ‘Item’ level is dropped from the index.

Selecting by Specific Row Index Combination (like Panel ‘major_axis’)

Use .loc with a tuple representing the full index or use .xs to cross-section.

# Select data for Item1 at a specific time
item1_time2 = multi_df.loc[('Item1', '2023-01-02')]
print("\nData for Item1 at 2023-01-02:")
print(item1_time2)

# Select data for a specific time across all items using xs
all_items_time3 = multi_df.xs('2023-01-03', level='Time')
print("\nData for 2023-01-03 across all Items:")
print(all_items_time3)

Output:

Data for Item1 at 2023-01-02:
VarA    3
VarB    4
VarC    5
Name: (Item1, 2023-01-02 00:00:00), dtype: int64

Data for 2023-01-03 across all Items:
       VarA  VarB  VarC
Item
Item1     6     7     8
Item2    18    19    20

Selecting Columns (like Panel ‘minor_axis’)

This works like standard DataFrame column selection.

# Select a single column
col_b = multi_df['VarB']
print("\nColumn 'VarB':")
print(col_b)

# Select multiple columns
cols_ac = multi_df[['VarA', 'VarC']]
print("\nColumns 'VarA' and 'VarC':")
print(ncols_ac)

Output:

Column 'VarB':
Item   Time
Item1  2023-01-01     1
       2023-01-02     4
       2023-01-03     7
       2023-01-04    10
Item2  2023-01-01    13
       2023-01-02    16
       2023-01-03    19
       2023-01-04    22
Name: VarB, dtype: int64

Columns 'VarA' and 'VarC':
                           VarA  VarC
Item  Time
Item1 2023-01-01     0     2
      2023-01-02     3     5
      2023-01-03     6     8
      2023-01-04     9    11
Item2 2023-01-01    12    14
      2023-01-02    15    17
      2023-01-03    18    20
      2023-01-04    21    23

Further Resources

Working with MultiIndex DataFrames is a fundamental skill for advanced Pandas usage. Refer to the official Pandas documentation for more details:

• MultiIndex / Advanced Indexing: https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.html
• pd.concat: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.concat.html

While Panel is deprecated, understanding how to represent similar data structures using MultiIndex is crucial for modern data analysis with Pandas.