Python Pandas - DataFrame

Pandas - DataFrame

A DataFrame is the workhorse data structure in Pandas. It’s a two-dimensional, size-mutable, and potentially heterogeneous tabular data structure with labeled axes (rows and columns). Think of it like a spreadsheet, an SQL table, or a dictionary of Series objects.

Key Features of DataFrame

Columns can hold different data types (integers, strings, floats, booleans, etc.).
Both rows and columns have labels, referred to as the Index and Columns respectively.
Size is mutable: you can add or delete columns and rows.
Supports arithmetic operations on both rows and columns.
Efficiently handles missing data (represented as NaN).
Provides powerful tools for slicing, dicing, grouping, merging, and reshaping data.

Conceptual Structure

Imagine a table containing student data:

Rows might be identified by a student ID (the index), and columns might represent ‘Name’, ‘Age’, ‘Major’, ‘GPA’ (the columns). Each column holds data of potentially different types.

          Name  Age      Major   GPA
Index
101        Alice   20  Economics  3.8
102          Bob   21    Physics  3.5
103      Charlie   19    History  3.9
...

Creating a DataFrame: `pd.DataFrame()`

The primary way to create a DataFrame is using the pandas.DataFrame constructor:

pandas.DataFrame(data=None, index=None, columns=None, dtype=None, copy=None)

Constructor Parameters:

Parameter	Description
data	The data to populate the DataFrame. Can be various types: NumPy ndarray, dictionary, list, Series, another DataFrame, etc.
index	Labels for the rows. Optional; defaults to a `RangeIndex` (0, 1, 2, …) if not provided.
columns	Labels for the columns. Optional; inferred from the input data (e.g., dict keys) or defaults to a `RangeIndex` if not provided and not inferrable.
dtype	Forces a specific data type for all columns. If `None`, types are inferred per column.
copy	Deprecated in recent Pandas versions. Previously controlled whether to copy input data (default `False`). Data copying behavior is now more nuanced; see Pandas documentation for specifics.

Ways to Create DataFrames

DataFrames can be created from various common data structures:

Dictionary of lists or NumPy arrays
List of dictionaries
Dictionary of Series
List of lists or NumPy array
Another DataFrame

Create an Empty DataFrame

You can create an empty DataFrame, which might be populated later.

Example: Empty DataFrame

# Import pandas, aliased as pd (standard convention)
import pandas as pd

empty_df = pd.DataFrame()
print(empty_df)

Output:

Empty DataFrame
Columns: []
Index: []

Create from Lists

You can create a DataFrame from a simple list (resulting in a single column) or a list of lists (where each inner list becomes a row).

Example 1: From a single list

import pandas as pd

data_list_simple = [10, 20, 30, 40, 50]
df_from_list = pd.DataFrame(data_list_simple, columns=['Numbers'])
print(df_from_list)

Output:

Example 2: From a list of lists

import pandas as pd

data_list_of_lists = [
    ['Alex', 25, 'Software Engineer'],
    ['Bob', 30, 'Data Scientist'],
    ['Clarke', 22, 'Web Developer']
]

# Provide column names
df_from_lol = pd.DataFrame(data_list_of_lists,
                           columns=['Name', 'Age', 'Job Title'])
print(df_from_lol)

Output:

     Name  Age          Job Title
0    Alex   25  Software Engineer
1     Bob   30     Data Scientist
2  Clarke   22      Web Developer

Example 3: Specifying Data Type

import pandas as pd
import numpy as np # For np.float64

data_list_of_lists = [
    ['Alex', 10],
    ['Bob', 12],
    ['Clarke', 13]
]

# Specify dtype for 'Age' column during creation (more common post-creation)
# Or specify a single dtype for all applicable columns
df_typed = pd.DataFrame(data_list_of_lists,
                        columns=['Name', 'Age'],
                        dtype=np.float64) # Applies float64 where possible

print(df_typed)
print('\nData Types:')
print(df_typed.dtypes)

Output:

     Name   Age
0    Alex  10.0
1     Bob  12.0
2  Clarke  13.0

Data Types:
Name     object  # Name cannot be float, stays as object
Age     float64
dtype: object

Note: The dtype parameter tries to cast columns. If a column cannot be cast (like ‘Name’ to float), its original inferred type (‘object’ for strings) is kept. It’s often more practical to adjust dtypes after creation using .astype().

Create from Dict of Lists or Arrays

This is a very common and often efficient way to create DataFrames. Dictionary keys become column names, and the lists or arrays become the column data. All lists/arrays must have the same length.

If no index is provided, a default RangeIndex (0, 1, 2, …) is created.

Example 1: Basic creation

import pandas as pd

data_dict = {
    'Student Name': ['Tom', 'Jack', 'Steve', 'Ricky'],
    'Score': [85, 92, 78, 88]
}

df_from_dict = pd.DataFrame(data_dict)
print(df_from_dict)

Output:

  Student Name  Score
0          Tom     85
1         Jack     92
2        Steve     78
3        Ricky     88

Note: The default index 0, 1, 2, 3 is automatically assigned.

Example 2: Providing a custom index

import pandas as pd

data_dict = {
    'Student Name': ['Tom', 'Jack', 'Steve', 'Ricky'],
    'Score': [85, 92, 78, 88]
}
custom_index = ['ID_101', 'ID_102', 'ID_103', 'ID_104']

df_indexed = pd.DataFrame(data_dict, index=custom_index)
print(df_indexed)

Output:

       Student Name  Score
ID_101          Tom     85
ID_102         Jack     92
ID_103        Steve     78
ID_104        Ricky     88

Note: The index parameter assigns meaningful labels to the rows.

Create from List of Dicts

Each dictionary in the list represents a row. Dictionary keys are automatically inferred as column names. If a dictionary is missing a key that exists in other dictionaries, NaN (Not a Number) will be placed in that cell.

Example 1: Missing data

import pandas as pd

data_list_of_dicts = [
    {'col_a': 1, 'col_b': 2},
    {'col_a': 5, 'col_b': 10, 'col_c': 20}, # This dict has 'col_c'
    {'col_a': 9, 'col_b': 15} # This one doesn't
]

df_from_lod = pd.DataFrame(data_list_of_dicts)
print(df_from_lod)

Output:

   col_a  col_b  col_c
0      1      2    NaN
1      5     10   20.0
2      9     15    NaN

Note: NaN is automatically inserted where data is missing (like ‘col_c’ in the first and third rows). Pandas infers the most suitable dtype for ‘col_c’ (float64 here, because NaN is a float).

Example 2: Specifying Index

import pandas as pd

data_list_of_dicts = [
    {'col_a': 1, 'col_b': 2},
    {'col_a': 5, 'col_b': 10, 'col_c': 20}
]
row_labels = ['Row_1', 'Row_2']

df_lod_indexed = pd.DataFrame(data_list_of_dicts, index=row_labels)
print(df_lod_indexed)

Output:

       col_a  col_b  col_c
Row_1      1      2    NaN
Row_2      5     10   20.0

Example 3: Specifying Columns

You can explicitly specify the columns and their order. If a specified column doesn’t exist as a key in the dictionaries, it will be filled with NaN.

import pandas as pd

data_list_of_dicts = [
    {'col_a': 1, 'col_b': 2},
    {'col_a': 5, 'col_b': 10, 'col_c': 20}
]
row_labels = ['Row_1', 'Row_2']

# Select and reorder existing columns
df1 = pd.DataFrame(data_list_of_dicts, index=row_labels, columns=['col_b', 'col_a'])

# Select existing columns and one non-existing column ('col_d')
df2 = pd.DataFrame(data_list_of_dicts, index=row_labels, columns=['col_a', 'col_c', 'col_d'])

print("DataFrame df1 (Selected existing columns):")
print(df1)
print("\nDataFrame df2 (Includes non-existing column 'col_d'):")
print(df2)

Output:

DataFrame df1 (Selected existing columns):
       col_b  col_a
Row_1      2      1
Row_2     10      5

DataFrame df2 (Includes non-existing column 'col_d'):
       col_a  col_c  col_d
Row_1      1    NaN    NaN
Row_2      5   20.0    NaN

Note: df1 shows columns ‘col_b’ and ‘col_a’ in the specified order. df2 includes ‘col_a’ and ‘col_c’ (with NaN where missing) and adds a new column ‘col_d’ filled entirely with NaN because it wasn’t in the input dictionaries.

Create from Dict of Series

Dictionary keys become column names, and the Series values become the column data. The resulting DataFrame’s index will be the union of all indices from the input Series. Missing values will be filled with NaN where indices don’t align.

Example

import pandas as pd

data_dict_of_series = {
    'ColOne': pd.Series([1.0, 2.0, 3.0], index=['a', 'b', 'c']),
    'ColTwo': pd.Series([1.0, 2.0, 3.0, 4.0], index=['a', 'b', 'c', 'd'])
}

df_from_dos = pd.DataFrame(data_dict_of_series)
print(df_from_dos)

Output:

   ColOne  ColTwo
a     1.0     1.0
b     2.0     2.0
c     3.0     3.0
d     NaN     4.0

Note: The final index is (‘a’, ‘b’, ‘c’, ‘d’). Since ‘ColOne’ didn’t have an index ‘d’, NaN is placed in that cell.

Column Operations

Selecting, adding, and deleting columns are fundamental DataFrame operations.

Column Selection

Access columns using dictionary-like square bracket notation [] or attribute access . (if the column name is a valid Python identifier and doesn’t conflict with DataFrame methods).

# Using df_from_dos from the previous example
print("Selecting 'ColOne' using []:")
print(df_from_dos['ColOne'])

# print("\nSelecting 'ColTwo' using attribute access:") # Use with caution
# print(df_from_dos.ColTwo)

print("\nSelecting multiple columns [['ColTwo', 'ColOne']]:")
print(df_from_dos[['ColTwo', 'ColOne']]) # Pass a list of column names

Output:

Selecting 'ColOne' using []:
a    1.0
b    2.0
c    3.0
d    NaN
Name: ColOne, dtype: float64

Selecting multiple columns [['ColTwo', 'ColOne']]:
   ColTwo  ColOne
a     1.0     1.0
b     2.0     2.0
c     3.0     3.0
d     4.0     NaN

Column Addition

Add a new column by assigning data to a new column label.

# Using df_from_dos
print("Original DataFrame:")
print(df_from_dos)

# Add a new column by assigning a Series (index aligns)
print("\nAdding 'ColThree' by assigning a Series:")
df_from_dos['ColThree'] = pd.Series([10, 20, 30], index=['a', 'b', 'c'])
print(df_from_dos)

# Add a new column by calculation using existing columns
print("\nAdding 'ColFour' = ColOne + ColThree:")
df_from_dos['ColFour'] = df_from_dos['ColOne'] + df_from_dos['ColThree']
print(df_from_dos)

# Add a new column with a scalar value (broadcasts to all rows)
print("\nAdding 'ColFive' with a scalar value:")
df_from_dos['ColFive'] = 100
print(df_from_dos)

Output:

Original DataFrame:
   ColOne  ColTwo
a     1.0     1.0
b     2.0     2.0
c     3.0     3.0
d     NaN     4.0

Adding 'ColThree' by assigning a Series:
   ColOne  ColTwo  ColThree
a     1.0     1.0      10.0
b     2.0     2.0      20.0
c     3.0     3.0      30.0
d     NaN     4.0       NaN

Adding 'ColFour' = ColOne + ColThree:
   ColOne  ColTwo  ColThree  ColFour
a     1.0     1.0      10.0     11.0
b     2.0     2.0      20.0     22.0
c     3.0     3.0      30.0     33.0
d     NaN     4.0       NaN      NaN

Adding 'ColFive' with a scalar value:
   ColOne  ColTwo  ColThree  ColFour  ColFive
a     1.0     1.0      10.0     11.0      100
b     2.0     2.0      20.0     22.0      100
c     3.0     3.0      30.0     33.0      100
d     NaN     4.0       NaN      NaN      100

Column Deletion

Use the del keyword or the .pop() or .drop() methods.

# Using df_from_dos from the previous step
print("DataFrame before deletion:")
print(df_from_dos)

# Using del (in-place)
print("\nDeleting 'ColOne' using del:")
del df_from_dos['ColOne']
print(df_from_dos)

# Using pop (removes and returns the column)
print("\nDeleting 'ColTwo' using pop:")
popped_col = df_from_dos.pop('ColTwo')
print("DataFrame after pop:")
print(df_from_dos)
print("\nPopped Column 'ColTwo':")
print(popped_col)

# Using drop (returns a new DataFrame, use inplace=True to modify original)
print("\nDeleting 'ColThree' using drop (returns new DataFrame):")
df_after_drop = df_from_dos.drop(columns=['ColThree']) # Use 'columns' argument
# Or: df_after_drop = df_from_dos.drop(['ColThree'], axis=1)
print(df_after_drop)
print("\nOriginal DataFrame after non-inplace drop (unchanged):")
print(df_from_dos)

Output:

DataFrame before deletion:
   ColOne  ColTwo  ColThree  ColFour  ColFive
a     1.0     1.0      10.0     11.0      100
b     2.0     2.0      20.0     22.0      100
c     3.0     3.0      30.0     33.0      100
d     NaN     4.0       NaN      NaN      100

Deleting 'ColOne' using del:
   ColTwo  ColThree  ColFour  ColFive
a     1.0      10.0     11.0      100
b     2.0      20.0     22.0      100
c     3.0      30.0     33.0      100
d     4.0       NaN      NaN      100

Deleting 'ColTwo' using pop:
DataFrame after pop:
   ColThree  ColFour  ColFive
a      10.0     11.0      100
b      20.0     22.0      100
c      30.0     33.0      100
d       NaN      NaN      100

Popped Column 'ColTwo':
a    1.0
b    2.0
c    3.0
d    4.0
Name: ColTwo, dtype: float64

Deleting 'ColThree' using drop (returns new DataFrame):
   ColFour  ColFive
a     11.0      100
b     22.0      100
c     33.0      100
d      NaN      100

Original DataFrame after non-inplace drop (unchanged):
   ColThree  ColFour  ColFive
a      10.0     11.0      100
b      20.0     22.0      100
c      30.0     33.0      100
d       NaN      NaN      100

Row Operations

Selecting, adding, and deleting rows involve using the index.

Row Selection: `.loc` and `.iloc`

Use .loc[] for label-based selection and .iloc[] for integer position-based selection.

import pandas as pd

data_dict = {
    'ColOne': pd.Series([1.0, 2.0, 3.0], index=['a', 'b', 'c']),
    'ColTwo': pd.Series([1.0, 2.0, 3.0, 4.0], index=['a', 'b', 'c', 'd'])
}
df = pd.DataFrame(data_dict)
print("DataFrame:")
print(df)

# Selection by label using .loc
print("\nSelecting row with label 'b' using .loc['b']:")
print(df.loc['b'])

# Selection by integer position using .iloc
print("\nSelecting row at position 2 (label 'c') using .iloc[2]:")
print(df.iloc[2])

# Selecting multiple rows by label
print("\nSelecting rows 'a' and 'd' using .loc[['a', 'd']]:")
print(df.loc[['a', 'd']])

# Selecting multiple rows by position
print("\nSelecting rows at positions 0 and 3 using .iloc[[0, 3]]:")
print(df.iloc[[0, 3]])

Output:

DataFrame:
   ColOne  ColTwo
a     1.0     1.0
b     2.0     2.0
c     3.0     3.0
d     NaN     4.0

Selecting row with label 'b' using .loc['b']:
ColOne    2.0
ColTwo    2.0
Name: b, dtype: float64

Selecting row at position 2 (label 'c') using .iloc[2]:
ColOne    3.0
ColTwo    3.0
Name: c, dtype: float64

Selecting rows 'a' and 'd' using .loc[['a', 'd']]:
   ColOne  ColTwo
a     1.0     1.0
d     NaN     4.0

Selecting rows at positions 0 and 3 using .iloc[[0, 3]]:
   ColOne  ColTwo
a     1.0     1.0
d     NaN     4.0

Selecting a single row returns a Series, while selecting multiple rows returns a DataFrame.

Row Slicing

Use slicing with .loc (label-based, endpoint inclusive) or .iloc (integer-based, endpoint exclusive).

# Using df from the previous example

# Slice rows by label (inclusive)
print("\nSlicing rows 'b' through 'd' using .loc['b':'d']:")
print(df.loc['b':'d'])

# Slice rows by position (exclusive)
print("\nSlicing rows from position 1 up to (not including) 3 using .iloc[1:3]:")
print(df.iloc[1:3])

# Direct integer slicing on DataFrame (df[1:3]) is possible but generally
# discouraged; prefer explicit .iloc for clarity and robustness.
# print("\nSlicing using df[1:3] (discouraged):")
# print(df[1:3])

Output:

Slicing rows 'b' through 'd' using .loc['b':'d']:
   ColOne  ColTwo
b     2.0     2.0
c     3.0     3.0
d     NaN     4.0

Slicing rows from position 1 up to (not including) 3 using .iloc[1:3]:
   ColOne  ColTwo
b     2.0     2.0
c     3.0     3.0

Addition of Rows

The recommended way to add rows is using pd.concat(). The older .append() method is deprecated and removed in Pandas 2.0 and should not be used.

import pandas as pd

df1 = pd.DataFrame([[1, 2], [3, 4]], columns=['A', 'B'], index=[0, 1])
df2 = pd.DataFrame([[5, 6], [7, 8]], columns=['A', 'B'], index=[2, 3]) # Use different index
df3 = pd.DataFrame([[9, 10], [11, 12]], columns=['A', 'C'], index=[4, 5]) # Different column

print("DataFrame df1:")
print(df1)
print("\nDataFrame df2:")
print(df2)
print("\nDataFrame df3:")
print(df3)

# Concatenate df1 and df2 (same columns)
combined_df = pd.concat([df1, df2])
print("\nCombined df1 and df2 using pd.concat():")
print(combined_df)

# Concatenate df1 and df3 (different columns, align on index)
combined_df_cols = pd.concat([df1, df3]) # outer join by default
print("\nCombined df1 and df3 using pd.concat() (outer join):")
print(combined_df_cols)

# If using older examples that had duplicate indices from append:
# df_old_append = pd.DataFrame([[1, 2], [3, 4]], columns=['A', 'B'], index=[0, 1])
# df_old_append2 = pd.DataFrame([[5, 6], [7, 8]], columns=['A', 'B'], index=[0, 1])
# combined_dup_index = pd.concat([df_old_append, df_old_append2], ignore_index=True)
# print("\nCombined with ignore_index=True to reset index:")
# print(combined_dup_index)

Output:

DataFrame df1:
   A  B
0  1  2
1  3  4

DataFrame df2:
   A  B
2  5  6
3  7  8

DataFrame df3:
    A   C
4   9  10
5  11  12

Combined df1 and df2 using pd.concat():
   A  B
0  1  2
1  3  4
2  5  6
3  7  8

Combined df1 and df3 using pd.concat() (outer join):
     A    B     C
0  1.0  2.0   NaN
1  3.0  4.0   NaN
4  9.0  NaN  10.0
5 11.0  NaN  12.0

pd.concat intelligently combines DataFrames. Use ignore_index=True if you want a fresh default integer index on the result.

Deletion of Rows

Use the .drop() method, specifying the row labels (index values) to remove and axis=0 (or omit axis as it defaults to 0).

# Using combined_df from the previous example
print("DataFrame before row deletion:")
print(combined_df)

# Drop rows with labels 0 and 3
# By default, drop returns a new DataFrame. Use inplace=True to modify.
df_dropped = combined_df.drop([0, 3], axis=0)
print("\nDataFrame after dropping rows 0 and 3:")
print(df_dropped)

# Example with duplicate labels (if created that way previously)
# df_dup = pd.DataFrame([[1,2],[3,4],[5,6]], columns=['A','B'], index=[0,1,0])
# print("\nDataFrame with duplicate index 0:")
# print(df_dup)
# print("\nDropping label 0 (removes all rows with that label):")
# print(df_dup.drop(0))

Output:

DataFrame before row deletion:
   A  B
0  1  2
1  3  4
2  5  6
3  7  8

DataFrame after dropping rows 0 and 3:
   A  B
1  3  4
2  5  6

If an index label is duplicated, drop() will remove all rows matching that label.

Python Pandas - DataFrame

Pandas - DataFrame

Key Features of DataFrame

Conceptual Structure

Creating a DataFrame: pd.DataFrame()

Ways to Create DataFrames

Create an Empty DataFrame

Example: Empty DataFrame

Create from Lists

Example 1: From a single list

Example 2: From a list of lists

Example 3: Specifying Data Type

Create from Dict of Lists or Arrays

Example 1: Basic creation

Example 2: Providing a custom index

Create from List of Dicts

Example 1: Missing data

Example 2: Specifying Index

Example 3: Specifying Columns

Create from Dict of Series

Example

Column Operations

Column Selection

Column Addition

Column Deletion

Row Operations

Row Selection: .loc and .iloc

Row Slicing

Addition of Rows

Deletion of Rows

Creating a DataFrame: `pd.DataFrame()`

Row Selection: `.loc` and `.iloc`