NumPy - I/O with NumPy

I/O Operations with NumPy

NumPy provides functions to save ndarray objects to disk and load them back into memory. This is essential for persisting results, sharing data, or working with datasets larger than available memory.

The primary I/O functions are:

• np.save() and np.load(): Handle NumPy’s native binary file format (.npy). Efficient for storing single NumPy arrays.
• np.savez() and np.load(): Handle NumPy’s archive format (.npz), which can store multiple arrays in one file (uncompressed).
• np.savez_compressed() and np.load(): Similar to savez, but saves multiple arrays in a compressed .npz archive. Useful for large arrays.
• np.savetxt() and np.loadtxt(): Handle plain text files (e.g., .txt, .csv). Human-readable but less efficient and may lose precision.
• np.genfromtxt(): A more robust function for reading structured text files, capable of handling missing values.

Binary Files (.npy format)

NumPy’s .npy format is the standard way to save a single NumPy array to a file. It stores the data, shape, dtype, and other metadata needed to reconstruct the array accurately, even on different machines.

numpy.save()

Saves a single array to a disk file with the .npy extension (automatically added if not specified).

import numpy as np

a = np.array([1, 2, 3, 4, 5])
filename = 'my_array.npy' # Explicitly add .npy is good practice

np.save(filename, a)
print(f"Array saved to {filename}")

numpy.load()

Loads an array from a .npy or .npz file.

import numpy as np

filename = 'my_array.npy'

try:
    b = np.load(filename)
    print(f"Array loaded from {filename}:")
    print(b)
    print(f"Data type: {b.dtype}")
except FileNotFoundError:
    print(f"Error: File '{filename}' not found. Please run the save example first.")

Output (if my_array.npy exists):

Array saved to my_array.npy
Array loaded from my_array.npy:
[1 2 3 4 5]
Data type: int64

The np.save() and np.load() functions have an allow_pickle parameter. By default (allow_pickle=False since NumPy 1.16.3), loading pickled objects is disabled for security reasons, as loading pickled data from untrusted sources can execute arbitrary code. Only set allow_pickle=True if you trust the source of the file.

Archive Files (.npz format)

To save multiple arrays into a single file, use np.savez() or np.savez_compressed().

import numpy as np

a = np.array([[1, 2], [3, 4]])
b = np.arange(5)
archive_filename = 'my_archive.npz'

# Save arrays, assigning names ('arr1', 'arr2') within the archive
np.savez(archive_filename, arr1=a, arr2=b)
print(f"Multiple arrays saved to {archive_filename}")

# Load the archive
data = np.load(archive_filename)

print("\nArrays loaded from archive:")
print(f"Array 'arr1':\n{data['arr1']}")
print(f"Array 'arr2':\n{data['arr2']}")

# Check the files contained in the archive
print(f"\nFiles in archive: {data.files}")

data.close() # Good practice to close the file object

Output:

Multiple arrays saved to my_archive.npz

Arrays loaded from archive:
Array 'arr1':
[[1 2]
 [3 4]]
Array 'arr2':
[0 1 2 3 4]

Files in archive: ['arr1', 'arr2']

np.savez_compressed() works identically but uses compression, resulting in smaller files at the cost of slightly slower save/load times.

Text Files (.txt, .csv)

np.savetxt() and np.loadtxt() are used for storing and retrieving array data in plain text format. This is useful for interoperability with other software or for human inspection, but less efficient than binary formats.

Example

import numpy as np

a = np.array([[1, 2.5, 3.1], [4.0, 5.6, 6.9]])
text_filename = 'my_text_data.csv'

# Save the array to a CSV file, specifying format and delimiter
np.savetxt(text_filename, a, fmt='%.2f', delimiter=',', header='col1,col2,col3')
print(f"Array saved to {text_filename}")

# Load the array back from the text file
# We need to skip the header row
b = np.loadtxt(text_filename, delimiter=',', skiprows=1)
print(f"\nArray loaded from {text_filename}:")
print(b)
print(f"Data type: {b.dtype}")

Output:

Array saved to my_text_data.csv

Array loaded from my_text_data.csv:
[[1.   2.5  3.1]
 [4.   5.6  6.9]]
Data type: float64

savetxt and loadtxt offer parameters like delimiter, fmt (format specifier), header, footer, comments, and skiprows to customize the output and input parsing.

For more complex text files, especially those with missing values, consider using np.genfromtxt(), which provides more flexibility in handling errors and filling missing data.

When to Use Which Format?

• .npy: Best for saving/loading single NumPy arrays quickly and accurately. The standard for NumPy-to-NumPy data exchange.
• .npz: Good for bundling multiple related arrays into one file.
• .npz (compressed): Use when file size is a major concern and the compression overhead is acceptable.
• .txt/.csv: Best for human readability, sharing data with non-Python tools (like spreadsheets), or when plain text is required. Be mindful of potential precision loss for floating-point numbers.