SciPy - Input and Output

SciPy - Input & Output

The scipy.io package provides a versatile set of functions for reading from and writing to various file formats. This is crucial for data exchange between different software environments or for persistent storage of scientific data. Some of the supported formats include:

• MATLAB (.mat files)
• IDL (Interactive Data Language files)
• Matrix Market (.mtx files)
• Wave audio files (.wav)
• ARFF (Attribute-Relation File Format)
• NetCDF (Network Common Data Form)
• And others. For very large datasets or more complex hierarchical data, consider libraries like HDF5 (h5py or tables).

Let’s focus on one of the most commonly used formats for scientific data exchange: MATLAB files.

MATLAB Files (.mat)

SciPy provides functions to load data from and save data to MATLAB’s .mat file format. This is particularly useful when collaborating with MATLAB users or using existing MATLAB data.

Function	Description
scipy.io.loadmat()	Loads a .mat file into a Python dictionary. Variables from the .mat file become keys in the dictionary.
scipy.io.savemat()	Saves a dictionary of variable names and NumPy arrays (or other compatible data types) to a .mat file.
scipy.io.whosmat()	Lists variables stored in a .mat file without loading the entire file into memory. Returns a list of tuples, each containing the variable name, its shape, and its data type.

Let’s see an example of saving and loading a .mat file:

import scipy.io as sio
import numpy as np

# Create a NumPy array
my_array = np.arange(15).reshape(3, 5)

# Data to save: a dictionary where keys are variable names for MATLAB
data_to_save = {'my_matrix': my_array, 'my_scalar': 42.0}

# Save data to a .mat file
file_path = 'my_data.mat'
sio.savemat(file_path, data_to_save)

print(f"Data saved to {file_path}")

# Now, load the .mat file
loaded_data = sio.loadmat(file_path)

print("\nContent of the loaded .mat file:")
# The loaded data is a dictionary. Let's inspect it.
for key, value in loaded_data.items():
    print(f"Variable name: {key}")
    if isinstance(value, np.ndarray):
        print(f"Type: {type(value)}, Shape: {value.shape}, Dtype: {value.dtype}")
        print(f"Data:\n{value}")
    else:
        print(f"Type: {type(value)}, Value: {value}")
    print("---")

# Access a specific variable
loaded_array = loaded_data['my_matrix']
print(f"\nAccessed 'my_matrix':\n{loaded_array}")

The output of this program would be similar to (exact meta-information might vary):

Data saved to my_data.mat

Content of the loaded .mat file:
Variable name: __header__
Type: <class 'bytes'>, Value: b'MATLAB 5.0 MAT-file, Platform: ...'
---
Variable name: __version__
Type: <class 'str'>, Value: 1.0
---
Variable name: __globals__
Type: <class 'list'>, Value: []
---
Variable name: my_matrix
Type: <class 'numpy.ndarray'>, Shape: (3, 5), Dtype: int64
Data:
[[ 0  1  2  3  4]
 [ 5  6  7  8  9]
 [10 11 12 13 14]]
---
Variable name: my_scalar
Type: <class 'numpy.ndarray'>, Shape: (1, 1), Dtype: float64
Data:
[[42.]]
---

Accessed 'my_matrix':
[[ 0  1  2  3  4]
 [ 5  6  7  8  9]
 [10 11 12 13 14]]

Notice that loadmat returns a dictionary containing the variables you saved, along with some metadata (like __header__, __version__, __globals__). Scalars are often loaded as 2D arrays; you might use squeeze_me=True in loadmat to convert them to 0D arrays if appropriate. For example: loaded_data = sio.loadmat(file_path, squeeze_me=True).

To inspect the contents of a MATLAB file without reading all data into memory, especially useful for large files, use whosmat:

import scipy.io as sio

# Assuming 'my_data.mat' was created by the previous example
file_path = 'my_data.mat'
variables_info = sio.whosmat(file_path)

print("Variables in the .mat file (name, shape, type):")
for var_name, var_shape, var_type in variables_info:
    print(f"- {var_name}: shape={var_shape}, type={var_type}")

The output of this program will be:

Variables in the .mat file (name, shape, type):
- my_matrix: shape=(3, 5), type=int64
- my_scalar: shape=(1, 1), type=float64

Additional Notes: When working with .mat files, especially those from newer MATLAB versions (v7.3 and later), they might be HDF5 based. scipy.io.loadmat can handle many of these, but for complex structures or very large HDF5-based .mat files, using a dedicated HDF5 library like h5py might offer more control. Always check the scipy.io.loadmat documentation for options like mat_dtype and chars_as_strings to handle data type conversions as needed.