Python Further Extensions

Python Extension Programming with the C API

Python’s flexibility allows integrating code written in compiled languages like C or C++. These compiled libraries, callable from Python, are known as “extensions.”

A Python extension module is essentially a standard shared library (like .so on Unix/Linux or .dll on Windows) that exposes an interface compatible with the Python interpreter.

Why Write C Extensions?

• Performance: For computationally intensive tasks where Python’s speed is insufficient, C/C++ code can provide significant speedups.
• Accessing C Libraries: To wrap existing C/C++ libraries and make them available within Python.
• Low-Level System Access: To perform operations not directly exposed by standard Python modules.

Modern Alternatives:

While the direct C API provides maximum control, several modern tools often simplify extension development:

• ctypes: Part of the standard library. Allows calling functions in shared libraries directly from Python without writing C code, good for simple interfaces.
• cffi: Third-party library. Allows calling C code from Python, often easier than the raw C API for complex interfaces.
• Cython: A language that makes writing C extensions for Python as easy as Python itself. Compiles Python-like code (with optional C type declarations) into optimized C extensions.
• Pybind11: A popular C++ library for creating Python bindings for C++ code.

This guide focuses on the direct C API, the lowest-level method.

Prerequisites for Writing C Extensions

• C Compiler: You need a C compiler compatible with your Python installation (e.g., GCC on Linux, MSVC on Windows).
• Python Development Headers: These files (Python.h and others) provide the definitions needed to interact with the Python interpreter. Install them via your system’s package manager (e.g., python3-dev on Debian/Ubuntu, python3-devel on Fedora/CentOS) or ensure they are included if you built Python from source. Windows installers usually include them.
• Build System Knowledge: Familiarity with compiling C code and using build tools.

Additionally, a good understanding of C programming and Python’s object model is assumed.

Structure of a Simple C Extension

A basic C extension module typically involves these parts:

• Include Python.h: Provides access to the Python C API.
• C Functions: The actual implementation of the functions you want to expose to Python.
• Method Definition Table: Maps Python function names to their C implementations.
• Module Definition Structure: Describes the module itself (name, docstring, methods).
• Initialization Function: The entry point called by Python when importing the module.

1. Include Python.h

This header file must be included first in your C source file.

#include <Python.h>

2. C Functions (Implementing Extension Logic)

These C functions will be called from Python. They typically take PyObject* arguments and return a PyObject*. PyObject is the fundamental C struct representing any Python object.

Common Signatures:

// Function taking variable arguments (like *args)
static PyObject * my_c_function(PyObject *self, PyObject *args);

// Function taking variable arguments and keyword arguments (like **kwargs)
static PyObject * my_c_function_kw(PyObject *self, PyObject *args, PyObject *kwargs);

// Function taking no arguments
static PyObject * my_c_function_noargs(PyObject *self);

self usually refers to the module object for module-level functions. args is a tuple of positional arguments. kwargs is a dictionary of keyword arguments.

Functions must return a PyObject*. To return Python’s None, use Py_RETURN_NONE. If an error occurs, return NULL after setting an appropriate Python exception.

Conventionally, C functions are declared static unless needed externally. Naming often combines module and function name (e.g., spam_system).

Example C function:

// Example C function implementation (details later)
static PyObject * spam_system(PyObject *self, PyObject *args) {
    const char *command;
    int sts;

    // Parse arguments from Python tuple 'args'
    if (!PyArg_ParseTuple(args, "s", &command)) {
        // PyArg_ParseTuple raises TypeError on failure
        return NULL; // Indicate error
    }

    // Call C library function
    sts = system(command);

    // Check for C errors and raise Python exception if needed
    if (sts < 0) {
        PyErr_SetString(PyExc_OSError, "system() failed");
        return NULL;
    }

    // Build and return a Python integer
    return PyLong_FromLong(sts);
}

3. Method Definition Table (PyMethodDef)

This array maps the Python function names, C function pointers, argument types, and docstrings.

Structure:

struct PyMethodDef {
    const char *ml_name;   /* Python function name */
    PyCFunction ml_meth;  /* C function pointer */
    int         ml_flags;  /* Argument flags (METH_VARARGS, METH_KEYWORDS, METH_NOARGS) */
    const char *ml_doc;    /* Docstring */
};

Flags determine the C function signature expected:

• METH_VARARGS: Function expects (PyObject *self, PyObject *args).
• METH_KEYWORDS: Function expects (PyObject *self, PyObject *args, PyObject *kwargs). Can be OR’ed with METH_VARARGS.
• METH_NOARGS: Function expects (PyObject *self). No arguments allowed from Python.

The table must end with a sentinel entry {NULL, NULL, 0, NULL}.

Example Table:

static PyMethodDef SpamMethods[] = {
    // {Python Name, C Function, Flags, Docstring}
    {"system", spam_system, METH_VARARGS,
     "Execute a shell command."},
    // Add other functions here...
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

4. Module Definition Structure (PyModuleDef)

This structure holds all information about the module itself.

Structure:

static struct PyModuleDef spammodule = {
    PyModuleDef_HEAD_INIT,
    "spam",              /* Module name */
    "Example module that provides a function.",  /* Module docstring */
    -1,                 /* Size of per-interpreter state, -1 means no state */
    SpamMethods         /* Method table defined above */
    /* Other fields like m_slots, m_traverse, m_clear, m_free can be NULL */
};

5. Initialization Function (PyInit_modulename)

This is the only non-static function. Python calls it when the module is first imported. Its name must be PyInit_ followed by the module name specified in the PyModuleDef.

It uses PyModule_Create() with the module definition structure.

Syntax:

PyMODINIT_FUNC // Macro ensuring correct visibility and return type
PyInit_spam(void) {
    return PyModule_Create(&spammodule);
}

Complete Simple Example (spam.c)

#define PY_SSIZE_T_CLEAN // Recommended for modern C API usage
#include <Python.h>
#include <stdlib.h> // For system()

// 1. C Function Implementation
static PyObject * spam_system(PyObject *self, PyObject *args) {
    const char *command;
    int sts;

    if (!PyArg_ParseTuple(args, "s", &command)) {
        return NULL;
    }
    sts = system(command);
     if (sts < 0) {
        PyErr_SetString(PyExc_OSError, "system() failed");
        return NULL;
    }
    return PyLong_FromLong(sts);
}

// 2. Method Definition Table
static PyMethodDef SpamMethods[] = {
    {"system", spam_system, METH_VARARGS, "Execute a shell command."},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

// 3. Module Definition Structure
static struct PyModuleDef spammodule = {
    PyModuleDef_HEAD_INIT,
    "spam",   /* name of module */
    "Example module documentation.", /* module documentation, may be NULL */
    -1,       /* size of per-interpreter state of the module, or -1 if the module keeps state in global variables. */
    SpamMethods
};

// 4. Initialization Function
PyMODINIT_FUNC PyInit_spam(void) {
    return PyModule_Create(&spammodule);
}

Building and Installing Extensions

Python’s standard build tools (setuptools, build) are used to compile and install extensions. distutils is largely legacy.

You typically need a setup.py (or pyproject.toml with build backend configuration) file.

Example setup.py (using setuptools):

from setuptools import setup, Extension

spam_module = Extension('spam', # Python module name
                        sources=['spam.c']) # List of source files

setup(
    name='spam',              # Package name
    version='1.0',            # Version
    description='Example C extension module',
    ext_modules=[spam_module] # List of Extension objects
)

Build/Install Commands:

# Recommended modern way (requires 'build' package: pip install build)
# Builds wheel and sdist in 'dist/' directory
python -m build

# Install the built wheel (replace with actual wheel filename)
pip install dist/spam-1.0-cp310-cp310-linux_x86_64.whl

# --- OR --- (Older setuptools direct install)
# Compile and install directly into the current Python environment
# Might require root/admin privileges depending on the environment
# python setup.py install

Importing and Using the Extension

Once installed, import and use it like any other Python module:

#!/usr/bin/env python3

import spam
import os

print("Docstring:", spam.__doc__)

# Example: List directory contents using the extension
try:
    # Use a safe command for demonstration
    command = f"ls -l {os.getcwd()}" # Use appropriate command for your OS
    status = spam.system(command)
    print(f"\nCommand exited with status: {status}")
except OSError as e:
    print(f"Error executing command: {e}")
except Exception as e:
    print(f"An unexpected error occurred: {e}")

Passing Parameters (PyArg_ParseTuple)

The PyArg_ParseTuple() function (and related PyArg_ParseTupleAndKeywords()) is used inside your C functions to extract arguments passed from Python.

Syntax:

int PyArg_ParseTuple(PyObject *args, const char *format, ...);

args: The PyObject* tuple containing positional arguments.

format: A format string specifying the expected argument types.

...: Pointers to C variables where the parsed values will be stored.

Returns true on success, false on failure (and sets a Python exception).

Common Format Codes:

Code	Type	Meaning
s	string	Python string -> const char* (UTF-8 assumed). Use es or et for explicit encoding control.
z	string or None	const char* or NULL.
i	integer	Python int -> C int.
l	integer	Python int -> C long.
L	integer	Python int -> C long long.
d	float	Python float -> C double.
f	float	Python float -> C float.
O	object	Python object -> PyObject* (borrowed reference). Use O! to type-check, O& for custom converter.
p	boolean	Python bool -> C int (non-zero for True).
(items)	tuple	Requires sequence, parses items according to enclosed format.
|		Indicates subsequent arguments are optional.
:		Followed by function name for error messages.
;		Followed by full error message.

Example Parsing Multiple Arguments:

static PyObject * my_func(PyObject *self, PyObject *args) {
    int i_val;
    double d_val;
    const char *s_val;
    PyObject *o_val;

    // Expect an int, double, string, and any object
    if (!PyArg_ParseTuple(args, "idsO", &i_val, &d_val, &s_val, &o_val)) {
        return NULL; // Error: PyArg_ParseTuple sets exception
    }

    printf("Parsed: int=%d, double=%f, string='%s'\n", i_val, d_val, s_val);
    // Process o_val using PyObject_Print or other API functions

    Py_RETURN_NONE; // Return None if successful
}

Returning Values (Py_BuildValue)

To return values from your C function back to Python, construct a PyObject* using functions like PyLong_FromLong, PyFloat_FromDouble, PyUnicode_FromString, or the versatile Py_BuildValue().

Py_BuildValue() uses a format string similar to PyArg_ParseTuple but takes C values as input.

Syntax:

PyObject* Py_BuildValue(const char *format, ...);

Returns a new reference to the created Python object, or NULL on error.

Common Format Codes:

Code	C Type	Meaning
s	const char*	C string -> Python string (UTF-8 assumed).
z	const char* or NULL	C string or NULL -> Python string or None.
i	int	C int -> Python int.
l	long	C long -> Python int.
L	long long	C long long -> Python int.
d	double	C double -> Python float.
f	float	C float -> Python float.
O	PyObject*	Passes object, increments reference count (owned by caller).
N	PyObject*	Passes object, steals reference (caller loses ownership).
(items)	C values…	Builds a Python tuple from C values.
[items]	C values…	Builds a Python list.
{items}	key, value, …	Builds a Python dict (key1, val1, key2, val2…).

Example Returning Values:

static PyObject * foo_add_subtract(PyObject *self, PyObject *args) {
    int a, b;
    if (!PyArg_ParseTuple(args, "ii", &a, &b)) {
        return NULL;
    }
    // Return a tuple (a + b, a - b)
    return Py_BuildValue("(ii)", a + b, a - b);
}

Reference Counting

Python uses reference counting for memory management. In C extensions, you must manage reference counts correctly using Py_INCREF(obj) to increment and Py_DECREF(obj) to decrement the count. Failure to do so leads to memory leaks or crashes.

Functions like PyLong_FromLong and Py_BuildValue return new references (caller owns them). PyArg_ParseTuple with O format code returns a borrowed reference (don’t decref unless you incref it first). When returning an object, ownership is generally passed to the caller.

Managing references carefully is one of the most complex aspects of C extension writing. Tools like Cython handle this automatically.