LISP - Binary I/O

Modern Lisp - Binary I/O

Binary I/O involves reading from and writing to data sources byte-by-byte. This is essential for tasks like processing images, audio files, network packets, or interacting with custom binary file formats. This chapter covers the modern, idiomatic way to handle binary data in Common Lisp.

Binary Streams in Common Lisp

In Common Lisp, all I/O operations happen on streams. A stream is an abstraction that represents a source or sink of data. When opening a file, you must specify the type of data the stream will handle. For binary I/O, we create a binary stream.

This is done by providing the :element-type argument to the open function or, more commonly, the with-open-file macro. A typical value for binary streams is '(unsigned-byte 8), which signifies that the stream will process 8-bit unsigned integers (i.e., individual bytes).

Core Functions for Binary I/O

Common Lisp provides a concise set of functions for binary stream manipulation:

Input Functions

• read-byte: Reads a single byte from a binary input stream. It signals an end-of-file error if the stream is at its end.
• read-sequence: Reads a sequence of bytes from a binary stream into a provided array (like a byte vector). This is far more efficient for bulk data.

Output Functions

• write-byte: Writes a single byte to a binary output stream.
• write-sequence: Writes a sequence of bytes (e.g., from a byte vector) to a binary output stream. This is the preferred method for writing blocks of data.

Best Practice: Managing Files with with-open-file

Always use the with-open-file macro to handle files. It ensures that the file stream is automatically and safely closed, even if an error occurs within its body. This prevents resource leaks.

The syntax is: (with-open-file (stream-variable path-to-file &key :direction :element-type ...))

Example 1: Writing and Reading Single Bytes

Writing a Binary File

This example writes two bytes (a magic number 255 and a version number 1) to a file.

;; The `with-open-file` macro ensures the stream is closed automatically.
(with-open-file (out "my-data.bin"
                  :direction :output
                  :element-type '(unsigned-byte 8)
                  :if-exists :supersede)
  (write-byte 255 out) ; Write a 'magic number'
  (write-byte 1 out))   ; Write a 'version number'

Reading the Binary File

Here, we read the two bytes back. Note the use of read-byte with optional arguments to handle the end-of-file condition gracefully without causing an error.

(with-open-file (in "my-data.bin"
                  :direction :input
                  :element-type '(unsigned-byte 8))
  (let ((magic-number (read-byte in nil :eof))
        (version (read-byte in nil :eof)))
    (unless (or (eq magic-number :eof) (eq version :eof))
      (format t "Magic Number: ~d, Version: ~d~%" magic-number version))
    ;; What happens if we read again?
    (let ((next-byte (read-byte in nil :eof)))
      (if (eq next-byte :eof)
          (format t "End of file reached as expected.~%")
          (format t "Unexpected data found: ~d~%" next-byte)))))

Output

Magic Number: 255, Version: 1
End of file reached as expected.

Example 2: Efficient Bulk I/O with Sequences

Writing and reading byte-by-byte is inefficient for large amounts of data. read-sequence and write-sequence are the correct tools for this job.

;; First, create some data to write.
(defvar *my-data* (make-array 4 :element-type '(unsigned-byte 8)
                                :initial-contents '(72 69 76 79))) ; "HELO"

;; Write the entire sequence to a file
(with-open-file (out "message.dat"
                  :direction :output
                  :element-type '(unsigned-byte 8)
                  :if-exists :supersede)
  (write-sequence *my-data* out))

;; Now, read the data back
(with-open-file (in "message.dat"
                 :direction :input
                 :element-type '(unsigned-byte 8))
  ;; Create a buffer to hold the incoming data.
  (let ((buffer (make-array 4 :element-type '(unsigned-byte 8))))
    (read-sequence buffer in)
    (format t "Read data: ~a~%" buffer)))

Output

Read data: #(72 69 76 79)

Common Pitfalls and Best Practices

• Endianness: Common Lisp itself doesn’t dictate endianness. When writing portable binary files (e.g., network protocols), you must manually manage byte order if you are dealing with multi-byte integers. Libraries like babel can help with this.
• Error Handling: The (read-byte stream nil :eof) pattern is crucial for detecting the end of a file without triggering a condition. The second argument (nil) means “don’t signal an error,” and the third (:eof) is the value to return instead.
• Efficiency: For any data larger than a few bytes, always prefer read-sequence and write-sequence over read-byte and write-byte in a loop.
• Resource Management: Always use with-open-file. It is the single most important practice for safe file handling in Lisp.