LISP - Macro Characters

Lisp - Reader Macros and Macro Characters

One of Common Lisp’s most powerful and unique features is its extensible parser, known as the reader. Macro characters are special characters that trigger custom parsing logic, called a reader macro, during the read phase. This allows you to extend Lisp’s syntax itself, creating domain-specific languages (DSLs) or convenient shorthand notations.

The Role of the Lisp Reader

Before Lisp code is evaluated or compiled, the reader processes the source text and transforms it into Lisp objects (S-expressions). When the reader encounters a macro character, it doesn’t read it as a normal character. Instead, it invokes the associated reader macro function, which then takes control, consumes subsequent text from the input stream, and returns a Lisp object.

Standard Macro Characters

You are already familiar with many standard macro characters:

• ( (Left Parenthesis): Starts reading a list. Recursively reads objects until a matching ) is found.
• ) (Right Parenthesis): Terminates reading a list.
• ' (Single Quote): A shorthand for the quote special form. 'foo is read as (quote foo).
• ; (Semicolon): Starts a single-line comment. The reader ignores the rest of the line.
• " (Double Quote): Starts and ends reading a string literal.
• ` (Backquote): Initiates a quasiquote template, allowing for structured code generation with selective evaluation.
• , (Comma): Used inside a backquoted expression to mark sub-expressions for evaluation.
• # (Hash/Dispatch): A dispatching macro character. What it does depends on the character that follows it (e.g., #' for a function object, #\ for a character literal, #(...) for a vector).

Defining a Custom Macro Character

You can define your own macro characters using set-macro-character. This function associates a character with a handler function that will be executed by the reader.

Example: A JSON-like Object Reader

Let’s create a simple reader macro. We will make { a macro character that reads a JSON-style object like {"key": "value"} and transforms it into a Lisp hash table.

;; First, define the reader function. It takes the stream and char as arguments.
(defun json-object-reader (stream char)
  (declare (ignore char)) ; We don't need the '{' character itself

  ;; The core logic will be to read a list of key-value pairs...
  (let ((pairs (read-delimited-list #\} stream t)))
    ;; ...and transform it into an expression that creates a hash-table.
    `(let ((table (make-hash-table :test #'equal)))
       ,@(loop for (key value) in pairs
               collect `(setf (gethash ,key table) ,value))
       table)))

;; Now, associate the '{' character with our reader function.
(set-macro-character #\{ #'json-object-reader)

;; We also need to tell the reader that '}' closes our form.
(set-macro-character #\} (get-macro-character #\) nil))

;; Let's use our new syntax!
(let ((my-data '{"name" "John Doe" "age" 30}))
  (print (gethash "name" my-data)))

How It Works

1. When the reader sees {, it calls json-object-reader.
2. (read-delimited-list #\} ...) reads all the forms inside the braces until it finds a }.
3. The function constructs a piece of Lisp code using a backquote. This code, when evaluated, will create and populate a hash table.
4. The reader replaces the original {...} text with this generated code.
5. Finally, Lisp evaluates this generated code, creating the hash table.

Output

"John Doe"

Best Practices and Warnings

While incredibly powerful, modifying the reader should be done with care:

• Readability: Non-standard syntax can make code difficult for others (and your future self) to read and understand.
• Portability: Overuse of custom reader macros can tie your code to a specific environment or project.
• Use Case: They are best used for creating embedded DSLs or for significant syntactic conveniences that genuinely improve productivity within a well-defined context.

In modern practice, it’s often preferred to use regular Lisp macros (defmacro) for DSLs, as they operate on standard Lisp syntax and are easier to reason about. Reader macros are reserved for when you truly need to change the fundamental syntax of the language.