How Procedural Macros Work

Your team generates the same boilerplate by hand: every new settings struct carries a pub fn name(&self) -> &'static str that returns the struct's name, and every few weeks someone adds a struct and forgets the method. The fix is to generate that impl. In Rust that means a procedural macro, and the first thing you discover is that it cannot live in the crate it helps.

A procedural macro is an ordinary Rust function that the compiler runs while it compiles you. Its contract is one line: it receives a proc_macro::TokenStream — the tokens of whatever the macro was invoked on — and returns a TokenStream that replaces the invocation in the caller's source. No runtime, no reflection: token stream in, token stream out, executed inside the rustc process at compile time.

That contract dictates the crate shape:

# crates/shapely-derive/Cargo.toml
[lib]
proc-macro = true          # the one line that makes this a proc-macro crate

[dependencies]
syn = { version = "2", features = ["full"] }
quote = "1"
proc-macro2 = "1"

proc-macro = true does three jobs at once. It forces the crate to be built for the host target, because the macro must execute on the machine running the compiler even when you cross-compile to something else. It restricts the crate's exports to exactly three item kinds — a function-like #[proc_macro], an attribute #[proc_macro_attribute], and a derive #[proc_macro_derive] — a plain pub fn helper is rejected with a "proc-macro crate types currently cannot export any items other than..." error. And it makes those macros visible only to crates that depend on this one, after it has been compiled.

Three entry-point forms map to three call syntaxes, and you will recognize all three from the standard library:

make_answer!();              // #[proc_macro]           — function-like
#[route("GET", "/health")]   // #[proc_macro_attribute] — item rewriting
#[derive(Config)]            // #[proc_macro_derive]    — impl generation
struct Config { retries: u32 }

The minimal version, before any parsing toolchain, is a function that returns tokens:

use proc_macro::TokenStream;

#[proc_macro]
pub fn make_answer(_input: TokenStream) -> TokenStream {
    "fn answer() -> u32 { 42 }".parse().unwrap()   // string -> tokens
}

Parsing a string works, but it hands you no structure and no source locations — which is exactly why syn and quote exist, and why the next two chapters build on them.

Two consequences bite in practice. You cannot use your own macro inside its own crate: the macro must be compiled before any code that invokes it, which is why the ecosystem splits into serde (types and traits) and serde_derive (the macro), with serde re-exporting the derive so you name one dependency. And macro code runs in the compiler: a panic! inside your macro surfaces as a compile error carrying your message, while a println! goes to the compiler's stdout and scrolls past in build noise.

Interview framing: "What is the difference between macro_rules! and a procedural macro?" comes up often. The crisp answer: macro_rules! is declarative token pattern matching run by the compiler's macro engine, with hygiene baked in; a proc macro is arbitrary Rust running as a program over tokens, so it can parse, inspect, and decide — at the price of its own crate, weaker hygiene, and compile time spent on every invocation. The preceding topic, Attribute Macros, is this same machinery with an item handed to you; this topic is the crate and toolchain underneath it.

Hand the compiler's token stream a struct definition and you get back something like:

struct # [derive (Debug)] pub Config { retries : u32 , endpoint : String }

A TokenStream is flat: tokens, identifiers, punctuation, and balanced delimiter groups, with every comment and most whitespace already thrown away. It has no notion of "field", "visibility", or "type" — those concepts only exist once something parses the stream. Two crates split the job of getting structure back.

syn parses a TokenStream into a typed abstract syntax tree of Rust syntax — DeriveInput for struct/enum/union input, ItemFn for functions, Expr for expressions, File for whole modules. The macro author never writes recursive-descent parsing by hand:

use proc_macro::TokenStream;
use syn::{parse_macro_input, DeriveInput};

#[proc_macro_derive(Config)]
pub fn derive_config(input: TokenStream) -> TokenStream {
    let ast: DeriveInput = parse_macro_input!(input as DeriveInput);
    // ast.ident, ast.data (Struct / Enum / Union), ast.attrs — typed fields, not tokens
    unimplemented!()
}

parse_macro_input! earns its keep on failure: instead of you matching every malformed case, it returns early from your function with a compile_error! whose span points at the caller's bad tokens. For your own validation you do the same by hand:

let err = syn::Error::new_spanned(&ast.ident, "Config structs need at least one field");
return err.to_compile_error().into();   // emits compile_error! tokens

A span is the source location attached to a token. It is the channel through which your macro's diagnostics land on the user's code rather than inside your crate, and it is also what quote uses to decide hygiene (chapter 3).

quote goes the other direction: Rust syntax written as a template, re-serialized into tokens.

use quote::quote;

let name = &ast.ident;
let expanded = quote! {
    impl #name {
        pub fn type_name() -> &'static str { stringify!(#name) }
    }
};
// expanded: proc_macro2::TokenStream  ->  .into()  ->  proc_macro::TokenStream

Every #ident is replaced by the tokens of the value bound to that name — anything implementing ToTokens. Two practical details. First, quote! produces proc_macro2::TokenStream, not the compiler's own type; the proc_macro2 crate is a faithful standalone reimplementation so syn and quote can run in unit tests, in rust-analyzer, and in tools like cargo expand, with .into() bridging back into proc_macro at the boundary. Second, building source with format! and .parse() is possible but loses spans, so errors point at the macro instead of the code — quote keeps your diagnostics anchored to the caller.

Which AST type you parse depends on what the macro is invoked on:

That last row is where proc macros leave macro_rules! behind. Implementing Parse means writing fn parse(input: ParseStream) -> Result<Self> and pulling tokens with input.parse::<Ident>()?, input.peek(Token![,]), and fork/lookahead — a real parser for a small DSL, with every failure becoming a spanned error for free. macro_rules! matchers cannot express lookahead or nested custom grammar; this is the capability you are buying.

Token output also has no pretty-printing; the compiler treats it as tokens, so raw expansion looks like the soup above. When readability matters, pipe the syn AST through prettyplease::unparse to emit formatted Rust — the trick behind tools that show expanded code as a real file. And note the phase: expansion runs before type checking, so anything your macro gets wrong structurally shows up as type errors in the user's file, blamed on the invocation — which is exactly why validating structure in the macro pays off.

Here is the whole workflow on a small but real macro: getter_for!(name) expands to pub fn get_name() -> &'static str { "name" }.

// crates/shapely/src/lib.rs
use proc_macro::TokenStream;
use quote::quote;
use syn::{parse_macro_input, Ident};

#[proc_macro]
pub fn getter_for(input: TokenStream) -> TokenStream {
    let field = parse_macro_input!(input as Ident);

    // Validate: a macro that accepts garbage produces better errors than one that
    // silently generates garbage.
    let text = field.to_string();
    if !text.chars().next().unwrap().is_ascii_lowercase() {
        return syn::Error::new_spanned(&field, "getter name must start lowercase")
            .to_compile_error()
            .into();
    }

    let getter = Ident::new(&format!("get_{text}"), field.span());
    quote! {
        pub fn #getter() -> &'static str { #text }
    }
    .into()
}

Consumer side:

use shapely::getter_for;
getter_for!(name);      // expands to: pub fn get_name() -> &'static str { "name" }
getter_for!(42);        // error: getter name must start lowercase — pointed at the 42

Then run the loop that makes proc-macro work tolerable: expand, read, adjust. cargo expand (a separate cargo install cargo-expand tool) compiles and prints the expanded source of your crate in readable form; without it, println!("{}", expanded) inside the macro shows the raw token soup, which is enough to check ordering but painful to review. When the output misbehaves, the sequence is always: print the tokens, confirm what you actually generated, fix the quote! template, re-expand.

Hygiene is the trap the loop will not reveal by itself. quote! stamps generated tokens with Span::call_site(), so identifiers you create resolve as if the user had typed them at the call site. A macro that generates fn get_name() collides with the user's own get_name, and a macro introducing a helper let count = ... next to the user's count does not shadow it hygienically the way macro_rules! locals do. Mitigations: prefix generated names unmistakably, or use Span::mixed_site() for bindings that should behave like macro_rules! hygiene (locals and labels resolve inside the macro; other names at the call site). Unlike Repetition, Hygiene, and Macro Patterns, where hygiene mostly protects you, in proc macros it is mostly on you.

Watch compile time while you iterate: syn's default features parse only derive-style inputs, while features = ["full"] teaches it to parse every item and expression — necessary for attribute macros over function bodies, and the main cost of large builds, since your macro runs once per invocation.

One more habit separates toy macros from maintainable ones: split the pure logic from the entry point. The #[proc_macro] function is a thin shell — you cannot call it from your own tests, because macros are unavailable inside their defining crate. So move the body into a function that takes and returns proc_macro2::TokenStream, and unit-test it directly:

fn expand(field: proc_macro2::TokenStream) -> proc_macro2::TokenStream { /* real work */ }

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn builds_getter() {
        let out = expand(syn::parse_str("name").unwrap()).to_string();
        assert!(out.contains("get_name"));
    }
}

The public entry point then only does .into(), parse_macro_input!, and error conversion. Tests run in milliseconds without spawning a compiler, and CI catches template regressions the way it catches everything else.

Interview framing: expect "show me a proc macro you wrote" more than definitions. Walk through this loop — parse with parse_macro_input!, validate with new_spanned, generate with quote!, verify with cargo expand — and name hygiene and compile time as the two things that bite. Next, the derive form of this machinery: Building a Custom derive.