~yosh@unix.dog

writing about pratt parsing so I remember it

posted

modified

update 2026-10-29: this post seemed to have been posted on a few link aggregator sites, spawned some discussion, and got bookmarked by people. that’s scary! that’s more attention than I anticipated for this post, especially since I wrote it mainly for myself to come back to as a reference without too much expectation on making it in a “teaching” style. as such, I put on my Due Diligence Hat and overhauled this post to make it more pleasant to read. the high-level outline of it is still the same, and it’s not some complete university-level polished writing, but the details are a bit smoother to follow now & I relate it more to pratt’s original paper. if you’re interested in the more unfiltered brain dump post, it’s available on the wayback machine or my website repository.

also, check out vinipsmaker’s site for more pratt parsing stuff :)


I got bored and decided to start making a C compiler. of course, one of the requirements of this is parsing C expressions1. my only experiences with “serious” parsing before this included:

I heard about this cool thing called “pratt parsing” before, but I never looked into it, so I wanted to try it out for this.

you won’t find much information about pratt parsing in a conventional learning path of computer science. the dragon book, for all it talks about parsing, doesn’t mention it. as far as I’m aware, most programming language courses will probably teach either LR or recursive descent depending on how academic it is. pratt parsing specifically appears to be relegated mostly to compiler development and PLT circles. the most mainstream description I know of it is in crafting interpreters, though wikipedia also mentions it with links to other posts.

I used the crafting interpreters text to get my bearings, but I also wanted to practice reading formal papers more, so I looked up Pratt’s original paper text and tried to exclusively use it as much as possible for implementing it. it did take a bit more time than I’d have liked to wrap my mind around the inner workings of the algorithm and convincing myself that it just works, but it did end up just clicking all at once.

interestingly, it clicked not when internalizing vaughan pratt’s own words, but rather when I noticed how pratt parsing related to the shunting yard algorithm. after that, I was able to understand the angle that vaughan pratt approached the algorithm, and now I see how wonderfully elegant it is :). this post will be organized in a similar manner: relating the shunting yard algorithm to pratt parsing, then seeing how pratt approached it and how it can be very cleanly implemented for really any kind of parsing.

shunting yard

the shunting yard algorithm, when boiled to its primitive form, is quite simple conceptually. there are two kinds of tokens: operators and literals. operators have “binding powers” associated with them—higher binding power means it gets evaluated “before” other operators. you have an input stream, output stream, and a stack for operators.

then, for every token in the input stream:

  • if it’s a literal token, just push it to the output stream
  • if it’s instead an operator token:
    • first, pop off from the stack all operator tokens that have higher or equal binding power than the current token. this could be zero tokens! push these to the output stream
    • after this, push the current operator token to the stack
  • at the end of the token stream, pop off all remaining operator tokens to the output.

for example, here’s a shunting yard implementation in C for a very reduced arithmetic grammar (digits 0-9 and operators + - * /):

C code & output for shunting yard
#include <stdio.h>
#include <ctype.h>

typedef struct {
    char ops[32]; // limit of 32 operators
    int idx; // index of last operator
} Stk;

void push(Stk *s, char op) {
    s->idx += 1;
    s->ops[s->idx] = op;
}
void pop(Stk *s)   { s->idx -= 1; }
char peek(Stk *s)  { return s->ops[s->idx]; }
char empty(Stk *s) { return s->idx == -1; }

int get_bindpower(char op) {
    static const int bps[] = { ['+'] = 1, ['-'] = 1, ['*'] = 2, ['/'] = 2 };
    return bps[op];
}

int main(int argc, char **argv) {
    char *str = argv[1];
    Stk _opstk = (Stk){ .idx = -1 };
    Stk *opstk = &_opstk;

    while (*str != '\0') {
        if (isdigit(*str)) {
            // number
            printf("EMIT '%c'\n", *str);
        } else {
            // operator (assumed to be valid)
            // get_bindpower(peek(opstk)) is the binding power of the last operator
            // get_bindpower(*str) is the binding power of the current operator
            while (!empty(opstk) && get_bindpower(peek(opstk)) >= get_bindpower(*str)) {
                printf("EMIT '%c'\n", peek(opstk));
                pop(opstk);
            }
            push(opstk, *str);
        }
        str += 1;
    }

    while (!empty(opstk)) {
        printf("EMIT '%c'\n", peek(opstk));
        pop(opstk);
    }

    return 0;
}
$ ./a.out '1+5*2+6'
EMIT '1'
EMIT '5'
EMIT '2'
EMIT '*'
EMIT '+'
EMIT '6'
EMIT '+'

the primitive version of the algorithm has the following properties:

  • left-associativity for all operators, as only one binding power is associated with each operator.
  • no proper “expression nesting” with e.g. parentheses
  • only infix and postfix operator support, as operators can not be “deferred” in any way.

the first two problems are relatively easy to solve without much modification. split left and right binding powers for the first one, and attach some special handling of parentheses for the second. however, the third issue is slightly more difficult, along with a few other extensions that pose their own problems. it’s also pretty annoying to handle errors for shunting yard, as all stack unwinding is explicit and a lot of context gets lost from it. it’s a good start, and is certainly fine for many real-world things, but I wouldn’t quite consider it robust. so how does it generalize to pratt parsing?

from shunting yard to pratt

the first insight to understanding pratt parsing is forming a different classification of tokens. instead of “literals” and “operators” from before, there are:

  • “nuds”, which are tokens that don’t expect something to the left. these include numbers (1, 2) and unary prefix operators (-1, ++ptr).
  • “leds”, which are tokens that do expect something to the left. these include all the standard operators from before (+, -, etc.).2

with this, one immediate observation is that any possible expression must begin with a nud. you can’t begin an expression with a token that expects something to its left, as there is no left!

next, the stack. handling operator precedence requires a stack of some kind. in shunting yard, it was explicit. for pratt parsing, the stack is implicit in nature; it’s the call stack!

this sounds handwave-y, but more specifically, moving back to the land of simple arithmetic: every time a number (nud) is consumed, some infix operator (led) or the end of the expression must follow. these infix operators expect an expression of their own on the right side of them. for those operators, we just call the “parse an expression” function again! that recursive call is equivalent to pushing to the stack in the shunting yard algorithm. we provide the recursive call with the binding power of the operator. once the call returns, that’s equivalent to “popping” said operator off the stack in shunting yard. the stack still exists; it’s just implicit now.

putting all this together, if we were to semantically, one-to-one, translate the shunting yard code from before to a pratt-style parser, we would get:

C code & output for a basic arithmetic pratt parser
#include <stdio.h>
#include <ctype.h>
#include <stdlib.h>

typedef struct {
    char *str;
} Parser;

char cur(Parser *p) { return *p->str; }
void bump(Parser *p) { p->str += 1; }

int get_bindpower(char op) {
    static const int bps[] = {
        ['+'] = 1, ['-'] = 1, ['*'] = 2, ['/'] = 2
    };
    return bps[op];
}

// instead of starting off with an immediate abstraction of
// code paths and whatnot, all the code is inlined into expr() so
// the control flow is more explicit. this will be abstracted later.
void
expr(Parser *parser, int last_operator_bindpower) {
    // all expressions *must* start with a "nud" token.
    // in this case, our only nuds are digits.
    if (isdigit(cur(parser))) {
        printf("EMIT '%c'\n", cur(parser));
    } else {
        fprintf(stderr, "expression doesn't start with a number\n");
        abort();
    }

    bump(parser);

    while (cur(parser) != '\0') {
        // assumed to be a valid binary operator (like the shunting yard code)

        // at this point, we're at an infix/postfix operator.
        // this is what pratt calls "leds" in his original paper.
        // when this check is false, it's kind of like "unwinding" the stack in shunting yard.
        char op = cur(parser);
        int current_operator_bindpower = get_bindpower(op);
        if (current_operator_bindpower >= last_operator_bindpower) {
            return;
        }

        bump(parser);
        expr(parser, current_operator_bindpower);

        printf("EMIT '%c'\n", op);
    }
}

int main(int argc, char **argv) {
    Parser parser = (Parser){ .str = argv[1] };

    // the initial call should be with a binding power less than that of any operator
    expr(&parser, 0);

    return 0;
}
$ ./a.out '1+5*2+6'
EMIT '1'
EMIT '5'
EMIT '2'
EMIT '*'
EMIT '+'
EMIT '6'
EMIT '+'

on its own, this doesn’t look too exciting. I made the stack implicit. big whuff. but! there are two neat things here:

  • “prefix” operators have first-class handling now in the first half of expr(), almost for free!
  • operators can have arbitrary code assigned to them that get run immediately. this is what allows for that recursive call to just work out.

these combined allow for simple & straightforward extensibility. before extending it, a few more observations:

  • if we have a token that represents “end of an expression”, then the while (cur(parser) != '\0') check is completely redundant; we can just imagine that the “end of expression” token is a led with the lowest binding power
  • we can shunt all our inlined code for “nud” and “led” handling to their own subroutines for cleanliness.

so, refactoring with those in mind, and adding parentheses, we get:

"final" pratt parser C code
#include <stdio.h>
#include <stdlib.h>

typedef struct {
    char *str;
} Parser;

char cur(Parser *p) { return *p->str; }
void bump(Parser *p) { p->str += 1; }

typedef void (*ParserFunc)(Parser *);

int
get_bindpower(char op) {
    static const int bps[] = {
        ['\0'] = 0, [')'] = 0,
        ['+'] = 1,  ['-'] = 1, ['*'] = 2, ['/'] = 2
    };
    return bps[op];
}

// forward declaration of expr() for calling in nud/led functions
void
expr(Parser *parser, int last_operator_bindpower);

// nud functions
void
paren(Parser *parser) {
    bump(parser);
    expr(parser, 0);
    if (cur(parser) != ')') {
        fprintf(stderr, "parentheses expression doesn't end with )\n");
        abort();
    }
    bump(parser);
}

void
num(Parser *parser) {
    printf("EMIT '%c'\n", cur(parser));
    bump(parser);
}

ParserFunc
get_nudfunc(char nud) {
    static const ParserFunc nuds[] = {
        ['('] = paren,
        ['0'] = num, ['1'] = num, ['2'] = num, ['3'] = num, ['4'] = num,
        ['5'] = num, ['6'] = num, ['7'] = num, ['8'] = num, ['9'] = num,
    };
    return nuds[nud];
}

// led functions
void
binary(Parser *parser) {
    char op = cur(parser);
    bump(parser);
    expr(parser, get_bindpower(op));
    printf("EMIT %c\n", op);
}

ParserFunc
get_ledfunc(char led) {
    static const ParserFunc leds[] = {
        ['+'] = binary, ['-'] = binary, ['*'] = binary, ['/'] = binary,
    };
    return leds[led];
}

// the main expression parsing call
void
expr(Parser *parser, int last_operator_bindpower) {
    ParserFunc nud_func = get_nudfunc(cur(parser));
    if (nud_func == NULL) {
        fprintf(stderr, "invalid nud: %c\n", cur(parser));
        abort();
    }
    nud_func(parser);

    // condition is now reversed, since this is the condition to stay in the loop
    // rather than break out of the loop.
    while (last_operator_bindpower < get_bindpower(cur(parser))) {
        ParserFunc led_func = get_ledfunc(cur(parser));
        if (led_func == NULL) {
            fprintf(stderr, "invalid led: %c\n", cur(parser));
            abort();
        }
        led_func(parser);
    }
}

int main(int argc, char **argv) {
    Parser parser = (Parser){ .str = argv[1] };

    // the initial call should be with a binding power equal to the "end token"
    expr(&parser, 0);

    return 0;
}
$ ./a.out '1+5*(2+6)'
EMIT '1'
EMIT '5'
EMIT '2'
EMIT '6'
EMIT '+'
EMIT '*'
EMIT '+'

from here, it’s pretty simple to see what can be done to change extend functionality. you get arbitrary pre-operator and post-operator code paths for every single “nud” or “led” possible. you have some arbitrary check for determining if one operator dominates another. you don’t even need to use “binding powers”! it’s the same power of recursive descent, but both easier to hack on and doesn’t nest nearly as much! how wonderful!

most implementations provide a bit more unified abstraction, where tokens are organized into “rules” and looked up to get its nud function, led function, and various binding powers or properties all at once. this is described in crafting interpreters and an example in my snippets repo. my example is for the same arithmetic grammar with the inclusion of a factorial operator (!).

hope this helps anyone else. happy parsing :)


  1. specifically, §6.5 Expressions in the C23 standard (technically that link is not The Proper C Standard, but a draft with one trivial difference to the proper standard) ↩︎

  2. for the curious, “nud” stands for "null denotation" and “led” for "left denotation" in pratt’s paper. ↩︎

back