Parser 5: PEG grammar without memoization

This is loosely based on Daan Leijen and Erik Meijers’ 2001 paper [1]. I say loosely as it lacks all the important elements demonstrated in the paper — efficiency, useful error messages, etc. — but it is a monadic parser.

(Why do you want a monadic parser? This author explains better than I)

Here, the type of a parser is ‘string -> maybe (a, string)’, where ‘a’ is your parse tree, and the result string is the remaining input. If the parse fails, None is returned instead of a tuple.

1
2
3
4
5
6
7
8
9
10
11
12
13

# simplified monadic (PEG) parser. no memoization, some backtracking.
#  parser :: str -> maybe (value, str)
 
def ret (value):
    return lambda s: (value, s)
 
def bind (p, *fs):
    def parse (s):
        res = p(s)
        for f in fs:
            res = res and f(res[0])(res[1])
        return res
    return parse

These are the monad operators return and bind. ‘Ret(v)’ produces a parser that consumes and empty string, producing the parse tree ‘v’, but the juice is in ‘bind’.

‘Bind(p, f)’ glues together a parser and a function. A parser ‘p’ consumes some input and produces a parse tree ‘v’. This value ‘v’ is then passed to a function ‘f’, returning a new parser to consume the remaining input. That is, ‘f’ chooses, based on the parse tree so far, which language to use to interpret the rest of the input.

This is extremely powerful, and permits monadic parsers the ability to recognize classes of context sensitive grammars, such as parsing XML, or loading new languages on the fly as you parse (Perl6, anyone?) or perhaps backtracking to recover if an evaluation of the parse tree fail (Perl5, anyone?).

It can also inhibit various parser optimizations if you’re not careful.

15
16
17
18
19
20
21
22
23

never = lambda s: False
 
def alt (*ps):
    def parse (s):
        for p in ps:
            res = p(s)
            if res: return res
        return False
    return parse

‘never’ is the MonadZero value, a sort of additive identity for the ‘alt’ MonadPlus operator. ‘alt’ produces a parser that will recognize any of the languages passed to it. ‘never’ recognizes nothing, so e.g. ‘alt(p, never)’ is equivalent to ‘p’ in the same way as ‘bind(p, ret)’ is equivalent to ‘p’.

You can ignore these equivalences for now — they’re useful though when it comes time to optimize grammars, but that’s not on today’s agenda.

From here on out, we have a handful of various other operators:

25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

empty = ret(None)
 
def then (p, *ps):
    return bind(p, lambda x: bind(then(*ps), lambda rest: ret([x]+rest)) if ps else ret([x]))
 
def repeat (p, res=[]):
    return alt( bind(p, lambda v: repeat(p, res+[v])),
                ret(res) )
def repeat1 (p):
    return bind(p, lambda v: repeat(p, [v]))
 
def pred (f):
    def parse (s):
        if s and f(s[0]): return (s[0], s[1:])
        else: return False
    return parse
 
def option (p):
    return alt(p, empty)
 
def repn (p, n):
    return then(*([p]*n))
 
def char (ch):
    return pred(lambda c: c==ch)
 
def literal (s):
    return then(*(char(c) for c in s))
 
def delay (fn):
    return bind(empty, lambda _:fn())
 
end = lambda s: False if s else (None, s)

‘Empty’ recognizes the empty string, ‘then’ chains parsers in sequence, ‘repeat’ is Klein star, ‘repeat1′ likewise plus. ‘option’ recognizes one or zero of a language, while ‘repn(p,n)’ recognizes exactly n occurrences of p.

‘Pred(fn)’ consumes one character ‘c’ for which ‘fn(c)’ is true (e.g. using str.isalpha). ‘char(c)’ recognizes the character ‘c’, and ‘literal(s)’ the sequence of characters in the string ‘s’. ‘delay(lambda:p)’ recognizes ‘p’ — and is merely an artifact of strict evaluation, as ‘p’ might not yet be defined. Finally, ‘end’ recognizes only the end of the input string; not very useful in today’s example, but it can be handy in parsers that perform lookahead.

You may not have noticed as it whizzed by, but this parser doesn’t perform full backtracking. The ‘alt’ operator attempts to parse using the first language, eagerly, and if it succeeds, returns immediately. If a subsequent parse operations should fail, no backtracking is performed to consider the alternate path. As such, this parser combinator library might be classified as a parsing expression grammar. Contrast this with context free grammars, where both sides of the alternation are considered equal for backtracking / lookahead purposes.

Size: ~60 lines

Parser 7: CSG with partial memoization

This next parser adds a couple tweaks. First, the type of the parser has changed to ‘string -> lcons of (a, string)’, where lcons is a kind of lazy list.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

class lcons (object):
    def __init__ (self, iter): self.iter = iter; self.value = None
    def force (self):
        if self.iter:
            try:
                self.value = (self.iter.next(), lcons(self.iter))
            except StopIteration:
                self.value = None
            self.iter = None
        return self.value
    def empty(self): return self.force() == None
    def head (self): return self.force()[0]
    def tail (self): return self.force()[1]
    def __iter__ (self):
        while not self.empty():
            yield self.head()
            self = self.tail()
 
def memoize(fn, _memo=dict()):
    def replc (s):
        k = (fn, s)
        if k not in _memo:
            _memo[k] = lcons(iter(fn(s)))
        return _memo[k]
    return replc

‘lcons’ only role is ensure that an iteration is computed only once; otherwise, we’ll be using Python iterators. Speaking of memoization, we’ll also memoize of our monad parser so that we don’t recompute them on backtrack. This … will consume quite a bit of memory, but will reduce the backtracking costs in some key places. You can disable this with otherwise no change in behavior.

27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

def ret (value):
    return lambda s: [(value, s)]
 
def bind (p, f, *fs):
    @memoize
    def parse (s):
        for res1 in p(s):
            for res2 in f(res1[0])(res1[1]):
                yield res2
    if fs: return bind(parse, *fs)
    return parse
 
def alt (*ps):
    @memoize
    def parse (s):
        for p in ps:
            for res in p(s):
                yield res
    return parse
 
def then (p, *ps):
    more = then(*ps) if ps else None
    return bind(p, lambda x: bind(more, lambda rest: ret([x]+rest)) if ps else ret([x]))

The monad parser combinators have received a facelift. ‘bind(p,f)’ now evaluates each possible parse of ‘p’, passing the result to ‘f’ and then enumerating the results from that parser, lazily yielding them to be memoized by ‘lcons’.

‘Alt’ has similarly been updated — each possible parse result from each parser is yielded in turn.

‘Then’ gets a minor performance improvement. As we’re memoizing parse results, we get a boost by reusing the same tail parser each time.

‘Ret’, ‘never’, ‘end’, ‘char’, and ‘pred’ are each updated to return arrays of results, but are otherwise unchanged. All the other parser combinators remain as before.

As a result of these changes, parser7 now supports nearly the class of context free grammars (CFG), save its inability to handle left recursion (perhaps the techniques in [2] could fix that). It can even handle ambiguous grammars, returning a parse forest instead of a parse tree. Of course, nothing is free — parser7 is slower than parser5, and most useful CFGs can be rewritten as PEGs with some effort.

Above and beyond CFGs, this parser continues to provide monadic bind, so can continue to parse a number of useful languages from the class of context sensitive languages. For example, the mini-xml parser in the samples bellow runs great under both libraries.

It’s a memory hog, but it’s also ~90 lines of vanilla Python.

Full source, samples, references

I’ve put the parser combinators and some samples up on codepad.org so you can get a feel for what the output looks like. It’s not pretty, but it’s pretty well structured:

• parser5: http://codepad.org/P5l2l6dm
• parser7: http://codepad.org/pmpqp1wI

[1] Parsec: Direct Style Monadic Parser Combinators For The Real World. Daan Leijen, Erik Meijer (2001).

[2] Packrat Parsers Can Support Left Recursion. Alessandro Warth, James R. Douglass, Todd Millstein (2007).

• “Yacc is dead” (ltu discussion)
• “Yacc is not dead“
• “Yacc is dead: and update“

Like Schrodinger’s cat, Yacc seems to be indeterminately alive or dead (though the last article conclusively opened the box for me).

Concepts are supposed to bring soundness in through constrained templates. Concepts look awfully a lot like type classes; they export functions and types, and are parameterized, and act as constraints on generic functions and other concepts. I checked the draft specification [2], and it even seems to permit parameterizing concepts on type constructors, just like Haskell! (er, C++ calls them class templates, not type constructors. to-may-to, to-mah-to)

But I worried I may be mistaken about concepts, as I’ve searched through google and literature and have yet to find a single example in literature demonstrating the use of template template concepts.

In case you’re curious what this might look like, here’s an educated guess:

concept Monad<template <> class m> {
  template<typename T, typename U>
  m<U> mbind(m<T>, function<m<U>(T)>);
 
  template<class T>
  m<T> mreturn(T);
};
 
template <template <typename> class m,
          class T,
          class U,
          class Iter>
requires M<m>
requires InputIterator<Iter, U>
m<T> foldM(Iter begin, Iter end, T i, function<m<T>(T, U)> f)
{
    if(begin == end)
        return mreturn(i);
    else
        return mbind(f(i, *begin), [=](T result){
            Iter next = begin;
            ++next;
            return foldM(next, end, result, f);
        });
}

Have template template concepts been covered thoroughly somewhere, and I’ve just missed it?

• [1] “C++ templates/traits versus Haskell typeclasses” (2005), by Sunil Kothari, Martin Sulzmann.
• [2] “Proposed Wording for Concepts (Revision 5)” (2008).
• [3] “Trip Report: Exit Concepts, Final ISO C++ Draft in ~18 Months” (2009), Herb Sutter.
• [4] “ConceptClang: An Implementation of C++ Concepts in Clang” [pdf]

Online REPs and REPLs

Today there’s 61 different languages on that list. That many, there’s gotta be at least one that strikes your fancy.

Best batch execution site: ideone.com with 50 unique languages.

Best interactive REPL site: repl.it with 16 unique languages.

But then turns around and cries out “one way to do it“, leaving the programmer perplexed as to how Guido van Rossum thought we should do things. For example, Guido hates tail calls, so recursion isn’t the one way to do it that he picked (note that his blog post and followups contain a large number of factual errors; read it as an opinion piece only).

During these fits, Python suffers itself to be a bondage and discipline language.

Apparently there is hope. One bit that causes me particular pain is that nested functions cannot rebind variables in the outer scope; only read from them. A case in point:

def f():
  x = 1
  def doubleIt():
    x *= 2 # local variable 'x' referenced before assignment
  doubleIt(); doubleIt();
  return x;

However, I was reading a piece on Factor, and it mentions that this restriction is lifted in Python 3.0. I’m still on 2.6 (the only differences I had been aware of were the somewhat arbitrary swapping of the ‘/’ and ” operators, and the insulting removal of map, filter, and reduce from Python 3.0). However, fixing this scoping rule, even if an extra keyword is needed, sure would be convenient for me.

Actually, there are a lot of improvements:

• Various APIs returns views instead of mutable list (copies).
• sorted is now built in. (I’ve had to write that myself so many times…)
• Apparently map and filter are still here (though he still went and put reduce all the way off in functools. I guess you can’t have it all -_-)
• Set literals, yay!

Now, if Guido would be so kind to add proper statement support to lambdas, I’ll switch ^_^. Oh, but Guido hates lambdas. Oh well, next time maybe.

– signed an ambivalent Python user

They reported that if they compiled code which calculated: 1130149156 >> -05701653 it generated different results on 32bit and 64bit operating systems. On 32bit machines it reported 0 but on 64bit machines, it reported 0x21a.

This is one of various areas of “undefined behavior” for which you can ask 2 engineers what it should do and get 3 answers, at least one being that “but I know there must be one!” I think I know where at least 2 of the answers are coming from …

The Hardware shift

On x64, Larry mentions that the sar instruction is used to carry out the operations. This instruction is not new to the x86 architecture, and in that architecture line’s early days you did not waste die space. So what’s cheap?

Enter the barrel shifter. It’s cheap, it’s small, you pick the maximum power-of-2 shift you want, and that’s what you pay for. For a 16-bit shift, you can support shifts of 0 to 15 and pay for 4 stages. If you want to shift out all 16, you might think “add another stage” — but it gets worse! It only takes rotation as input. If you need to identify greater shifts than 31, then you’re toast.

No, what you consider doing is detect shifts > 15, and add a an extra stage to mask away the entire output. But wait, it’s 1972, and you don’t have either the die space nor the extra cycles to waste on a conditional masking stage (masking to 4 bits of shift input is free — you just don’t connect the other 12 input bits to anything). You just tell the programmer to never shift too much, use the bottom most bits, and ignore the rest. (1 >> 128) == 1.

In time the processor is expanded to support 32-bit quantities and 64-bit quantities, but each time you need to remain compatible, so the sar instruction continues to mask away the upper bits.

The Programming Language

I’m gonna gloss over the whole “undefined behavior” thing. Enough others have discussed it. Even languages or implementations which aim to minimize it will occasionally miss edge cases or fall short.

And C takes a very brutal route of making a great many things “undefined behavior” in the name of permitting performance shortcuts. For instance, signed arithmetic overflow yields undefined behavior. This might seem strange now, but at the time 2′s compliment arithmetic was not common. Choosing 2′s compliment wrap-around would have introduced perf penalties on 1′s compliment architectures as they did not generally include native 2′s compliment instructions, and vice-versa. A modern spin on this are special purpose DSP chips that default to saturating arithmetic.

In any case, I imagine a similar case stands for shifting. One architecture may prefer precise shifting, but x86 prefers to take shifts in modulo. Rather than specify one behavior and penalize compilation on all other hardware, the C language says instead “this area off limits — if you want a particular slow-but-precise behavior, implement it yourself”

The Software Shift

The 32-bit x86 CRT routine mentioned was _allshr (the extra leading underscore comes from cdecl calling convention on x86), and you can find its source in your Visual C++ product installation, under %programfiles%/Microsoft Visual Studio 10.0/vc/crt/src/intel/llshr.asm. It leads off:

;
; Handle shifts of 64 bits or more (if shifting 64 bits or more, the result
; depends only on the high order bit of edx).
;
        cmp     cl,64
        jae     short RETSIGN

Interesting — for large shifts, it bails early. Following it is another branch, switching between different implementations for 0-31 and 32-63 bit. In any case, the resulting 3 code paths are short, with the cl >= 64 path being the shortest of all.

Now, the designer could have considered masking instead of comparing — but for random shift amounts, that would have been a longer code sequence to execute*. And by most reasonable interpretations, returning the sign bit is the right answer.

So there you have it

Each implementation is most efficient and reasonable, when and where it was designed, and both are permitted by a language standard that valued efficiency above most else. And in that respect, both implementations were perfect … at least, they were when originally written.

* yes, the conditional branch is probably a bigger cost than the savings… today. While I don’t know when this code was written, it mentions PROC NEAR, leading me to think that perhaps it comes from a time before superscalar architectures, where branches were cheap(er) and arithmetic was costly. You might be want to change it now, but that decision opens up back-compat questions.

Well, maybe not for typographers. I get the impression this is all entry level stuff.

For instance,

I understand that many people were taught early in life to double-space their sentences. I was too. But double-spacing is a habit held over from the typewriter age. It has never been part of standard typography. Because typewriter fonts were unusually proportioned, a double space helped set off sentences better. Today, since we don’t use typewriter fonts, double spaces aren’t necessary or desirable.

Let’s see that paragraph again, but with double spaces:

I understand that many people were taught early in life to double-space their sentences.   I was too.   But double-spacing is a habit held over from the typewriter age.   It has never been part of standard typography.   Because typewriter fonts were unusually proportioned, a double space helped set off sentences better.   Today, since we don’t use typewriter fonts, double spaces aren’t necessary or desirable.

Wow. Not only do they teach you, but they then demonstrate, right there in the middle of the lesson.  It can’t be easier.

(Yes, I had to rewrite this post to remove all those double-space-after-periods that I too had been trained to type)

http://igoro.com/archive/gallery-of-processor-cache-effects/

When someone bugs me about “X is too slow, because it has to make a virtual call”, and I get annoyed, it’s because a hot virtual call is an overhead of some dozen cycles or so.  Missing the cache?  In the thousands.  Don’t get me wrong, virtual calls can matter for many reasons, but that all flies out the window the moment you’re working on any non-trivial sized data set.  If your objects are 100′s of bytes large, you don’t worry about the virtual calls, you worry about shuffling their member slots around to squeeze more out of your caches.

“It better not allocate”

My other favorite perf quote from this month: “It better not allocate — this call needs to take 100 microseconds or less”.  On my dev box, on the default Win7 heap, an uncontended small allocation (and the pairing free) is 120 ns — or 0.12 microseconds.  My personal favorite small object allocator can hit down to 0.020 microseconds sustained.

We could allocate thousands of objects per call and still come in under budget.

"We're having a problem sending email out of the department."

"What's the problem?" I asked.

"We can't send mail more than 500 miles," the chairman explained.

I choked on my latte.  "Come again?"

A must read for CS/IT folks.

Anyway, he also posted a his personal history of programming language study, and it got me thinking about my own personal programming languages history.

It all started with Logo…

1992: Logo/LogoWriter.  After a basic intro class (4th grade) of moving a turtle around, I would occasionally try out various programming exercise cards, where I learned about variables, loops, and functions.  Unfortunately, the advanced exercises required a version of the software the school didn’t have.

1995: VB for dummies, and a cloned copy of a Visual Basic 3 floppy diskette.  Remember floppy disks?  Yeah, me neither.  First real introduction to arrays, graphics and UI controls.  Wanted to make a clock app, so I learned trig early from my dad with sketches of triangles on a post-it note.  Some time later got to enjoy VB 6, GDI, and BitBlt for bitmapped graphics.  I remember hating BitBlt, it seemed to have a really annoying set of params; I wished it was written in a real programming language, like VB for instance.

1999: C, Java.  I don’t remember the exact order, but I made the jump from VB to C (with thanks to K&R), and mode 13h graphics.  That was the simple mode right?  First introduction to algorithms, wrote a 2d ‘Doom’-style raytracer after reading about it in one of those so-so game programming in 21 days books.  I say ‘so-so’ because to even have a hope of learning anything about a programming topic in only 21-days, it either has to have very narrow focus or, in this case, absolutely no depth. Also spent a little time in C++, but found it very complicated and confusing; I only got as far as I did because the AP test for Computer Science was in C++ the year I was taking it. Finally finished up this period hacking around in Java, made a bunch of browser Applets, as was the style at the time.

2000: VHDL, Perl, Java – Skipped intro CS courses.  Learning VHDL for digital logic course, and Java for algorithms courses.  Picked up a book on Perl for the fun of it, used it to parse and process Counter-Strike logs and generate webpages of stats.

2001: back to Basics – Spent the summer working at a computer camp for kids, instructing in robotics kits programmed in Parallax Basic.  Later that year, switched back to Visual Basic; the .Net version, which now rivaled Java in functionality.  This ended up being my language of choice for some time, marrying true OO semantics like Java with the wonderful Basic syntax.

2003: honed my programming and design skills finishing my CS degree.  Touched Prolog and Lisp briefly in one of those programming languages survey courses, but didn’t have a particularly good instructor.  Senior year I interned at Microsoft where I got to dig back into C for a while.

2004: C# – Started fulltime at Microsoft, where I was to be working in C#, so I learned C#.  “Oh, this is basically VB.Net with a Java-like syntax.”  Heh, yeah, sounds funny now maybe, but only because (statistically) you don’t know anyone who uses VB.Net.  I still say, for CLR 1.0 languages, it had the better syntax.  Learned C# 2.0 generics while it was fresh off the line, and learned a bunch of stuff about the difference between the CLR’s programming model and that of C.

2006: C++ – Was told I was to be working in C++, so I learned C++.  C++ Pocket Reference for syntax on the go, Stroustrup’s TC++PL 2nd ed for semantics (make sure you get special edition though), Meyers’ Effective C++ 3rd ed for learning a huge selection of gotchas and design issues, and Alexandrescu’s Modern C++ Design for teaching me an exciting selection of generic and generative programming techniques using templates.

2008: Python, JavaScript – Sparked by a conversation with a co-worker, started learning Python.  Was really surprised about how much easier many programming problems became.  I realized I didn’t really have a good idea what was out there.  Tinkered with JavaScript.  Started looking into programming language origins.

2009: Lisp, Scheme, Haskell, ML – Inevitably a pass through Lisp.  It seems a lot of programmers go through this phase at one point, and it makes sense; McCarthy created Lisp precisely to describe computing, and that clarity is refreshing.  Scheme, and the book SICP, introduced me to different computing models including first class continuations, streams (aka lazy lists), and the amb operator.  Haskell led to more exploration of lazy evaluation, better understanding and insights in generics, and its supremely practical typeclasses.

Currently tinkering with Standard ML, as a simplified alternative to Haskell; hacking together small tasks in Python; and whatever my new day job brings in.

~~~

Oh, and somewhere in school I briefly tried “eMbedded Visual Basic”.  That’s where I first learned it’s possible to start with a good programming language and cut it down to something similar but that *really* sucks.  For those who never heard of it, they removed a variety of language features including user defined types, the compiler error messages were inscrutable, and it was slow.

If you were a developer on that project, I’m really sorry, but better languages have been implemented over a weekend.  I don’t blame you.  Maybe you didn’t have a whole weekend.  Or were held hostage.  That would explain it I think.