Build Your Own Python Template Engine

Template engines like Jinja and Mako can considerably simplify the generation of HTML output. In this blog post, we'll implement a simple, yet flexible Python template engine, in under 50 lines. The exact same principles are used by the Qree template engine.

Best Online Collaboration Tools

Running Example

To enable discussion, let's consider a specific example.

Let's say you're tasked with creating a webpage that displays the top 3 most popular articles on your site. You've already written a function, get_top_articles(n), for getting the top n articles. The only step remaining is to output the HTML.

We'll use this situation as a running example throughout most of this post.

Without Any Temple Engine (Case 1)

Without a template engine, we may write:

def output_top_articles_html ():
    articles = get_top_artilces(3)
    html = '''<!doctype html><html>
    <head>
        <title>Top Articles</title>
    </head>
    <body>
        <ul>
            <li><a href="/article/%s">%s</a></li>
            <li><a href="/article/%s">%s</a></li>
            <li><a href="/article/%s">%s</a></li>
        </ul>
    </body></html>''' % (
        escape_html(articles[0].id), escape_html(articles[0].title),
        escape_html(articles[1].id), escape_html(articles[1].title),
        escape_html(articles[2].id), escape_html(articles[2].title),
    )

To keep things simple, no stylesheets, meta-tags, header/footer etc. are included. And we assume that escape_html() is a reliable HTML-escape function.

While the function outputs HTML, it has major issues:

The number of articles (3) is hard-coded. If the number were larger, say 30, the function would become unbearably long and almost entirely unreadable.
Assuming that other areas on the webpage would also need dynamic substitution, %s-formatting isn't viable. (If header/footer etc. were included, they too would likely need dynamic substitutions.)
Amid the flurry of everyday development activities, future developers may forget to ardently call escape_html(), making our webpage susceptible to XSS.

Using A Template Engine (Case 2)

For contrast, let's say you're using Qree for templating. You could set-up your template as a separate file, say top-articles.html, and then render it via Python:

First, in top-articles.html:

<!doctype html><html>
<head>
    <title>Top Articles</title>
</head>
<body>
    <ul>
    @= for article in data['articles']:
    @{
        <li><a href="/article/{{: article.id :}}">{{: article.title :}}</a></li>
    @}
    </ul>
</body></html>

Then, in Python:

def output_top_articles_html (n):
    articles = get_top_artilces(n)
    return qree.renderPath('top-articles.html', data={'articles': articles})

In stark contrast to the previous example, here:

The number of articles isn't hard-coded.
Additional information can be passed to the template via data.
Qree auto-escapes everything between {{: and :}} tags.

Without A Template Engine, Again! (Case 3)

It's quite clear the using a template engine has advantages. But couldn't some of those advantages be achieved with just regular Python? Let's try:

esc = escape_html    # Short alias.

def output_top_articles_html (n):
    articles = get_top_artilces(n)
    html = '''<!doctype html><html>
    <head>
        <title>Top Html Pages</title>
    </head>
    <body>
        <ul>'''
    for article in articles:
        html += (
            '<li><a href="/articles/' + esc(article.id) +
            '">' + esc(article.title) + '</a></li>'
        )
    html += '''
        </ul>
    </body></html>
    '''
    # Finally:
    return html;

Above, although we aren't using a template engine:

The number of articles is no longer hard-coded.
Additional substitutions can be made with a bit more ease.
Escaping with esc() is a bit easier, but only marginally so.

Notice The Similarity?

Case 3 isn't nearly as readable as Case 2. But did you notice the similarities? Particularly, please compare:

Case 2's {{: article.id :}} with Case 3's + esc(article.id) +
Case 2's {{: article.title :}} with Case 3's + esc(article.title) +
Case 2's for article in data['articles'] with Case 3's for article in articles

Gosh! If you look carefully, it appears that Qree is just contextually replacing snippets like {{: article.id :}} with those like + esc(article.id) + etc. And it seems to be relying on @{ and @} for inferring indentation. Could it be that simple?

Yes! That's all Qree does. You've hit the nail on the head!

Qree uses Python's built-in exec() and eval() to execute in-template code. That's why it supports all language features. And of course, before exec(), the quotes need to be placed/replaced properly for maintaining context.

miniQree: Under 50 lines

Qree itself is under 200 lines. Here's a mini-version thereof, under 50 lines.

miniQree.py:

escHtml = lambda s: (str(s).replace("&", "&amp;").replace("<", "&lt;")
    .replace(">", "&gt;").replace('"', "&quot;").replace("'", "&#x27;")
)

def renderStr (tplStr, data=None):
    "Renders template string `tplStr` using `data`."
    fnStr = "def templateFn (data):\n"
    indentCount = 4
    indentify = lambda: " " * indentCount
    fnStr += indentify() + "from miniQree import escHtml\n"
    fnStr += indentify() + "output = ''\n"
    for line in tplStr.splitlines(True):
        lx = line.lstrip()
        if lx.startswith("@="):
            fnStr += indentify() + lx[2 : ].lstrip() + "\n"
        elif lx.startswith("@{"):
            indentCount += 4
        elif lx.startswith("@}"):
            indentCount -= 4
        else:
            fnStr += indentify() + "output += " + "'''" + (
                line.replace("{{=",  "''' + str(")
                    .replace("=}}",  ") + '''")
                    .replace("{{:",  "''' + escHtml(")
                    .replace(":}}",  ") + '''")
            ) + "'''\n"
    fnStr += indentify() + "return output\n"
    #print(fnStr)    # <-- Uncomment to see built `fnStr`.
    exec(fnStr)
    fn = eval('templateFn')
    return fn(data)

How It Works

Our goal is to build the template function fn(data), which renders the template as desired. To build fn, we build and execute the string fnStr. The string fnStr is simply the code-representation of fn itself.

We use escHtml for escaping HTML, and indentCount & indentify() for managing indentation. We increase indentCount on encountering @{ and decrease on @}.

We use ''' for quoting non-code bits; simply replacing {{: with ''' + escHtml( and replacing :}} with ) + '''. Thus, {{: data['foo'] :}} becomes ''' + escHtml( data['foo'] ) + '''. The leading and trailing ''' are meant for the surrounding bits.

In addition to {{: and :}}, we've introduced {{= and =}} for non-escaped interpolation, which may be desirable for use with Bleach or a similar HTML-sanitization tool.

Un-comment #print(fnStr) to see the actual function built by renderStr.

Testing

Let's add a test case to miniQree.py:

def test_renderStr ():
    assert renderStr("plain") == "plain"
    tpl = "Hello, {{: data + '!' :}}"
    assert renderStr(tpl, "World") == 'Hello, World!'
    assert renderStr("{{: data :}}", "> World") == '&gt; World'
    assert renderStr("{{= data =}}", "> World") == '> World'
    tpl = '''
    @= for n in range(1, data+1):
    @{
        {{: n :}} is {{: 'EVEN' if n % 2 == 0 else 'ODD' :}}
    @}'''
    expected = '''
        1 is ODD\n        2 is EVEN
        3 is ODD\n        4 is EVEN\n'''
    assert renderStr(tpl, data=4) == expected

Run tests with pytest miniQree.py or via the Python shell:

>>> import miniQree
>>> miniQree.test_renderStr()
>>> # No exception => Tests passed.

Limitations

miniQree.renderStr() has a number of limitations, such as, it:

Can't handle templates that themselves include '''.
Can't handle substitutions sandwiched between ', eg. '{{: data :}}'
Can't rename the data variable within templates.
Can't override default tags like {{:, =}} etc.
Doesn't validate indentation/de-indentation prior to rendering.
Doesn't validate nesting of escape/interpolation tags.

Qree, which is about 4x bigger than miniQree, doesn't suffer from any of the above limitations. Plus, it's reasonably well-tested. Check out Qree's docs/code to know more.

Other Template Engines

Not all template engines rely on exec(). Some use Regular Expressions, Parsing Expression Grammars and/or other tools instead. A large reason behind miniQree.py's compactness is that it uses exec(). This is a bit cheeky, as by relying on exec(), Python's lexing and parsing is (indirectly) relied upon.

Warning: As Qree and miniQree rely on exec(), they should NOT be used to render untrusted templates.

This Page Is Rendered With Qree!

ViloLog, the platform that powers this blog, uses Qree for templating. This webpage, the one you're reading right now, was rendered with Qree!

Over To You

If you have any questions, comments or suggestions; please feel free to write to me at [email protected]. I'll be happy to hear from you.