Functional Programming In Python
The Complete, Practical, Step-By-Step Guide
The web is littered with theoretical articles about functional programming. But when it comes to practically helpful material, there's very little. This guide aims to fill that gap. Instead of including technical definitions of terms like "Higher Order Functions" or "Tail Call Optimization", this guide takes a practical, hands-on approach.
Aside: At Polydojo, we love functional programming. But it takes a while for newcomers to stop thinking in classical OOP terms. Hopefully, this guide will help new team members transition to Functional Programming faster.
What Is Functional Programming?
In functional programming, you treat functions as values. To illustrate what I mean, let's turn to a bit of JavaScript:
var n = 1;
var f = function (x) { return x + 1; };
In the above snippet:
- We set the variable
nto the value1. - We set the variable
fto the valuefunction (x) { return x + 1; }.
Just as 1 is value, function (x) { return x + 1; } is also a value.
That is, just like numbers, functions are values. And this holds true in Python too!
Python Is Fully Functional!
Let's translate the above JavaScript snippet to Python:
n = 1;
f = lambda x: x + 1;
Or equivalently:
n = 1;
def f (x):
return x + 1;
In JavaScript, the functional nature of the language is abundantly clear. Python's functional nature is clear in the lambda form, but not so much in the def form. Nonetheless, lambda and def are equivalent, and functions are values in Python.
Quick Intro To lambda Expressions
If you're familiar with lambda expressions, you may safely skip this section. If not, don't worry. lambda expressions are just a short-hand way for creating single-statement functions.
The syantax is of the form: lambda <params> : <expression>. The <expression> is evaluated and returned, but the return keyword should not be used.
As a quick illustration, here are a few def-form and lambda-form equivalents:
def checkEven (n):
return n % 2 == 0;
checkEven = lambda n: n % 2 == 0;
def add (x, y):
return x + y;
add = lambda x, y: x + y;
def factorial (n):
return 1 if n == 1 else n * factorial(n - 1);
factorial = lambda n: 1 if n == 1 else n * factorial(n - 1);
Functions Are Values. So What?
If a function is a value, then just like any other value, you should be able to:
- pass it as an argument to a function,
- define it locally within the scope a function, and
- return it from a function.
For example:
-
Just as you can call
squareRoot(2), you should be able to callmemoize(squareRoot), wherememoizeandsquareRootare both functions. -
Within a function, just as you can set
i = 0, you should be able to set
add1 = lambda x: x + 1, whereiandadd1are both local variables. -
In a function, just as you can
return 0, you should be able to
return lambda x: x * x.
Aside: If a programming language doesn't allow you to pass, return or locally define functions, then it isn't functional. Luckily for us, Python and JavaScript both are entirely functional languages!
Hmm, That's Interesting. But SO WHAT?
To see the power of functional programming, let's say you're tasked with writing a utility library that includes a number of mathematical transformation functions.
Computing Squares:
Your library must include a function, squareList(.), which accepts a list of numbers, squares each number, and returns the list thereof. That's pretty easy:
def squareList (nList):
result = [];
for n in nList:
result.append(n * n);
return result;
Computing Halves:
Your library must also include a function, halfList(.), which accepts a list of integers, halves each number, and returns the list thereof. Equally easy:
def halfList (nList):
result = [];
for n in nList:
result.append(n / 2.0);
return result;
Computing Factorials:
Your library must also include a function, factorialList(.), which accepts a list of positive integers, computes the factorial of each, and returns the list thereof. That's easy too!
def factorial (n):
return 1 if n == 1 else n * factorial(n - 1);
def factorialList (nList):
result = [];
for n in nList:
result.append(factorial(n));
return result;
Notice The Similarity?
Did you notice that the functions we just implemented above are very similar. They all seem to follow the pattern:
def somethingList (nList):
result = [];
for n in nList:
result.append(someFunc(n));
return result;
The only difference seems to be someFunc, which is either a squaring, halving or factorial-computing function.
Wouldn't it be nice, if we could write a function that'd accept a list (like nList) and a transformation (like factorial), and then simply map that transformation onto the list? Something like:
def mapTransform (func, nList):
result = [];
for n in nList:
result.append(func(n));
return result;
Then, using mapTransform(), and retaining factorial() we could write:
def squareList (nList):
return mapTransform(lambda n: n*n, nList);
def halfList (nList):
return mapTransform(lambda n: n/2.0, nList);
def factorialList (nList):
return mapTransform(factorial, nList);
Boom! That's Functional Programming!
By passing the transformation function func as a parameter to mapTransform, implementing our functions became so much simpler! Instead of repeating the same logic over and over again, we implemented the transformation-mapping logic just once, in mapTransform.
And map() is built-in!
The idea of mapping a transformation function onto a list is so common, that Python's built-in map() does exactly that. The built-in map() is far more flexible than mapTransform(). And instead of returning a list, it returns an iterable. So, using map(), we could write:
def squareList (nList):
return list(map(lambda n: n*n, nList));
# Alternatively, just for illustration:
squareList = lambda nList: list(map(lambda n: n*n, nList));
But honestly, given map(), is there even any need to implement our library at all? Maybe not?
Aside: Skipping List-Comprehensions
List-comprehensions are syntactically beautiful, and can be used instead of map() (and filter()) in many places. But at Polydojo, we strictly prefer map() instead. Stylistically, we think map() is clearer. Also, if we have to port code to JavaScript,map() calls are easier to translate.
Filtering - More Library Functions
Let's continue with the library example. Let's say that in addition to transformation functions, you're tasked with writing filtering functions too.
Filtering Evens
Your library should include a function keepEvens() which accepts a list of integers and returns a list of just the even numbers therein. This is easy:
def keepEvens (nList):
result = [];
for n in nList:
if n % 2 == 0:
result.append(n);
return result;
Filtering Leap Years:
Your library should include a function keepLeaps() which accepts a list of positive integers representing years (like 1990, 2000 etc.) and returns a list with just the leap years therein.
Remember, a year is a leap year if it's a multiple of 4 and 100, or otherwise if it's a multiple of 4 but not of 100.
def checkLeap (n):
return n % 400 == 0 or (n % 100 != 0 and n % 4 == 0);
def keepLeaps (nList):
result = [];
for n in nList:
if checkLeap(n):
result.append(n);
return result;
Filtering Primes
Write a function keepPrimes() which accepts a list of positive integers and returns a list of just the prime numbers therein.
Remember that a number n is prime if it has just two factors, 1 and n.
def checkPrime (n):
factorCount = 0;
for i in range(1, n + 1):
if n % i == 0:
factorCount += 1;
if factorCount > 2:
return False;
return (factorCount == 2);
def keepPrimes (nList):
result = [];
for n in nList:
if checkPrime(n):
result.append(n);
return result;
Aside: You can probably write a more efficient implementation for checkPrime(). But that's outside the scope of this guide.
Notice Another Pattern?
Did you notice that all the filtering examples also follow a pattern? That between keepLeaps() and keepPrimes(), the only difference is checkLeap(n) and checkPrime(n). The rest of the code is identical!
More generally, the only difference between the examples is the filtering-condition itself, which can be easily captured via a function parameter. Consider:
def keepWhere (checkFunc, nList):
result = [];
for n in nList:
if checkFunc(n):
result.append(n);
return result;
Now, using keepWhere(), and retaining the definitions of checkLeap() and checkPrime(), we re-implement our functions as:
def keepEvens (nList):
return keepWhere(lambda n: n % 2 == 0, nList);
def keepLeaps (nList):
return keepWhere(checkLeap, nList);
def keepPrimes (nList):
return keepWhere(checkPrime, nList);
Boom Again! That's Functional Programming!
Clearly, the above implementation is far cleaner in comparison with the non-functional examples above.
And filter() is built-in too!
Like map() is built-in for mapping, filter() is built-in for filtering. And like map(), filter() returns an iterable, not a list.
Using the built-in filter(), we can write:
def keepEvens (nList):
return list(filter(lambda n: n % 2 == 0, nList));
# Alternatively, for illustration only:
keepEvens = lambda nList: list(filter(lambda n: n%2 == 0, nList))
Keep mapli() and filterli() handy:
If you frequently find yourself writing list(map(.)) or list(filter(.)), consider keeping the following functions handy:
mapli = lambda seq, fn: list(map(fn, seq));
filterli = lambda seq, fn=None: list(filter(fn, seq));
The li suffix in mapli and filterli is short for list.
You'll notice that we've reversed the parameter order. The reversed order is something that you may want to skip, but at Polydojo, it's done to achieve parity with Underscore.js' _.map(), _.filter(), etc. Also, by having the function as the second parameter, multi-line lambdas become a bit-more readable.
def filterLeaps (nList):
return filterli(nList, lambda n: (
n % 400 == 0 or (n % 100 != 0 and n % 4 == 0)
));
Combining map() & filter():
In many situations, you may need to selectively map certain elements only.
Squaring Evens:
Let's write a function that accepts a list of integers and returns a list of the squares of the evens therein.
def squareEvens (nList):
evens = filterli(nList, lambda n: n % 2 == 0);
return mapli(evens, lambda n: n * n);
# Alternatively, _just_ for illustration:
squareEvens = lambda nList: mapli(
filterli(nList, lambda n: n % 2 == 0),
lambda n: n * n,
);
Tip: Don't write complex lambdas.
While the above two implementations are equivalent, the def-based version is far more readable. Writing convoluted, highly nested lambda-expressions is a bad idea. Please respect future developers (including yourself) who'll be working with your code.
Tip: Keep mapWhere() handy.
Practically, filtering first and then mapping is far more common than mapping first and then filtering. As filtering first is common, you may want to keep the following helper at hand:
def mapWhere(seq, transform, where):
return list(map(transform, filter(where, seq)));
Using mapWhere, we can write squareEvens as:
def squareEvens (nList):
return mapWhere(nList, lambda n: n * n, where=lambda n: n % 2 == 0)
Returning Functions
So far, we've passed functions as parameters to other functions. Clearly, doing so saved us a lot of work. Now, let's take a look at a situation where we'd write a function that returns a function.
Greetings!
Before diving into a real-world examples, consider something simple:
def greet (name="World"):
return "Hello, %s!" % name;
Above, greet is a simple greeter, with the default greeting "Hello, World!".
Let's say that instead you'd like the default greeting to be "Hello, friend!". In that case:
def greetFriend (name="friend"):
return "Hello, %s!" % name;
And to make the default greeting "Hello, beautiful!", we'd have:
def greetBeautiful (name="beautiful"):
return "Hello, %s!" % name;
Did you notice something? All the greeters follow the same pattern. Could we write a function for creating greeters with different default greetings? We sure can!
def make_greet (defaultName):
def greet (name=defaultName):
return "Hello, %s!" % name;
return greet;
greet = make_greet("World");
greetFriend = make_greet("friend");
greetBeautiful = make_greet("beautiful");
The function make_greet defines and returns a greet function. The argument passed to make_greet becomes the default value for greet. Thus, make_greet helps make greeters with different default greetings.
Note that each time make_greet is called, a different greet function is defined and returned. Thus, greet and greetFriend are entirely different functions. For clarity, compare the above with the following:
def buildDict (name="Jon"):
return {"name": name};
jon = buildDict(); # {"name": "Jon"}
tom = buildDict("Tom") # {"name": "Tom"}
jim = buildDict("Jim") # {"name": "Jim"}
It's clear that jon, tom and jim are entirely different dictionaries. Similarly, greet, greetFriend and greetBeautiful are entirely different functions.
Adder Maker
Consider the following:
def makeAdd (n):
def add (m):
return n + m;
return add;
Then, in an interactive Python shell:
>>> add1 = makeAdd(1)
>>> add1(10)
11
>>> add5 = makeAdd(5)
>>> add5(100)
105
>>> makeAdd(2)(3)
5
>>>
What's makeAdd doing? It accepts a parameter n, and returns a function add. In turn, the returned function, i.e. add, accepts a parameter m and returns n + m.
Notice that although n isn't directly a parameter to add, add can access n because n exists in an outer scope.
VF: Validation Functions
Just last week, we released VF: Validation Functions, which is an open source library for validating dict schemas. Almost every function in VF returns a function!
Here, we'll implement a mini-version of VF.
Let's say you're building a blogging app and storing data as dicts in a NoSQL-ish database like MongoDB, CouchDB or PogoDB. You'll have to deal with posts, comments, users, notifications, email-subscriptions, and other types of documents/dicts.
For validating the schema of a dict, say a post, we'd like to write a schema specification as follows:
POST_SCHEMA_SPEC = {
"_id": lambda x: type(x) is str,
"title": lambda x: type(x) is str and 5 <= len(x) <= 50,
"body": lambda x: type(x) is str and 50 <= len(x) <= 5000,
"userId": lambda x: type(x) is str,
"published_time": lambda x: type(x) is int,
}
Basically, POST_SCHEMA_SPEC says that each post should have
- an
'_id'property of typestr; - a
'title'property of typestr, with length between 5 and 50 characters; - etc.
Let's write a utility function that accepts a dict-type document doc along with a schema (also a dict), and checks if doc matches schema:
In utils.py:
def checkSchema (doc, schema):
if set(doc.keys()) != set(schema.keys()):
return False;
# otherwise ...
for key in doc:
if not schema[key](doc[key]):
return False;
# otherwise ...
return True;
The checkSchema function simply checks that doc and schema have the same keys, and that each value in doc satisfies the corresponding check-function in schema.
We have a handful of document types: posts, comments, users, notifications, and email-subscriptions. Ideally, the schema for each should be defined in a separate (model-specific) module. For example, postModel.py should deal with posts, commentModel.py with comments, etc.
Now, in postModel.py, we could explicitly pass POST_SCHEMA_SPEC to checkSchema on each call, but as it'll be a module-wide constant, wouldn't it make more sense to define checkPostSchema(doc)?
In postModel.py:
from utils import checkSchema;
POST_SCHEMA = {
"_id": lambda x: type(x) is str,
"title": lambda x: type(x) is str and 5 <= len(x) <= 50,
"body": lambda x: type(x) is str and 50 <= len(x) <= 5000,
"userId": lambda x: type(x) is str,
"published_time": lambda x: type(x) is int,
};
def checkPostSchema (doc):
return checkSchema (doc, POST_SCHEMA);
# ... etc ...
Similarly, in commentModel.py:
from utils import checkSchema;
COMMENT_SCHEMA = {
# ... etc ...
};
def checkCommentSchema (doc):
return checkSchema(doc, COMMENT_SCHEMA);
# ... etc ...
Continuing similarly, there would be schema-aware checkers in userModel.py, notificationModel.py, etc. And while this setup is already quite modular and readable, consider:
In utils.py:
def make_schemaChecker (schema):
def checkSchema (doc):
if set(doc.keys()) != set(schema.keys()):
return False;
# otherwise ...
for key in doc:
if not schema[key](doc[key]):
return False;
# otherwise ...
return True;
return checkSchema;
The function make_schemaChecker(schema) defines and returns the function checkSchema(doc), which is bound to the schema.
Then in postModel.py:
from utils import make_schemaChecker;
checkPostSchema = make_schemaChecker({
"_id": lambda x: type(x) is str,
"title": lambda x: type(x) is str and 5 <= len(x) <= 50,
"body": lambda x: type(x) is str and 50 <= len(x) <= 5000,
"userId": lambda x: type(x) is str,
"published_time": lambda x: type(x) is int,
});
# ... etc ...
As you can see, there's no need for writing any intermediate function; and for that matter, no need to even define a separate variable (like POST_SCHEMA_SPEC) for holding the schema specification. Plus, the name of the function make_schemaChecker is pretty self-explanatory. As a result, the code is more readable.
Of course, like postModel.py, similar improvements can be made in commentModel.py, notificationModel.py, etc.
Nesting Support
Continuing the above example, to really see the benefit of make_schemaChecker, let's take a peek at userModel.py:
from utils import make_schemaChecker;
checkUserSchema = make_schemaChecker({
"_id": lambda x: type(x) is str,
"name": make_schemaChecker({
"first": lambda x: type(x) is str,
"last": lambda x: type(x) is str,
}),
"email": lambda x: type(x) is str and '@' in x,
"address": make_schemaChecker({
"street": lambda x: type(x) is str,
"city": lambda x: type(x) is str,
"state": lambda x: type(x) is str,
"zip": lambda x: type(x) is str,
}),
"joined_timestamp": lambda x: type(x) is int,
# ... etc ...
});
Did you notice the check-functions against the "name" and "address" keys? As make_schemaChecker returns a function, it can easily be used inline for defining nested schemas!
Aside: You should probably use regular expressions for validating email addresses. We've skipped that above.
VF includes a number of helpers for making simple checker functions. Check out VF's docs/code to know more.
Private Variables In Python:
You've probably heard that Python doesn't have private variables. From an OOP standpoint, I suppose that's true. But every functional programming includes private variables, because you can easily setup variables that are private by scope.
To see what I mean, consider the following:
bank.py:
def buildBankAccount (initialBalance):
private = {
"balance": initialBalance,
"transactions": [{"initial": initialBalance}],
};
account = {};
def getBalance ():
return private["balance"];
account["getBalance"] = getBalance;
def deposit (amount):
private["balance"] += amount;
private["transactions"].append({"deposit": amount});
account["deposit"] = deposit;
def withdraw (amount):
if amount > private["balance"]:
raise Exception("Insufficient Funds");
# otherwise ...
private["balance"] -= amount;
private["transactions"].append({"withdraw": amount});
account["withdraw"] = withdraw;
return account;
The function buildBankAccount builds and returns an account dictionary. The private dictionary is not returned, and is not directly exposed via account. The only way to effect private['balance'] is to call either account['deposit'] or account['withdraw'].
Further, a trail of transactions is maintained in private['transactions'], making sure that all deposits/withdrawals are properly logged.
In an interactive Python shell:
>>> from bank import buildBankAccount
>>>
>>> account = buildBankAccount(1000)
>>> account['getBalance']()
1000
>>> account['deposit'](100)
>>> account['getBalance']()
1100
>>> account['withdraw'](10)
>>> account['getBalance']()
1090
>>>
>>> account.keys()
dict_keys(['getBalance', 'deposit', 'withdraw'])
>>>
What's happening here is that the private dict is accessible within buildBankAccount, but not accessible outside. In effect, private is an OOP-style private variable!
Because getBalance, deposit and withdraw are all defined within buildBankAccount, they can access private. Note that account['deposit'] is simply a reference to the deposit function.
While account is a dictionary, it feels like an object. Wouldn't it be nice if we could say account.getBalance() instead of account['getBalance']()? It sure would! And with Dotsi, you can!
Dotsi: Dot-Accessible Dictionaries
Dotsi is an open-source library that brings JavaScript-like dot-access to Python dicts. That is, using Dotsi, we can write account.getBalance(), instead of account['getBalance']().
Re-implementing bank.py using Dotsi:
import dotsi;
def buildBankAccount (initialBalance):
private = dotsi.fy({
"balance": initialBalance,
"transactions": [{"initial": initialBalance}],
});
account = dotsi.fy({});
def getBalance ():
return private.balance;
account.getBalance = getBalance;
def deposit (amount):
private.balance += amount;
private.transactions.append({"deposit": amount});
account.deposit = deposit;
def withdraw (amount):
if amount > private["balance"]:
raise Exception("Insufficient Funds");
# otherwise ...
private.balance -= amount;
private.transactions.append({"withdraw": amount});
account.withdraw = withdraw;
return account;
In an interactive Python shell:
>>> from bank import buildBankAccount
>>>
>>> account = buildBankAccount(2000)
>>> account.getBalance()
2000
>>> account.deposit(200)
>>> account.getBalance()
2200
>>> account.withdraw(100)
>>> account.getBalance()
2100
>>>
Convention: Builders & Makers
At Polydojo, we follow the convention that:
-
Functions that return functions are called makers, and their names should begin with
make. For example,make_greet,make_schemaChecker. -
Functions that build object-like dictionaries are called builders, and their names should begin with
build. For example,buildBankAccountetc.
You Needn't Write Classes
Given that we can write builder functions, there's practically no need to write new classes. At Polydojo, the only good reason to write a class is to subclass Exception. Apart from subclassing Exception, you probably don't need classes. If you look at our open-source contributions, you'll barely find any classes.
Decorators:
So far, we have:
- Passed functions as arguments to other functions.
- Defined functions locally within other functions.
- Returned functions from functions.
But sometimes, we may want to do all three of those things at the same time!
Caching Example
Let's say you have three slow-running functions: slow1(), slow2() and slow3(). Each function accepts a single string parameter and, always produces the same output for a given input.
Consider the following:
slow1_cache = {};
def faster1 (input):
if input not in slow1_cache:
slow1_cache[input] = slow1(input);
return slow1_cache[input];
Each time faster1 is called with an input, we check if that input is already in slow1_cache. If it's in there, we'll simply return the corresponding result from the cache. If not, we'll call slow1(input), add the result to the cache and then return it.
Thus, when faster1('foo') is called for the first time, slow1('foo'), will be called, which, let's say, evalutes to 'bar'. At that point slow1_cache will include {'foo': 'bar'}. The next time faster1('foo') is called, we'll return 'bar' directly from the cache.
Similarly, we can write faster2() and faster3():
slow2_cache = {};
def faster2 (input):
if input not in slow2_cache:
slow2_cache[input] = slow2(input);
return slow2_cache[input];
slow3_cache = {};
def faster3 (input):
if input not in slow3_cache:
slow3_cache[input] = slow3(input);
return slow3_cache[input];
Clearly, there's a pattern here. Instead writing a separate faster (cached) version for every slow function, could we write a function that produces a faster version of a slow function? Let's try:
def fasterfy (slowFunc):
cache = {};
def fasterFunc (input):
if input not in cache:
cache[input] = slowFunc(input);
return cache[input];
return fasterFunc;
Now, we can write:
faster1 = fasterfy(slow1);
faster2 = fasterfy(slow2);
faster3 = fasterfy(slow3);
Clearly, using fasterfy is a big help. The code is cleaner, smaller and far more readable.
But now that we've defined faster1, we'll need to refactor the rest of our codebase to use faster1 instead of slow1. And this similarly applies to faster2 and faster3 as well. So, we'll need to make multiple changes across our codebase. But instead of making all those changes, could we make slow1 an alias for the new faster1? Consider:
faster1 = fasterfy(slow1);
slow1 = faster1;
faster2 = fasterfy(slow2);
slow2 = faster2;
faster3 = fasterfy(slow3);
slow3 = faster3;
Or equivalently:
slow1 = fasterfy(slow1);
slow2 = fasterfy(slow2);
slow3 = fasterfy(slow3);
Writing slow1 = fasterfy(slow1) is very similar to writing number = number + 1. After slow1 = fasterfy(slow1), slow1 becomes a cached version of it's former self. And since this cached version is also called slow1, we could avoid making changes to the rest of our codebase!
Decorator Syntax
The idea of writing something like slow1 = fasterfy(slow1) is so common, that there's a special syntax for that. Assuming that the time module and fasterfy are imported/defined, the following two examples are equivalent:
Without Decorator Syntax:
def slow4 (x):
time.sleep(5); # Sleep 5 sec.
return "foobar";
slow4 = fasterfy(slow4);
With Decorator Syntax:
@fasterfy
def slow4 (x):
time.sleep(5); # Sleep 5 sec.
return "foobar";
The @fasterfy decorator syntax is just syntactic sugar. Writing @fasterfy just before def is equivalent to writing slow1 = fasterfy(slow1) immediately after the def-block.
Aside: The function slow4 doesn't do anything interesting, but it's useful for demonstration purposes.
General Decorators
The fasterfy() decorator requires the decorated function (i.e. slow4 etc.) to accept exactly one parameter. But to write a more general decorator, we should use *args and **kwargs instead; so that the decorator can be used to decorate any function.
import time;
def timefy (fn):
"Simple timing decorator.";
def wrapper (*args, **kwargs):
"Wrapper function.";
t0 = time.time();
result = fn(*args, **kwargs);
print("Time taken: %s sec." % (time.time() - t0));
return result;
return wrapper;
@timefy
def slow_add (x, y):
"Simple addition.";
time.sleep(x + y);
return x + y;
The @timefy decorator prints the time taken before returning the decorated function's result. At it uses *args and **kwargs, it can can be used to decorate any function. Let's see it in action:
In an interactive Python shell:
>>> slow_add(1, 1)
Time taken: 2.006281852722168 sec.
2
>>> slow_add(1, 2)
Time taken: 3.00673508644104 sec.
3
>>> slow_add(2, 2)
Time taken: 4.0057408809661865 sec.
4
>>>
As expected, the time taken in seconds is roughly equal to the sum of the numbers passed. That's entirely by definition.
Note that there's no print() call in slow_add(). The printing is done by @timefy. Thus, decorators can be designed to have such (desirable) side-effects.
Wrapper Updating:
Continuing with the previous example, in the same Python shell:
>>> slow_add.__name__
'wrapper'
>>> slow_add.__doc__
'Wrapper function.'
>>>
The above result may seem strange at first, as it says that slow_add's name and docstring (documentation-string) are 'wrapper' and 'Wrapper function.' respectively. But if you think about it, isn't that expected?
Remember that @timefy is equivalent to slow_add = timefy(slow_add). What does timefy return? It returns wrapper. Thus, slow_add.__name__ is 'wrapper', not 'slow_add'.
But obviously, slow_add's docstring being 'Wrapper function.' is not very helpful. It would be nice if we could retain it's original name and docstring, i.e. 'slow_add' and 'Simple addition.' respectively.
Python's functools module has a helper for that, functools.update_wraper(). Let's use it:
import time;
import functools;
def timefy (fn):
"Simple timing decorator.";
def wrapper (*args, **kwargs):
"Wrapper function.";
t0 = time.time();
result = fn(*args, **kwargs);
print("Time taken: %s sec." % (time.time() - t0));
return result;
return functools.update_wrapper(wrapper, fn); # <-- Line Changed
@timefy
def slow_add (x, y):
"Simple addition.";
time.sleep(x + y);
return x + y;
Now, in a Python shell:
>>> slow_add.__name__
'slow_add'
>>> slow_add.__doc__
'Simple addition.'
>>>
The functools module also includes functools.wraps(), which is a convenience around functools.update_wrapper(). You should check it out.
Combining Decorators:
Retaining the definition of timefy() above, let's write a general caching decorator and then rewrite slow_add():
import json;
def cachefy (fn):
"JSON-serialization based caching decorator.";
cache = {};
def wrapper (*args, **kwargs):
jkey = json.dumps([args, kwargs]);
if jkey not in cache:
cache[jkey] = fn(*args, **kwargs);
return cache[jkey];
return functools.update_wrapper(wrapper, fn);
@timefy # <-- Same as before
@cachefy # <-- Newly added!
def slow_add (x, y):
"Simple addition.";
time.sleep(x + y);
return x + y;
The cachefy decorator uses json.dumps() for generating a hashable cache-key, jkey. As it uses json.dumps(), cachefy can only work with functions that accept JSON-serializable arguments. (Otherwise, a TyepError will be raised.)
The interesting thing to note, of course, is that slow_add now has two decorators! Without using decorator syntax, that's equivalent to:
slow_add = timefy(cachefy(slow_add));
Let's see it in action:
>>> slow_add(2, 3); # First call, should be slow.
Time taken: 5.0025999546051025 sec.
5
>>> slow_add(2, 3); # Subsequent calls should be FAST!
Time taken: 5.412101745605469e-05 sec.
5
>>> slow_add(2, 3); # FAST!
Time taken: 6.389617919921875e-05 sec.
5
>>>
As you can see, the first call to slow_add(2, 3) was slow, taking about 5 seconds. But subsequent calls were super-fast, taking about 50 microseconds; as the result would've be returned directly from the cache.
Aside: A microsecond is one-millionth of a second. Thus, a response time of 50 microseconds is 100,000 times faster in comparison with 5 seconds.
Returning Decorators:
Alright. We've seen that decorators are functions that accept and return functions. But as decorators are themselves functions, could we write a function that returns a decorator?
Consider:
def make_transformDecorator (transform):
def transformDecorator (fn):
def wrapper (*args, **kwargs):
return transform(fn(*args, **kwargs));
return functools.update_wrapper(wrapper, fn);
return transformDecorator;
upperCase = make_transformDecorator(str.upper);
@upperCase
def concat(x, y):
return x + y;
And in a Python shell:
>>> concat('foo', 'bar')
'FOOBAR'
>>> concat('Sumukh', 'Barve')
'SUMUKHBARVE'
>>> concat('Poly', 'dojo')
'POLYDOJO'
>>>
What's happening here?
First and foremost, note that make_transformDecorator is NOT itself a decorator. But instead, it returns transformDecorator, which IS indeed a decorator. Thus, upperCase = make_transformDecorator(str.upper) creates a decorator that calls str.upper on the decorated function's result.
Hence:
concat('Poly', 'dojo')
=> str.upper('Poly' + 'dojo')
=> 'POLYDOJO'
Instead of explicitly defining upperCase, we could have:
@make_transformDecorator(str.upper)
def concat (x, y):
return x + y;
Flask, Bottle & Vilo: They All Return Decorators!
If you're building a web-app with Flask, you'll often write something like:
@app.route("/hello")
def route_hello ():
return "Hello, World!";
What's happening here? It's simple: app.route() returns a decorator for binding route_hello() to the "/hello" path.
Bottle and Vilo also follow a similar, decorator-based routing pattern.
Aside: As the author of Vilo, I'm obviously biased. But it's a pretty sleek framework! In fact, this webpage, the one you're reading right now, is served via Vilo. (This blog is powered by ViloLog, which builds atop Vilo.)
You Now Know Functional Programming!
In this guide, we've covered a number of aspects of functional programming. We passed functions as arguments, defined them locally, returned them, did all of that together by defining decorators, and we even wrote a function that returns a decorator.
I sincerely hope that you had fun. And I hope that this guide will help propel you further on your path toward eschewing classical OOP in favor of Functional Programming. The folks who wrote Lisp and Scheme really knew what they were doing. And while Python and JavaScript have a different syntax from Lisp/Scheme, the languages are entirely functional.
Everything I said in this guide about Polydojo's practices, to the best of my knowledge, is entirely true. You can actually check out our open source contributions to find examples of maker functions, builder functions, and of course, barely any classes.
As we gather feedback from you, we'll expand this guide with additional examples and explanations. And at this point, if you'd like to explore functional programming further, I first recommend perusing the docs for Python's built-in functools module.
Thank You!
Thank you, for reading all the way to the end. I sincerely hope that this guide was helpful. If you have any questions, comments or suggestions, please feel free to ask. You can email me at [email protected]. I'll always be happy to hear from you.
PS:
Any function that accepts/returns a function is called a Higher Order Function. I find this a bit weird, as it seems to imply that such functions are somehow special. They aren't. They're just functions. And all functions are equally awesome.