Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Table of Contents

Introduction

- About this tutorial

- So what's Erlang?

- Don't drink too much Kool-Aid

- What you need to dive in

- Where to get help

Starting Out

- The Shell

- Shell Commands

Starting Out (for real)

- Numbers

- Invariable Variables

- Atoms

- Boolean Algebra and Comparison Operators

- Tuples

- Lists

- List Comprehensions

- Bit Syntax

- Binary Comprehensions

Modules

- What are modules

- Module Declaration

- Compiling the Code

- More About Modules

Syntax in Functions

- Pattern Matching

- Guards, Guards!

- What the If!?

- In Case ... of

- Which to use?

Types (or lack thereof)

- Dynamite-strong Typing

- Type Conversions

- To Guard a Data Type

- For Type Junkies

Recursion

- Hello recursion!

- Length

- Length of Tail Recursion

- More recursive functions

- Quick, Sort!

- More than lists

- Thinking recursively

Higher Order Functions

- Let's get functional

- Anonymous functions

- Maps, filters, folds and more

Errors and Exceptions

- Not so fast!

- A Compilation of Errors

- No, YOUR logic is wrong!

- Run-time Errors

- Raising Exceptions

- Dealing with Exceptions

- Wait, there's more!

- Try a try in a tree

Functionally Solving Problems

- Reverse Polish Notation Calculator

- Heathrow to London

A Short Visit to Common Data Structures

- Won't be too long, promised!

- Records

- Key-Value Stores

- Arrays

- A Set of Sets

- Directed Graphs

- Queues

- End of the short visit

The Hitchhiker's Guide to Concurrency

- Don't Panic

- Concepts of Concurrency

- Not Entirely Unlike Linear Scaling

- So long and thanks for all the fish!

More On Multiprocessing

- State Your State

- We love messages, but we keep them secret

- Time Out

- Selective Receives

Errors and Processes

- Links

- It's a Trap!

- Monitors

- Naming Processes

Designing a Concurrent Application

- Understanding the Problem

- Defining the Protocol

- Lay Them Foundations

- An Event Module

- The Event Server

- Hot Code Loving

- I Said, Hide Your Messages

- A Test Drive

- Adding Supervision

- Namespaces (or lack thereof)

What is OTP?

- It's The Open Telecom Platform!

- The Common Process, Abstracted

- The Basic Server

- Specific Vs. Generic

Clients and Servers

- Callback to the Future

- .BEAM me up, Scotty!

Rage Against The Finite-State Machines

- What Are They?

- Generic Finite-State Machines

- A Trading System Specification

- Game trading between two players

- That Was Quite Something

- Fit for the Real World?

Coming Soon

- Event Handlers

- Supervisors

- More OTP related chapters

- Other stuff I have not started thinking about yet

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Introduction

About this tutorial

Note: this e-book is composed out of the excellent web-tutorial written by Frédéric Trottier-Hébert which is available online at LearnYouSomeErlang.com under Creative Commons Attribution Non-Commercial No Derivative License.

This is the beginning of Learn You Some Erlang for Great Good! Reading this tutorial should be one of your first steps in learning Erlang, so let's talk about it a bit.

First of all, I began growing the idea of writing this after reading Miran Lipovača's Learn You a Haskell for great Good! (LYAH) tutorial; I thought he did a great job making the language attractive and the learning experience friendly. As I already knew him, I asked him how he felt about me writing an Erlang version of his book. He liked the idea, being somewhat interested in Erlang.

So here I am typing this. Of course there were other sources to my motivation: I mainly find the entry to the language to be hard (the web has sparse documentation and otherwise you need to buy books), and I thought the community would benefit from a LYAH-like guide. Less importantly, I've seen people attributing Erlang too much or not enough merit sometimes based on sweeping generalizations. Then there are people who sure as hell believe Erlang is nothing but hype. If I'd like to convince them otherwise, I know they're not likely to read this in the first place.

This book thus wants itself to be a way to learn Erlang for people who have basic knowledge of programming in imperative languages (such as C/C++, Java, Python, Ruby, etc) and may or may not know functional programming (Haskell, Scala, Erlang, Clojure, OCaml...). I also want to write this book in a honest manner, selling Erlang for what it is, acknowledging its weaknesses and strengths.

So what's Erlang?

First of all, Erlang is a functional programming language. If you have ever worked with imperative languages, statements such as i++ may be normal to you; in functional programming they are not allowed. In fact, changing the value of any variable is strictly forbidden! This may sound weird at first, but if you remember your math classes, it's in fact how you've learned it:

y = 2

x = y + 3

x = 2 + 3

x = 5

Had I added the following:

x = 5 + 1

x = x

∴ 5 = 6

You would have been very confused. Functional programming recognizes this: If I say x is 5, then I can't logically claim it is also 6! This would be dishonest. This is also why a function with the same parameter should always return the same result:

x = add_two_to(3) = 5

∴ x = 5

Functions always returning the same result for the same parameter is called referential transparency. It's what lets us replace add_two_to(3) with 5, as the result of 3+2 will always be 5. That means we can then glue dozens of functions together in order to resolve more complex problems while being sure nothing will break. Logical and clean isn't it? There's a problem though:

x = today() = 2009/10/22

-- wait a day --

x = today() = 2009/10/23

x = x

∴ 2009/10/22 = 2009/10/23

Oh no! My beautiful equations! They suddenly all turned wrong! How come my function returns a different result every day?

Obviously, there are some cases where it's useful to break referential transparency. Erlang has this very pragmatic approach with functional programming: obey its purest principles (referential transparency, avoiding mutable data, etc), but break away from them when real world problems pop up.

Now, we defined Erlang as a functional programming language, but there's also a large emphasis on concurrency and high reliability. To be able to have dozens of tasks being performed at the same time, Erlang uses the actor model, and each actor is a separate process in the virtual machine. In a nutshell, if you were an actor in Erlang's world, you would be a lonely person, sitting in a dark room with no window, waiting by your mailbox to get a message. Once you get a message, you react to it in a specific way: you pay the bills when receiving them, you respond to Birthday cards with a "Thank you" letter and you ignore the letters you can't understand.

Erlang's actor model can be imagined as a world where everyone is sitting alone in their own room and can perform a few distinct tasks. Everyone communicates strictly by writing letters and that's it. While it sounds like a boring life (and a new age for the postal service), it means you can ask many people to perform very specific tasks for you, and none of them will ever do something wrong or make mistakes which will have repercussions on the work of others; they may not even know the existence of people other than you (and that's great).

To escape this analogy, Erlang forces you to write actors (processes) that will share no information with other bits of code unless they pass messages to each other. Every communication is explicit, traceable and safe.

When we defined Erlang, we did so at a language level, but in a broader sense, this is not all there is to it: Erlang is also a development environment as a whole. The code is compiled to bytecode and runs inside a virtual machine. So Erlang, much like Java and kids with ADD, can run anywhere. The standard distribution includes (among others) development tools (compiler, debugger, profiler, test framework), the Open Telecom Platform (OTP) Framework, a web server, a parser generator, and the mnesia database, a key-value storage system able to replicate itself on many servers, supporting nested transactions and letting you store any kind of Erlang data.

The VM and the libraries also allow you to update the code of a running system without interrupting any program, distribute your code with ease on many computers and manage errors and faults in a simple but powerful manner.

We'll see how to use most of these tools and achieve safety later on, but for now, I'll tell you about a related general policy in Erlang: Let it crash. Not like a plane with dozens of passengers dying, but more like a tightrope walker with a safety net under him. While you should avoid making mistakes, you won't need to check for every type or error condition in most cases.

Erlang's ability to recover from errors, organize code with actors and making it scale with distribution and concurrency all sound awesome, which brings us to the next section...

Don't drink too much Kool-Aid

There may be many little yellowish-orange sections named like this one around the book (you'll recognize them when you see them). Erlang is currently gaining lots of popularity due to zealous talks which may lead people to believe it's more than what it really is. These reminders will be there to help you keep your feet on the ground if you're one of these overenthusiastic learners.

The first case of this is related to Erlang's massive scaling abilities due to its lightweight processes. It is true that Erlang processes are very light: you can have hundreds of thousands of them existing at the same time, but this doesn't mean you have to use it that way just because you can. For example, creating a shooter game where everything including bullets is its own actor is madness. The only thing you'll shoot with a game like this is your own foot. There is still a small cost in sending a message from actor to actor, and if you divide tasks too much, you will make things slower!

I'll cover this with more depth when we're far enough into the learning to actually worry about it, but just keep in mind that randomly throwing parallelism at a problem is not enough to make it go fast. Don't be sad; there are times when using hundreds of processes will both be possible and useful! It's just not happening all the time.

Erlang is also said to be able to scale in a directly proportional manner to how many cores your computer has, but this is usually not true: it is possible, but most problems do not behave in a way that lets you just run everything at the same time.

There's something else to keep in mind: while Erlang does some things very well, it's technically still possible to get the same results from other languages. The opposite is also true; evaluate each problem as it needs to be, and choose the right tool according to the problem being addressed. Erlang is no silver bullet and will be particularly bad at things like image and signal processing, operating system device drivers, etc. and will shine at things like large software for server use (i.e.: queues, map-reduce), doing some lifting coupled with other languages, higher-level protocol implementation, etc. Areas in the middle will depend on you. You should not necessarily wall yourself in server software with Erlang: there have been cases of people doing unexpected and surprising things. One example is IANO, a robot created by the UNICT team, which uses Erlang for its artificial intelligence and won the silver medal at the 2009 eurobot competition. Another example is Wings 3D, an open source 3D modeler (but not a renderer) written in Erlang and thus cross-platform.

What you need to dive in

All you need to get started is a text editor and the Erlang environment. You can get the source code and the Windows binaries from the official Erlang website. I won't go into much installation details, but for Windows, just download and run the binary files. Don't forget to add your Erlang directory to your PATH system variable to be able to access it from the command line.

On Debian-based Linux distributions, you should be able to install the package by doing $ apt-get install erlang. On Fedora (if you have 'yum' installed), you can achieve the same by typing # yum install erlang. However, these repositories often hold outdated versions of the Erlang packages; Using an outdated version could give you some differences with what you'll get from this tutorial and a hit in performance with certain applications. I thus encourage you to compile from source. Consult the README file within the package and Google to get all the installing details you'll need, they'll do a far better job than I ever will.

If you're on OSX, I can't really help you guys as I don't own a Mac, but there are ways to install Erlang with the help of MacPorts.

Note: at the time of this writing, I'm using Erlang version R13B+, so for best results, you should use that version or newer ones.

Where to get Help

There are a few places where you can get help. If you're using linux, you can access the man pages for good technical documentation. Erlang has a lists module (which we'll soon see): to get the documentation on lists, just type in $ erl -man lists.

On Windows, the installation should include HTML documentation. You can download it at any time from the official erlang site, or consult one of the cleaner alternative sites.

Good coding practices can be found here once you feel you need to get everything clean. The code in this book will attempt to follow these guidelines, too.

Now, there are times where just getting the technical details isn't enough. When that happens, I tend to turn to two main sources: the official mailing list (you should follow it just to learn a bunch) and the #erlang and #erlang-otp channels on irc.freenode.net.

Oh and if you're the type of person to go for cookbooks and pre-made recipes, trapexit is the place you're looking for. They also mirror the mailing lists as a forum and a general wiki, which can always be helpful.

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Starting Out

The Shell

In Erlang, you can test most of your stuff in an emulator; it will run your scripts when compiled and deployed, but it will also let you edit stuff live. To start the shell in Linux, open a terminal and then type in $ erl. If you've set up everything fine, you should see text like this:

Erlang R13B01 (erts-5.7.2) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.2  (abort with ^G)

Congratulations, you're running the Erlang shell!

For Windows users, you can still run the erl.exe shell, but it's recommended you instead use werl.exe, which can be found in your start menu (programs > Erlang). Werl is a windows-only implementation of the Erlang shell, having its own window with scrollbars and supporting command-line editing (like copy-pasting, which got to be a pain with the standard cmd.exe shell in Windows). The erl shell is still required if you want to redirect standard input or output, or use pipelines.

We'll be able to enter and run code in the emulator, but first, let's see how we can get around in it.

Shell Commands

The Erlang shell has a built-in line editor based on a subset of Emacs, a popular text editor that's been in use since the 70s. If you know Emacs, you should be fine. For the others, you'll do fine anyway.

First of all, if you type some text and then go ^A (Ctrl+A), you should see your cursor moving to the beginning of the line. ^E (Ctrl+E) gets you to the end. You can use arrow keys to go forward, backwards, show previous or next lines so you can repeat code.

If you type something like li and then press "tab", the shell will have completed the terms for you to lists:. Press tab again, and the shell will suggest you many functions to use after. This is Erlang completing the module lists and then suggesting functions from it. You may find the notation weird, but don't worry, you'll get familiar with it soon enough.

I think we've seen enough of shell functionality to be alright, except for one thing: we don't know how to leave! There's a fast way to find how. Just type in help(). and you should get information on a bunch of commands you can use in the shell. We'll use some of them at a later point, but the only line of concern to us in order to get out is

q() -- quit - shorthand for init:stop()

So this is one way to do it (in fact, two ways). But this won't help us if the shell freezes! If you were paying attention, when you started the shell, there was a comment about 'aborting with ^G'. Let's do that, and then press h to get help!

User switch command

--> h

c [nn]            - connect to job

i [nn]            - interrupt job

k [nn]            - kill job

j                 - list all jobs

s [shell]         - start local shell

r [node [shell]]  - start remote shell

q        - quit erlang

? | h             - this message

-->

If you type in i then c, Erlang should stop the currently running code and bring you back to a responsive shell. j will give you a list of processes running (a star after a number indicates this is the job you are currently running), which you can then interrupt with i followed by the number. If you use k, you will kill the shell as it is instead of just interrupting it. Press s to start a new one.

Eshell V5.7.2  (abort with ^G)

1> "OH NO THIS SHELL IS UNRESPONSIVE!!! *hits ctrl+G*"

User switch command

--> k

--> c

Unknown job

--> s

--> j

2* {shell,start,[]}

--> c 2

Eshell V5.7.2  (abort with ^G)

1> "YESS!"

If you read back the help text, you'll notice we can start remote shells. I won't get into details right now, but this should give you an idea of what the Erlang VM can do apart from running code. For now, let's get things started (for real).

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Starting Out (for real)

Erlang is a relatively small and simple language (in the way C is simpler than C++). There are a few basic data types built in the language, and as such, this chapter will cover most of them. Reading it is strongly advised as it explains the building blocks for all the programs you'll write with Erlang later on.

Numbers

In the Erlang shell, expressions have to be terminated with a period followed by whitespace (line break, a space etc.), otherwise they won't be executed. You can separate expressions with commas, but only the result of the last one will be shown (the others are still executed). This is certainly unusual syntax for most people and it comes from the days Erlang was implemented directly in Prolog, a logic programming language.

Open the Erlang shell as described in the previous chapters and let's type them things!

1> 2 + 15.

17

2> 49 * 100.

4900

3> 1892 - 1472.

420

4> 5 / 2.

2.5

5> 5 div 2.

2

6> 5 rem 2.

1

You should have noticed Erlang doesn't care if you enter floating point numbers or integers: both types are supported when dealing with arithmetic. Integers and floating values are pretty much the only types of data Erlang's mathematical operators will handle transparently for you. However, if you want to have the integer-to-integer division, use div, and to have the modulo operator, use rem (remainder).

Note that we can use several operators in a single expression, and mathematical operations obey the normal precedence rules.

7> (50 * 100) - 4999.

1

8> -(50 * 100 - 4999).

-1

9> -50 * (100 - 4999).

244950

If you want to express integers in other bases than base 10, just enter the number as Base#Value (given Base is in the range 2..36):

10> 2#101010.

42

11> 8#0677.

447

12> 16#AE.

174

Awesome! Erlang has the power of the calculator you have on the corner of your desk with a weird syntax on top of it! Absolutely exciting!

Invariable Variables

Doing arithmetic is alright, but you won't go far without being able to store results somewhere. For that, we'll use variables. If you have read the intro to this book, you'll know that variables can't be variable in functional programming. The basic behavior of variables can be demonstrated with these 7 expressions (note that variables begin with an uppercase letter):

1> One.

* 1: variable 'One' is unbound

2> One = 1.

1

3> Un = Uno = One = 1.

1

4> Two = One + One.

2

5> Two = 2.

2

6> Two = Two + 1.

** exception error: no match of right hand side value 3

7> two = 2.

** exception error: no match of right hand side value 2

The first thing these commands tell us is that you can assign a value to a variable exactly once; then you can 'pretend' to assign a value to a variable if it's the same value it already has. If it's different, Erlang will complain. It's a correct observation, but the explanation is a bit more complex and depends on the = operator. The = operator (not the variables) has the role of comparing values and complaining if they're different. If they're the same, it returns the value:

8> 47 = 45 + 2.

47

9> 47 = 45 + 3.

** exception error: no match of right hand side value 48

What this operator does when mixed with variables is that if the left-hand side term is a variable and it is unbound (has no value associated to it), Erlang will automatically bind the right-hand side value to the variable on the left-hand side. The comparison will consequently succeed and the variable will keep the value in memory.

This behavior of the = operator is the basis of something called 'Pattern matching', which many functional programming languages have, although Erlang's way of doing things is usually regarded as more flexible and complete than alternatives. We'll see pattern matching with more detail when we visit the tuple and list types in this very chapter, and also with functions in the following chapters.

The other thing the commands 1-7 told us is that variable names must begin with a capital letter. Command 7 failed because the word two had a lowercase letter to begin with. Technically, variables can start with an underscore ('_') too, but by convention their use is restricted to values you do not care about, yet you felt it was necessary to document what it contains.

You can also have variables that are only an underscore:

10> _ = 14+3.

17

11> _.

* 1: variable '_' is unbound

Unlike any other kind of variable, it won't ever store any value. Totally useless for now, but you'll know it exists when we need it.

Note: If you're testing in the shell and save the wrong value to a variable, it is possible to 'erase' that variable by using the function f(Variable).. If you wish to clear all variable names, do f()..

These functions are there only to help you when testing and only work in the shell. When writing real programs, we won't be able to destroy values that way. Being able to do it only in the shell makes sense if you acknowledge Erlang being usable in industrial scenarios: it is wholly possible to have a shell being active for years without interruption... Let's bet that the variable X would be used more than once in that time period.

Atoms

There is a reason why variables names can't begin with a lowercase character: atoms. Atoms are literals, constants with their own name for value. What you see is what you get and don't expect more. The atom cat means "cat" and that's it. You can't play with it, you can't change it, you can't smash it to pieces; it's cat. Deal with it.

While single words starting with a lowercase letter is a way to write an atom, there's more than one manner to do it:

1> atom.

atom

2> atoms_rule.

atoms_rule

3> atoms_rule@erlang.

atoms_rule@erlang

4> 'Atoms can be cheated!'.

'Atoms can be cheated!'

5> atom = 'atom'.

atom

An atom should be enclosed in single quotes (') if it does not begin with a lower-case letter or if it contains other characters than alphanumeric characters, underscore (_), or @.

Expression 5 also shows than an atom with single quotes is exactly the same as a similar atom without them.

I compared atoms to constants having their name as their values. You may have worked with code that used constants before: as an example, let's say I have values for eye colors: BLUE -> 1, BROWN -> 2, GREEN -> 3, OTHER -> 4. You need to match the name of the constant to some underlying value. Atoms let you forget about the underlying values: my eye colors can simply be 'blue', 'brown', 'green' and 'other'. These colors can be used anywhere in any piece of code: the underlying values will never clash and it is impossible for such a constant to be undefined! If you really want constants with values associated to them, there's a way to do it that we'll see in chapter 4 (Modules).

An atom is therefore mainly useful to express or qualify data coupled with it. Used alone, it's a bit harder to find a good use to it. This is why we won't spend more time toying with them; their best use will come when coupled with other types of data.

Don't drink too much Kool-Aid:

Atoms are really nice and a great way to send messages or represent constants. However there are pitfalls to using atoms for too many things: an atom is referred to in an "atom table" which consumes memory (4 bytes/atom in a 32-bit system, 8 bytes/atom in a 64-bit system). The atom table is not garbage collected, and so atoms will accumulate until the system tips over, either from memory usage or because 1048577 atoms were declared.

This means atoms should not be generated dynamically for whatever reason; if your system has to be reliable and user input lets someone crash it at will by telling it to create atoms, you're in serious trouble. Atoms should be seen as tools for the developer because honestly, it's what they are.

Note: some atoms are reserved words and can not be used except for what the language designers wanted them to be: function names, operators, expressions, etc. These are: after and andalso band begin bnot bor bsl bsr bxor case catch cond div end fun if let not of or orelse query receive rem try when xor

Boolean Algebra & Comparison operators

One would be in pretty deep trouble if one couldn't tell the difference between what's small and big, what's true and false. As any other language, Erlang has ways to let you use boolean operations and to compare items.

Boolean algebra is dirt simple:

1> true and false.

false

2> false or true.

true

3> true xor false.

true

4> not false.

true

5> not (true and true).

false

Note: the boolean operators and and or will always evaluate arguments on both sides of the operator. If you want to have the short-circuit operators (which will only evaluate the right-side argument if it needs to), use andalso and orelse.

Testing for equality or inequality is also dirt simple, but has slightly different symbols from those you see in many other languages:

6> 5 =:= 5.

true

7> 1 =:= 0.

false

8> 1 =/= 0.

true

9> 5 =:= 5.0.

false

10> 5 == 5.0.

true

11> 5 /= 5.0.

false

First of all, if your usual language uses == and != to test for and against equality, Erlang uses =:= and =/=. The three last expressions (lines 9 to 11) also introduce us to a pitfall: Erlang won't care about floats and integers in arithmetic, but will do so when comparing them. No worry though, because the == and /= operators are there to help you in these cases. This is important to remember whether you want exact equality or not.

Other operators for comparisons are < (less than), > (greater than), >= (greater than or equal to) and =< (less than or equal to). That last one is backwards (in my opinion) and is the source of many syntax errors in my code. Keep an eye on that =<.

12> 1 < 2.

true

13> 1 < 1.

false

14> 1 >= 1.

true

15> 1 =< 1.

true

What happens when doing 5 + llama or 5 == true? There's no better way to know than trying it and subsequently getting scared by error messages!

12> 5 + llama.

** exception error: bad argument in an arithmetic expression

in operator  +/2

called as 5 + llama

Welp! Erlang doesn't really like you misusing some of its fundamental types! The emulator returns a nice error message here. It tells us it doesn't like one of the two arguments used around the + operator!

Erlang getting mad at you for wrong types is not always true though:

13> 5 =:= true.

false

Why does it refuse different types in some operations but not others? While Erlang doesn't let you add anything with everything, it will let you compare them. This is because the creators of Erlang thought pragmaticism beats theory and decided it would be great to be able to simply write things like general sorting algorithms that could order any term. It's there to make your life simpler and can do so the vast majority of the time.

There is one last thing to keep in mind when doing boolean algebra and comparisons:

14> 0 == false.

false

15> 1 < false.

true

Chances are you're pulling your hair if you come from procedural languages or most object-oriented languages. Line 14 should evaluate to true and line 15 to false! After all, false means 0 and true is anything else! Except in Erlang. Because I lied to you. Yes, I did that. Shame on me.

Erlang has no such things as boolean true and false. The terms true and false are atoms, but they are integrated well enough into the language you shouldn't have a problem with that as long as you don't expect false and true to mean anything but false and true.

Note: The correct ordering of each element in a comparison is the following:

number < atom < reference < fun < port < pid < tuple < list < bit string

You don't know all these types of things yet, but you will get to know them through the book. Just remember that this is why you can compare anything with anything! To quote Joe Armstrong, one of the creators of Erlang: "The actual order is not important - but that a total ordering is well defined is important."

Tuples

A tuple is a way to organize data. It's a way to group together many terms when you know how many there are. In Erlang, a tuple is written in the form {Element1, Element2, ..., ElementN}. As an example, you'd give me the coordinates (x,y) if you wanted to tell me the position of a point in a Cartesian graph. We can represent this point as a tuple of two terms:

1> X = 10, Y = 4.

4

2> Point = {X,Y}.

{10,4}

In this case, a point will always be two terms. Instead of carrying the variables X and Y around the place, you only have to carry one instead. However, what can I do if I receive a point and only want the X coordinate? It's not hard to extract that information. Remember that when we assigned values, Erlang would never complain if they were the same. Let's exploit that! You may need to clean the variables we had set with f().

3> Point = {4,5}.

{4,5}

4> {X,Y} = Point.

{4,5}

5> X.

4

6> {X,_} = Point.

{4,5}

From then on we can use X to get the first value of the tuple! How did that happen? First, X and Y had no value and were thus considered unbound variables. When we set them in the tuple {X,Y} on the left-hand side of the = operator, the = operator compares both values: {X,Y} vs. {4,5}. Erlang is smart enough to unpack the values from the tuple and distribute them to the unbound variables on the left-hand side. Then the comparison is only {4,5} = {4,5}, which obviously succeeds! That's one of the many forms of pattern matching.

Note that on expression 6, I used the anonymous _ variable. This is exactly how it's meant to be used: to drop the value that would usually be placed there since we won't use it. The _ variable is always seen as unbound and acts as a wildcard for pattern matching. Pattern matching to unpack tuples will only work if the number of elements (the tuple's length) is the same.

7> {_,_} = {4,5}.

{4,5}

8> {_,_} = {4,5,6}.

** exception error: no match of right hand side value {4,5,6}

Tuples can also be useful when working with single values. How so? The simplest example is temperature:

9> Temperature = 23.213.

23.213

Well, it sounds like a good day to go to the beach... Wait, is this temperature in Kelvin, Celsius or Fahrenheit?

10> PreciseTemperature = {celsius, 23.213}.

{celsius,23.213}

11> {kelvin, T} = PreciseTemperature.

** exception error: no match of right hand side value {celsius,23.213}

This throws an error, but it's exactly what we want! This is, again, pattern matching at work. The = operator ends up comparing {kelvin, T} and {celsius, 23.213}: even if the variable T is unbound, Erlang won't see the celsius atom as identical to the kelvin atom when comparing them. An exception is thrown which stops the execution of code. By doing so, the part of our program that expects a temperature in Kelvin won't be able to process temperatures sent in Celsius. This makes it easier for the programmer to know what is being sent around and also works as a debugging aid. A tuple which contains an atom with one element following it is called a 'tagged tuple'. Any element of a tuple can be of any type, even another tuple:

12> {point, {X,Y}}.

{point,{4,5}}

What if we want to carry around more than one Point though?

Lists!

Lists are the bread and butter of many functional languages. They're used to solve all kinds of problems and are undoubtedly the most used data structure in Erlang. Lists can contain anything! Numbers, atoms, tuples, other lists; your wildest dreams in a single structure. The basic notation of a list is [Element1, Element2, ..., ElementN] and you can mix more than one type of data in it:

1> [1, 2, 3, {numbers,[4,5,6]}, 5.34, atom].

[1,2,3,{numbers,[4,5,6]},5.34,atom]

Simple enough, right?

2> [97, 98, 99].

"abc"

Uh oh! This is one of the most disliked things in Erlang: strings! Strings are lists and the notation is absolutely the exact same! Why do people dislike it? Because of this:

3> [97,98,99,4,5,6].

[97,98,99,4,5,6]

4> [233].

"é"

Erlang will print lists of numbers as numbers only when at least one of them could not also represent a letter! There is no such thing as a real string in Erlang! This will no doubt come to haunt you in the future and you'll hate the language for it. Don't despair, because there are other ways to write strings we'll see later in this chapter.

Don't drink too much Kool-Aid:

This is why you may have heard Erlang is said to suck at string manipulation: there is no built-in string type like in most other languages. This is because of Erlang's origins as a language created and used by telecom companies. They never (or rarely) used strings and as such, never felt like adding them officially. However, most of Erlang's lack of sense in string manipulations is getting fixed with time: The VM now natively supports Unicode strings, and overall gets faster on string manipulations all the time.

There is also a way to store strings as a binary data structure, making them really light and faster to work with. All in all, there are still some functions missing from the standard library and while string processing is definitely doable in Erlang, there are somewhat better languages for tasks that need lots of it, like Perl or Python.

To glue lists together, we use the ++ operator. The opposite of ++ is -- and will remove elements from a list:

5> [1,2,3] ++ [4,5].

[1,2,3,4,5]

6> [1,2,3,4,5] -- [1,2,3].

[4,5]

7> [2,4,2] -- [2,4].

[2]

8> [2,4,2] -- [2,4,2].

[]

Both ++ and -- are right-associative. This means the elements of many -- or ++ operations will be done from right to left, as in the following examples:

9> [1,2,3] -- [1,2] -- [3].

[3]

10> [1,2,3] -- [1,2] -- [2].

[2,3]

Let's keep going. The first element of a list is named the Head, and the rest of the list is named the Tail. We will use two built-in functions (BIF) to get them.

11> hd([1,2,3,4]).

1

12> tl([1,2,3,4]).

[2,3,4]

Note: built-in functions (BIFs) are usually functions that could not be implemented in pure Erlang, and as such are defined in C, or whichever language Erlang happens to be implemented on (it was Prolog in the 80's). There are still some BIFs that could be done in Erlang but were still implemented in C in order to provide more speed to common operations. One example of this is the length(List) function, which will return the (you've guessed it) length of the list passed in as the argument.

Accessing or adding the head is fast and efficient: virtually all applications where you need to deal with lists will always operate on the head first. As it's used so frequently, there is a nicer way to separate the head from the tail of a list with the help of pattern matching: [Head|Tail]. Here's how you would add a new head to a list:

13> List = [2,3,4].

[2,3,4]

14> NewList = [1|List].

[1,2,3,4]

When processing lists, as you usually start with the head, you want a quick way to also store the tail to later operate on it. If you remember the way tuples work and how we used pattern matching to unpack the values of a point ({X,Y}), you'll know we can get the first element (the head) sliced off a list in a similar manner.

15> [Head|Tail] = NewList.

[1,2,3,4]

16> Head.

1

17> Tail.

[2,3,4]

18> [NewHead|NewTail] = Tail.

[2,3,4]

19> NewHead.

2

The | we used is named the cons operator (constructor). In fact, any list can be built with only cons and values:

20> [1 | []].

[1]

21> [2 | [1 | []]].

[2,1]

22> [3 | [2 | [1 | []] ] ].

[3,2,1]

This is to say any list can be built with the following formula: [Term1| [Term2 | [... | [TermN]]]]. Lists can thus be defined recursively as a head preceding a tail, which is itself a head followed by more heads. In this sense we could imagine a list being a bit like an earthworm: you can slice it in half and you'll then have two worms.

The ways Erlang lists can be built are sometimes confusing to people who are not used to similar constructors. To help you get familiar with the concept, read all of these examples (hint: they're all equivalent):

[a, b, c, d]

[a, b, c, d | []]

[a, b | [c, d]]

[a, b | [c | [d]]]

[a | [b | [c | [d]]]]

[a | [b | [c | [d | [] ]]]]

With this understood, you should be able to deal with list comprehensions.

Note: Using the form [1 | 2] gives what we call an 'improper list'. Improper lists will work when you pattern match in the [Head|Tail] manner, but will fail to be used with standard functions of Erlang (even length()). This is because Erlang expects proper lists. Proper lists end with an empty list as their last cell. When declaring an item like [2], the list is automatically formed in a proper manner. As such, [1|[2]] would work! Improper lists, although syntactically valid, are of very limited use outside of user-defined data structures.

List Comprehensions

List comprehensions are ways to build or modify lists. They also make programs short and easy to understand compared to other ways of manipulating lists. It's based off the idea of set notation; if you've ever taken mathematics classes with set theory or if you've ever looked at mathematical notation, you probably know how that works. Set notation basically tells you how to build a set by specifying properties its members must satisfy. List comprehensions may be hard to grasp at first, but they're worth the effort. They make code cleaner and shorter, so don't hesitate to try and type in the examples until you understand them!

An example of set notation would be . That set notation tells you the results you want will be all real numbers who are equal to their own square. The result of that set would be {0,1}. Another set notation example, simpler and abbreviated would be {x : x > 0}. Here, what we want is all numbers where x > 0.

List comprehensions in Erlang are about building sets from other sets. Given the set {2n : n in L} where L is the list [1,2,3,4], the Erlang implementation would be:

1> [2*N || N <- [1,2,3,4]].

[2,4,6,8]

Compare the mathematical notation to the Erlang one and there's not a lot that changes: brackets ({}) become square brackets ([]), the colon (:) becomes two pipes (||) and the word 'in' becomes the arrow (<-). We only change symbols and keep the same logic. In the example above, each value of [1,2,3,4] is sequentially pattern matched to N. The arrow acts exactly like the = operator, with the exception that it doesn't throw exceptions.

You can also add constraints to a list comprehension by using operations that return boolean values. if we wanted all the even numbers from one to ten, we could write something like:

2> [X || X <- [1,2,3,4,5,6,7,8,9,10], X rem 2 =:= 0].

[2,4,6,8,10]

Where X rem 2 =:= 0 checks if a number is even. Practical applications come when we decide we want to apply a function to each element of a list, forcing it to respect constraints, etc. As an example, say we own a restaurant. A customer enters, sees our menu and asks if he could have the prices of all the items costing between $3 and $10 with taxes (say 7%) counted in afterwards.

3> RestaurantMenu = [{steak, 5.99}, {beer, 3.99}, {poutine, 3.50}, {kitten, 20.99}, {water, 0.00}].

[{steak,5.99},

{beer,3.99},

{poutine,3.5},

{kitten,20.99},

{water,0.0}]

4> [{Item, Price*1.07} || {Item, Price} <- RestaurantMenu, Price >= 3, Price =< 10].

[{steak,6.409300000000001},{beer,4.2693},{poutine,3.745}]

Of course, the decimals aren't rounded in a readable manner, but you get the point. The recipe for list comprehensions in Erlang is therefore NewList = [Expression || Pattern <- List, Condition1, Condition2, ... ConditionN]. The part Pattern <- List is named a Generator expression. You can have more than one!

5> [X+Y || X <- [1,2], Y <- [2,3]].

[3,4,4,5]

This runs the operations 1+2, 1+3, 2+2, 2+3. So if you want to make the list comprehension recipe more generic, you get: NewList = [Expression || GeneratorExp1, GeneratorExp2, ..., GeneratorExpN, Condition1, Condition2, ... ConditionN]. Note that the generator expressions coupled with pattern matching also act as a filter:

6> Weather = [{toronto, rain}, {montreal, storms}, {london, fog},

6>            {paris, sun}, {boston, fog}, {vancouver, snow}].

[{toronto,rain},

{montreal,storms},

{london,fog},

{paris,sun},

{boston,fog},

{vancouver,snow}]

7> FoggyPlaces = [X || {X, fog} <- Weather].

[london,boston]

If an element of the list 'Weather' doesn't match the {X, fog} pattern, it's simply ignored in the list comprehension whereas the = operator would have thrown an exception.

There is one more basic data type left for us to see for now. It is a surprising feature that makes interpreting binary data easy as pie.

Bit Syntax!

Most languages have support for manipulating data such as numbers, atoms, tuples, lists, records and/or structs, etc. Most of them also only have very raw facilities to manipulate binary data. Erlang goes out of its way to provide useful abstractions when dealing with binary values with pattern matching taken to the next level. It makes dealing with raw binary data fun and easy (no, really), which was necessary for the telecom applications it was created to help with. Bit manipulation has a unique syntax and idioms that may look kind of weird at first, but if you know how bits and bytes generally work, this should make sense to you. You may want to skip the rest of this chapter otherwise.

Bit syntax encloses binary data between << and >>, splits it in readable segments, and each segment is separated by a comma. A segment is a sequence of bits of a binary (not necessarily on a byte boundary, although this is the default behaviour). Say we want to store an orange pixel of true color (24 bits). If you've ever checked colors in Photoshop or in a CSS style sheet for the web, you know the hexadecimal notation has the format #RRGGBB. A tint of orange is #F09A29 in that notation, which could be expanded in Erlang to:

1> Color = 16#F09A29.

15768105

2> Pixel = <<Color:24>>.

<<240,154,41>>

This basically says "Put the binary values of #F09A29 on 24 bits of space (Red on 8 bits, Green on 8 bits and Blue also on 8 bits) in the variable Pixel." The value can later be taken to be written to a file. This doesn't look like much, but once written to a file, what you'd get by opening it in a text editor would be a bunch of unreadable characters. When you read back from the file, Erlang would interpret the binary into the nice <<240,151,41>> format again!

What's more interesting is the ability to pattern match with binaries to unpack content:

3> Pixels = <<213,45,132,64,76,32,76,0,0,234,32,15>>.

<<213,45,132,64,76,32,76,0,0,234,32,15>>

4> <<Pix1,Pix2,Pix3,Pix4>> = Pixels.

** exception error: no match of right hand side value <<213,45,132,64,76,32,76,

0,0,234,32,15>>

5> <<Pix1:24, Pix2:24, Pix3:24, Pix4:24>> = Pixels.

<<213,45,132,64,76,32,76,0,0,234,32,15>>

What we did on command 3 is declare what would be precisely 4 pixels of RGB colors in binary.

On expression 4, we tried to unpack 4 values from the binary content. It throws an exception, because we have more than 4 segments, we in fact have 12! So what we do is tell Erlang that each variable on the left side will hold 24 bits of data. That's what Var:24 means. We can then take the first pixel and unpack it further into single color values:

6> <<R:8, G:8, B:8>> = <<Pix1:24>>.

<<213,45,132>>

7> R.

213

"Yeah that's dandy. What if I only wanted the first color from the start though? will I have to unpack all these values all the time?" Hah! Doubt not! Erlang introduces more syntactic sugar and pattern matching to help you around:

8> <<R:8, Rest/binary>> = Pixels.

<<213,45,132,64,76,32,76,0,0,234,32,15>>

9> R.

213

Nice, huh? That's because Erlang accepts more than one way to describe a binary segment. Those are all valid:

Value

Value:Size

Value/TypeSpecifierList

Value:Size/TypeSpecifierList

where Size is always in bits and TypeSpecifierList represents one or more of the following:

Type

Possible values: integer | float | binary | bytes | bitstring | bits | utf8 | utf16 | utf32

This represents the kind of binary data used. Note that 'bytes' is shorthand for 'binary' and 'bits' is shorthand for 'bitstring'. When no type is specified, Erlang assumes an 'integer' type.

Signedness

Possible values: signed | unsigned

Only matters for matching when the type is integer. The default is 'unsigned'.

Endianness

Possible values: big | little | native

Endianness only matters when the Type is either integer, utf16, utf32, or float. This has to do with how the system reads binary data. As an example, the BMP image header format holds the size of its file as an integer stored on 4 bytes. For a file that has a size of 72 bytes, a little-endian system would represent this as <<72,0,0,0>> and a big-endian one as <<0,0,0,72>>. One will be read as '72' while the other will be read as '1207959552', so make sure you use the right endianness. There is also the option to use 'native', which will choose at run-time if the CPU uses little-endianness or big-endianness natively. By default, endianness is set to 'big'.

Unit

written unit:Integer

This is the size of each segment, in bits. The allowed range is 1..256 and is set by default to 1 for integers, floats and bit strings and to 8 for binary. The utf8, utf16 and utf32 types require no unit to be defined. The multiplication of Size by Unit is equal to the number of bits the segment will take and must be evenly divisible by 8. The unit size is usually used to ensure byte-alignment.

The TypeSpecifierList is built by separating attributes by a '-'.

Some examples may help digest the definitions:

10> <<X1/unsigned>> =  <<-44>>.

<<"Ô">>

11> X1.

212

12> <<X2/signed>> =  <<-44>>.

<<"Ô">>

13> X2.

-44

14> <<X2/integer-signed-little>> =  <<-44>>.

<<"Ô">>

15> X2.

-44

16> <<N:8/unit:1>> = <<72>>.

<<"H">>

17> N.

72

18> <<N/integer>> = <<72>>.

<<"H">>

19> <<Y:4/little-unit:8>> = <<72,0,0,0>>.

<<72,0,0,0>>

20> Y.

72

You can see there are more than one way to read, store and interpret binary data. This is a bit confusing, but still much simpler than using the usual tools given by most languages.

The standard binary operations (shifting bits to left and right, binary 'and', 'or', 'xor', or 'not') also exist in Erlang. Just use the functions bsl (Bit Shift Left), bsr (Bit Shift Right), band, bor, bxor, and bnot.

2#00100 = 2#00010 bsl 1.

2#00001 = 2#00010 bsr 1.

2#10101 = 2#10001 bor 2#00101.

With that kind of notation and the bit syntax in general, parsing and pattern matching binary data is a piece of cake. One could parse TCP segments with code like this:

<<SourcePort:16, DestinationPort:16,

AckNumber:32,

DataOffset:4, _Reserved:4, Flags:8, WindowSize:16,

CheckSum: 16, UrgentPointer:16,

Payload/binary>> = SomeBinary.

The same logic can then be applied to anything binary: video encoding, images, other protocol implementations, etc.

Don't drink too much Kool-Aid:

Erlang is slow compared to languages like C or C++. Unless you are a patient person, it would be a bad idea to do stuff like converting videos or images with it, even though the binary syntax makes it extremely interesting as I hinted above. Erlang is just not that great at heavy number crunching.

Take note, however, that Erlang is still mighty fast for applications that do not require number crunching: reacting to events, message passing (with the help of atoms being extremely light), etc. It can deal with events in matters of milliseconds and as such is a great candidate for soft-real-time applications.

There's a whole other aspect to binary notation: bit strings. Binary strings are bolted on top of the language the same way they are with lists, but they're much more efficient in terms of space. This is because normal lists are linked lists (1 'node' per letter) while bit strings are more like C arrays. Bit strings use the syntax <<"this is a bit string!">>. The downside of bit strings compared to lists is a loss in simplicity when it comes to pattern matching and manipulation. Consequently, people tend to use bit strings when storing text that won't be manipulated too much or when space efficiency is a real issue. We'll see strings and bit strings more in depth in chapter 9. For the time being, that's all we need to know.

Note: Even though bit strings are pretty light, you should avoid using them to tag values. It could be tempting to use string literals to say {<<"temperature">>,50}, but always use atoms when doing that. Previously in this chapter, atoms were said to be taking only 4 or 8 bytes in space, no matter how long they are. By using them, you'll have basically no overhead when copying data from function to function or sending it to another Erlang node on another server.

Conversely, do not use atoms to replace strings because they are lighter. Strings can be manipulated (splitting, regular expressions, etc) while atoms can only be compared and nothing else.

Binary Comprehensions

Binary comprehensions are to bit syntax what list comprehensions are to lists: a way to make code short and concise. They are relatively new in the Erlang world as they were there in previous revisions of Erlang, but required a module implementing them to use a special compile flag in order to work. Since the R13B revisions (those used here), they've become standard and can be used anywhere, including the shell:

1> [ X || <<X>> <= <<1,2,3,4,5>>, X rem 2 == 0].

[2,4]

The only change in syntax from regular list comprehensions is the <- which became <= and using binaries (<<>>) instead of lists ([]). Earlier in this chapter we've seen an example where there was a binary value of many pixels on which we used pattern matching to grab the RGB values of each pixel. It was alright, but on larger structures, it would become possibly harder to read and maintain. The same exercise can be done with a one-line binary comprehension, which is much cleaner:

2> Pixels = <<213,45,132,64,76,32,76,0,0,234,32,15>>.

<<213,45,132,64,76,32,76,0,0,234,32,15>>

3> RGB = [ {R,G,B} || <<R:8,G:8,B:8>> <= Pixels ].

[{213,45,132},{64,76,32},{76,0,0},{234,32,15}]

Changing <- to <= let us use a binary stream as a generator. The complete binary comprehension basically changed binary data to integers inside tuples. Another binary comprehension syntax exists to let you do the exact opposite:

4> << <<R:8, G:8, B:8>> ||  {R,G,B} <- RGB >>.

<<213,45,132,64,76,32,76,0,0,234,32,15>>

Be careful, as the elements of the resulting binary require a clearly defined size if the generator returned binaries:

5> << <<Bin>> || Bin <- [<<3,7,5,4,7>>] >>.

** exception error: bad argument

6> << <<Bin/binary>> || Bin <- [<<3,7,5,4,7>>] >>.

<<3,7,5,4,7>>

It's also possible to have a binary comprehension with a binary generator, given the fixed-size rule above is respected:

7> << <<(X+1)/integer>> || <<X>> <= <<3,7,5,4,7>> >>.

<<4,8,6,5,8>>

Note: At the time of this writing, binary comprehensions were seldom used and not documented very well. As such, it was decided not to dig more than what is necessary to identify them and understand their basic working. To understand more bit syntax as a whole, read the white paper defining their specification.

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Modules

What are modules

Working with the interactive shell is often considered a vital part of using dynamic programming languages. It is useful to test all kinds of code and programs. Most of the basic data types of Erlang were used without even needing to open a text editor or saving files. You could drop your keyboard, go play ball outside and call it a day, but you would be terrible Erlang programmer if you stopped right there. Code needs to be saved somewhere to be used!

This is what modules are for. Modules are a bunch of functions regrouped in a single file, under a single name. Additionally, all functions in Erlang must be defined in modules. You have already used modules, perhaps without realizing it. The BIFs mentioned in the previous chapter, like hd or tl, actually belong to the erlang module, as well as all of the arithmetic, logic and Boolean operators. BIFs from the erlang module differ from other functions as they are automatically imported when you use Erlang. Every other function defined in a module you will ever use needs to be called with the form Module:Function(Arguments).

You can see for yourself:

1> erlang:element(2, {a,b,c}).

b

2> element(2, {a,b,c}).

b

3> lists:seq(1,4).

[1,2,3,4]

4> seq(1,4).

** exception error: undefined shell command seq/2

Here, the seq function from the list module was not automatically imported, while element was. The error 'undefined shell command' comes from the shell looking for a shell command like f() and not being able to find it. There are some functions from the erlang module which are not automatically imported, but they're not used too frequently.

Logically, you should put functions about similar things inside a single module. Common operations on lists are kept in the lists module, while functions to do input and output (such as writing to the terminal or in a file) are regrouped in the io module. One of the only modules you will encounter which doesn't respect that pattern is the aforementioned erlang module that has functions which do math, conversions, deal with multiprocessing, fiddle with the virtual machine's settings, etc. They have no point in common except being built-in functions. You should avoid creating modules like erlang and instead focus on clean logical separations.

Module Declaration

When writing a module, you can declare two kinds of things: functions and attributes. Attributes are metadata describing the module itself such as its name, the functions that should be visible to the outside world, the author of the code, and so on. This kind of metadata is useful because it gives hints to the compiler on how it should do its job, and also because it lets people retrieve useful information from compiled code without having to consult the source.

There is a large variety of module attributes currently used in Erlang code across the world; as a matter of fact, you can even declare your own attributes for whatever you please. There are some pre-defined attributes that will appear more frequently than others in your code. All module attributes follow the form -Name(Attribute).. Only one of them is necessary for your module to be compilable:

-module(Name).

This is always the first attribute (and statement) of a file, and for good reason: it's the name of the current module, where Name is an atom. This is the name you'll use to call functions from other modules. The calls are made with the M:F(A) form, where M is the module name, F the function, and A the arguments.

It's time to code already! Our first module will be very simple and useless. Open your text editor and type in the following, then save it under useless.erl:

-module(useless).

This line of text is a valid module. Really! Of course it's useless without functions. Let's first decide what functions will be exported from our 'useless' module. To do this, we will use another attribute:

-export([Function1/Arity, Function2/Arity, ..., FunctionN/Arity]).

This is used to define what functions of a module can be called by the outside world. It takes a list of functions with their respective arity. The arity of a function is an integer representing how many arguments can be passed to the function. This is critical information, because different functions defined within a module can share the same name if and only if they have a different arity. The functions add(X,Y) and add(X,Y,Z) would thus be considered different and written in the form add/2 and add/3 respectively.

Note: Exported functions represent a module's interface. It is important to define an interface revealing strictly what is necessary for it to be used and nothing more. Doing so lets you fiddle with all the other [hidden] details of your implementation without breaking code that might depend on your module.

Our useless module will first export a useful function named 'add', which will take two arguments. The following -export attribute can be added after the module declaration:

-export([add/2]).

And now write the function:

add(A,B) ->

A + B.

The syntax of a function follows the form Name(Args) -> Body., where Name has to be an atom and Body can be one or more Erlang expressions separated by commas. The function is ended with a period. Note that Erlang doesn't use the 'return' keyword. 'Return' is useless! Instead, the last logical expression of a function to be executed will have its value returned to the caller automatically without you having to mention it.

Add the following function (why yes, every tutorial needs a 'Hello world' example! Even at the fourth chapter!), without forgetting to add it to the -export attribute.

%% Shows greetings.

%% io:format/1 is the standard function used to output text.

hello() ->

io:format("Hello, world!~n").

What we see from this function is that comments are single-line only and begin with a % sign (using %% is purely a question of style.) The hello/0 function also demonstrates how to call functions from foreign modules inside yours. In this case, io:format/1 is the standard function to output text, as written in the comments.

A last function will be added to the module, using both functions add/2 and hello/0:

greet_and_add_two(X) ->

hello(),

add(X,2).

Do not forget to add greet_and_add_two/1 to the exported function list. The calls to hello/0 and add/2 don't need to have the module name prepended to them because they were declared in the module itself.

Had you wanted to be able to call io:format/1 in the same manner as add/2 or any other function defined within the module, you could have added the following module attribute at the beginning of the file: -import(io, [format/1]).. Then you could have called format("Hello, World!~n"). directly. More generally, the -import attribute follows this recipe:

-import(Module, [Function1/Arity, ..., FunctionN/Arity]).

Importing a function is not much more than a shortcut for programmers when writing their code. Erlang programmers are often discouraged from using the -import attribute as some people find it reduces the readability of code. In the case of io:format/2, the function io_lib:format/2 also exists. Finding which one is used means going to the top of the file to see from which module it was imported. Consequently, leaving the module name in is considered good practice. Usually, the only functions you'll see imported come from the lists module: its functions are used with a higher frequency than those from most other modules.

Your useless module should now look like the following file:

-module(useless).

-export([add/2, hello/0, greet_and_add_two/1]).

add(A,B) ->

A + B.

%% Shows greetings.

%% io:format/1 is the standard function used to output text.

hello() ->

io:format("Hello, world!~n").

greet_and_add_two(X) ->

hello(),

add(X,2).

We are done with the "useless" module. You can save the file under the name useless.erl. The file name should be the module name as defined in the -module attribute, followed by '.erl', which is the standard Erlang source extension.

Before showing how to compile the module and finally try all its exciting functions, we will see how to define and use macros. Erlang macros are really similar to C's '#define' statements, mainly used to define short functions and constants. They are simple expressions represented by text that will be replaced before the code is compiled for the VM. Such macros are mainly useful to avoid having magic values floating around your modules. A macro is defined as a module attribute of the form: -define(MACRO, some_value). and is used as ?MACRO inside any function defined in the module. A 'function' macro could be written as -define(sub(X,Y), X-Y). and used like ?sub(23,47), later replaced by 23-47 by the compiler. Some people will use more complex macros, but the basic syntax stays the same.

Compiling the code

Erlang code is compiled to bytecode in order to be used by the virtual machine. You can call the compiler from many places: $ erlc flags file.erl when in the command line, compile:file(FileName) when in the shell or in a module, c() when in the shell, etc.

It's time to compile our useless module and try it. Open the Erlang shell, type in:

1> cd("/path/to/where/you/saved/the-module/").

"Path Name to the directory you are in"

ok

By default, the shell will only look for files in the same directory it was started in and the standard library: cd/1 is a function defined exclusively for the Erlang shell, telling it to change the directory to a new one so it's less annoying to browse for our files. When this is done, do the following:

2> c(useless).

{ok,useless}

If you have another message, make sure the file is named correctly, that you are in the right directory and that you've made no mistake in your module. Once you successfully compile code, you'll notice that a useless.beam file was added next to useless.erl in your directory. This is the compiled module. Let's try our first functions ever:

3> useless:add(7,2).

9

4> useless:hello().

Hello, world!

ok

5> useless:greet_and_add_two(-3).

Hello, world!

-1

6> useless:not_a_real_function().

** exception error: undefined function useless:not_a_real_function/0

The functions work as expected: add/2 adds numbers, hello/0 outputs "Hello, world!", and greet_and_add_two/1 does both! Of course, you might be asking why hello/0 returns the atom 'ok' after outputting text. This is because Erlang functions and expressions must always return something, even if they would not need to in other languages. As such, io:format/1 returns 'ok' to denote a normal condition, the absence of errors.

Expression 6 shows an error being thrown because a function doesn't exist. If you have forgotten to export a function, this is the kind of error message you will have when trying it out.

Note: If you were ever wondering, '.beam' stands for Bogdan/Björn's Erlang Abstract Machine, which is the VM itself. Other virtual machines for Erlang exist, but they're not really used anymore and are history: JAM (Joe's Abstract Machine, inspired by Prolog's WAM and old BEAM, which attempted to compile Erlang to C, then to native code. Benchmarks demonstrated little benefits in this practice and the concept was given up.

There are a whole lot of compilation flags existing to get more control over how a module is compiled. You can get a list of all of them in the Erlang documentation. The most common flags are:

-debug_info

Erlang tools such as debuggers, code coverage and static analysis tools will use the debug information of a module in order to do their work.

-{outdir,Dir}

By default, the Erlang compiler will create the 'beam' files in the current directory. This will let you choose where to put the compiled file.

-export_all

Will ignore the -export module attribute and will instead export all functions defined. This is mainly useful when testing and developing new code, but should not be used in production.

-{d,Macro} or {d,Macro,Value}

Defines a macro to be used in the module, where Macro is an atom. This is more frequently used when dealing when unit-testing, ensuring that a module will only have its testing functions created and exported when they are explicitly wanted. By default, Value is 'true' if it's not defined as the third element of the tuple.

To compile our useless module with some flags, we could do one of the following:

7> compile:file(useless, [debug_info, export_all]).

{ok,useless}

8> c(useless, [debug_info, export_all]).

{ok,useless}

You can also be sneaky and define compile flags from within a module, with a module attribute. To get the same results as from expressions 7 and 8, the following line could be added to the module:

-compile([debug_info, export_all]).

Then just compile and you'll get the same results as if you manually passed flags. Now that we're able to write down functions, compile them and execute them, it's time to see how far we can take them!

Note: another option is to compile your Erlang module to native code. Native code compiling is not available for every platform and OS, but on those that support it, it can make your programs go faster (about 20% faster, based on anecdotal evidence). To compile to native code, you need to use the hipe module and call it the following way: hipe:c(Module,OptionsList). You could also use c(Module,[{hipe,o3}]). when in the shell to achieve similar results. Note that the .beam file generated will no longer be portable across platforms like regular ones.

More About Modules

Before moving on to learning more about writing functions and barely useful snippets of code, there are a few other miscellaneous bits of information that might be useful to you in the future that I'd like to discuss.

The first one concerns metadata about modules. I mentioned in the beginning of this chapter that module attributes are metadata describing the module itself. Where can we find this metadata when we don't have an access to the source? Well the compiler plays nice with us: when compiling a module, it will pick up most module attributes and store them (along with other information) in a module_info/0 function. You can see the metadata of the useless module the following way:

9> useless:module_info().

[{exports,[{add,2},

{hello,0},

{greet_and_add_two,1},

{module_info,0},

{module_info,1}]},

{imports,[]},

{attributes,[{vsn,[174839656007867314473085021121413256129]}]},

{compile,[{options,[]},

{version,"4.6.2"},

{time,{2009,9,9,22,15,50}},

{source,"/home/ferd/learn-you-some-erlang/useless.erl"}]}]

10> useless:module_info(attributes).

[{vsn,[174839656007867314473085021121413256129]}]

The snippet above also shows an additional function, module_info/1 which will let you grab one specific piece of information. You can see exported functions, imported functions (none in this case!), attributes (this is where your custom metadata would go), and compile options and information. Had you decided to add -author("An Erlang Champ"). to your module, it would have ended up in the same section as vsn. There are limited uses to module attributes when it comes to production stuff, but they can be nice when doing little tricks to help yourself out: I'm using them in my testing script for this book to annotate functions for which unit tests could be better; the script looks up module attributes, finds the annotated functions and shows a warning about them.

Note: vsn is an automatically generated unique value differentiating each version of your code, excluding comments. It is used in code hot-loading (upgrading an application while it runs, without stopping it) and by some tools related to release handling. You can also specify a vsn value yourself if you want: just add -vsn(VersionNumber) to your module.

Another point that would be nice to approach regards general module design: avoid circular dependencies! A module A should not call a module B that also calls module A. Such dependencies usually end up making code maintenance difficult. In fact, depending on too many modules even if they're not in a circular dependency can make maintenance harder. The last thing you want is to wake up in the middle of the night only to find a maniac software engineer or computer scientist trying to gouge your eyes out because of terrible code you have written.

For similar reasons (maintenance and fear for your eyes), it is usually considered a good practice to regroup functions that have similar roles close together. Starting and stopping an application or creating and deleting a record in some database are examples of such a scenario.

Well, that's enough for the pedantic moralizations. How about we explore Erlang a little more?

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Syntax in functions

Pattern Matching

Now that we have the ability to store and compile our code, we can begin to write more advanced functions. Those that we have written so far were extremely simple and a bit underwhelming. We'll get to more interesting stuff. The first function we'll write will need to greet someone differently according to gender. In most languages you would need to write something similar to this:

function greet(Gender,Name)

if Gender == male then

print("Hello, Mr. %s!", Name)

else if Gender == female then

print("Hello, Mrs. %s!", Name)

else

print("Hello, %s!", Name)

end

With pattern-matching, Erlang saves you a whole lot of boilerplate code. A similar function in Erlang would look like this:

greet(male, Name) ->

io:format("Hello, Mr. ~s!", [Name]);

greet(female, Name) ->

io:format("Hello, Mrs. ~s!", [Name]);

greet(_, Name) ->

io:format("Hello, ~s!", [Name]).

I'll admit that the printing function is a lot uglier in Erlang than in many other languages, but that is not the point. The main difference here is that we used pattern matching to define both what parts of a function should be used and bind the values we need at the same time. There was no need to first bind the values and then compare them! So instead of:

function(Args)

if X then

Expression

else if Y then

Expression

else

Expression

We write:

function(X) ->

Expression;

function(Y) ->

Expression;

function(_) ->

Expression.

in order to get similar results, but in a much more declarative style. Each of these function declarations is called a function clause. Function clauses must be separated by semicolons (;) and together form a function declaration. A function declaration counts as one larger statement, and it's why the final function clause ends with a period. It's a "funny" use of tokens to determine workflow, but you'll get used to it. At least you'd better hope so because there's no way out of it!

Note: io:format's formatting is done with the help of tokens being replaced in a string. The character used to denote a token is the tilde (~). Some tokens are built-in such as ~n, which will be changed to a line-break. Most other tokens denote a way to format data. The function call io:format("~s!~n",["Hello"]). includes the token ~s, which accepts strings and bitstrings as arguments, and ~n. The final output message would thus be "Hello!\n". Another widely used token is ~p, which will print an Erlang term in a nice way (adding in indentation and everything).

The io:format function will be seen in more details in later chapters dealing with input/output with more depth, but in the meantime you can try the following calls to see what they do: io:format("~s~n",[<<"Hello">>]), io:format("~p~n",[<<"Hello">>]), io:format("~~~n"), io:format("~f~n", [4.0]), io:format("~30f~n", [4.0]). They're a small part of all that's possible and all in all they look a bit like printf in many other languages. If you can't wait until the chapter about I/O, you can read the online documentation to know more.

Pattern matching in functions can get more complex and powerful than that. As you may or may not remember from a few chapters ago, we can pattern match on lists to get the heads and tails. Let's do this! Start a new module called functions in which we'll write a bunch of functions to explore many pattern matching avenues available to us:

-module(functions).

-compile(export_all). %% replace with -export() later, for God's sake!

The first function we'll write is head/1, acting exactly like erlang:hd/1 which takes a list as an argument and returns its first element. It'll be done with the help of the cons operator (|):

head([H|_]) -> H.

If you type function:head([1,2,3,4]). in the shell (once the module is compiled), you can expect the value '1' to be given back to you. Consequently, to get the second element of a list you would create the function:

second([_,X|_]) -> X.

The list will just be deconstructed by Erlang in order to be pattern matched. Try it in the shell!

1> c(functions).

{ok, functions}

2> functions:head([1,2,3,4]).

1

3> functions:second([1,2,3,4]).

2

This could be repeated for lists as long as you want, although it would be impractical to do it up to thousands of values. This can be fixed by writing recursive functions, which we'll see how to do later on. For now, let's concentrate on more pattern matching. The concept of free and bound variables we discussed in Starting out (for real) still holds true for functions: we can then compare and know if two parameters passed to a function are the same or not. For this, we'll create a function same/2 that takes two arguments and tells if they're identical:

same(X,X) ->

true;

same(_,_) ->

false.

And it's that simple. Before explaining how the function works, we'll go over the concept of bound and unbound variables again, just in case:

Here, the bridegroom is sad because in Erlang, variables can never change value: no freedom! Joking aside, unbound variables are variables without any values attached to them (like our little bum on the right). Binding a variable is simply attaching a value to an unbound variable. In the case of Erlang, when you want to assign a value to a variable that is already bound, an error occurs unless the new value is the same as the old one. Let's imagine our guy on the left has married one of two twins: if the second twin comes around, he won't differentiate them and will act normally. If a different woman comes around, he'll complain. You can go back to the subchapter about Invariable Variables if this concept is not clear to you.

Back to our code: what happens when you call same(a,a) is that the first X is seen as unbound: it automatically takes the value a. Then when Erlang goes over to the second argument, it sees X is already bound. It then compares it to the a passed as the second argument and looks to see if it matches. The pattern matching succeeds and the function returns true. If the two values aren't the same, this will fail and go to the second function clause, which doesn't care about its arguments (when you're the last to choose, you can't be picky!) and will instead return false. Note that this function can effectively take any kind of argument whatsoever! It works for any type of data, not just lists or single variables. As a rather advanced example, the following function prints a date, but only if it is formatted correctly:

valid_time({Date = {Y,M,D}, Time = {H,Min,S}}) ->

io:format("The Date tuple (~p) says today is: ~p/~p/~p,~n",[Date,Y,M,D]),

io:format("The time tuple (~p) indicates: ~p:~p:~p.~n", [Time,H,Min,S]);

valid_time(_) ->

io:format("Stop feeding me wrong data!~n").

Note that it is possible to use the = operator in the function head, allowing us to match both the content inside a tuple ({Y,M,D}) and the tuple as a whole (Date). The function can be tested the following way:

4> c(functions).

{ok, functions}

5> functions:valid_time({{2011,09,06},{09,04,43}}).

The Date tuple ({2011,9,6}) says today is: 2011/9/6,

The time tuple ({9,4,43}) indicates: 9:4:43.

ok

6> functions:valid_time({{2011,09,06},{09,04}}).

Stop feeding me wrong data!

ok

There is a problem though! This function could take anything for values, even text or atoms, as long as the tuples are of the form {{A,B,C}{D,E,F}}. This denotes one of the limits of pattern matching: it can either specify really precise values such as a known number of atom, or abstract values such as the head|tail of a list, a tuple of N elements, or anything (_ and unbound variables), etc. To solve this problem, we use guards.

Guards, Guards!

Guards are additional clauses that can go in a function's head to make pattern matching more expressive. As mentioned above, pattern matching is somewhat limited as it cannot express things like a range of value or certain types of data. A concept we couldn't represent is counting: is this 12 years old basketball player too short to play with the pros? Is this distance too long to walk on your hands? Are you too old or too young to drive a car? You couldn't answer these with simple pattern matching. I mean, you could represent the driving question such as:

old_enough(0) -> false;

old_enough(1) -> false;

old_enough(2) -> false;

...

old_enough(14) -> false;

old_enough(15) -> false;

old_enough(_) -> true.

But it would be incredibly impractical. You can do it if you want, but you'll be alone to work on your code forever. If you want to eventually make friends, start a new guards module so we can type in the "correct" solution to the driving question:

old_enough(X) when X >= 16 -> true;

old_enough(_) -> false.

And you're done! As you can see, this is much shorter and cleaner. Note that a basic rule for guard expression is they must return true to succeed. The guard will fail if it returns false or if it throws an exception. Suppose we now forbid people who are over 104 years old to drive. Our valid ages for drivers is now from 16 years old up to 104 years old. We need to take care of that, but how? Let's just add a second guard clause:

right_age(X) when X >= 16, X =< 104 ->

true;

right_age(_) ->

false.

The comma (,) acts in a similar manner to the operator andalso and the semicolon (;) acts a bit like orelse (described in "Starting Out (for real)"). Both guard expressions need to succeed for the whole guard to pass. We could also represent the function the opposite way:

wrong_age(X) when X < 16; X > 104 ->

true;

wrong_age(_) ->

false.

And we get correct results from that too. Test it if you want (you should always test stuff!). In guard expressions, the semi-colon (;) acts like the orelse operator: if the first guard fails, it then tries the second, and then the next one, until either one guard succeeds or they all fail.

You can use a few more functions than comparisons and boolean evaluation in functions, including math operations (A*B/C >= 0) and functions about data types, such as is_integer/1, is_atom/1, etc. (We'll get back on them in the following chapter). One negative point about guards is that they will not accept user-defined functions because of side effects. Erlang is not a purely functional programming language (like Haskell is) because it relies on side effects a lot: you can do I/O, send messages between actors or throw errors as you want and when you want. There is no trivial way to determine if a function you would use in a guard would or wouldn't print text or catch important errors every time it is tested over many function clauses. So instead, Erlang just doesn't trust you (and it may be right to do so!)

That being said, you should be good enough to understand the basic syntax of guards to understand them when you encounter them.

Note: I've compared , and ; in guards to the operators andalso and orelse. They're not exactly the same, though. The former pair will catch exceptions as they happen while the later won't. What this means is that if there is an error thrown in the first part of the guard X >= N; N >= 0, the second part can still be evaluated and the guard might succeed; if an error was thrown in the first part of X >= N orelse N >= 0, the second part will also be skipped and the whole guard will fail.

However (there is always a 'however'), only andalso and orelse can be nested inside guards. This means (A orelse B) andalso C is a valid guard, while (A; B), C is not. Given their different use, the best strategy is often to mix them as necessary.

What the If!?

Ifs act like guards and share guards' syntax, but outside of a function clause's head. In fact, the if clauses are called Guard Patterns. Erlang's ifs are different from the ifs you'll ever encounter in most other languages; compared to them they're weird creatures that might have been more accepted had they had a different name. When entering Erlang's country, you should leave all you know about ifs at the door. Take a seat because we're going for a ride.

To see how similar to guards the if expression is, look at the following examples:

-module(what_the_if).

-export([heh_fine/0]).

heh_fine() ->

if 1 =:= 1 ->

works

end,

if 1 =:= 2; 1 =:= 1 ->

works

end,

if 1 =:= 2, 1 =:= 1 ->

fails

end.

Save this as what_the_if.erl and let's try it:

1> c(what_the_if).

./what_the_if.erl:12: Warning: no clause will ever match

./what_the_if.erl:12: Warning: the guard for this clause evaluates to 'false'

{ok,what_the_if}

2> what_the_if:heh_fine().

** exception error: no true branch found when evaluating an if expression

in function  what_the_if:heh_fine/0

Uh oh! the compiler is warning us that no clause from the if on line 12 (1 =:= 2, 1 =:= 1) will never match because its only guard evaluates to false. Remember, in Erlang, everything has to return something, and if expressions are no exception to the rule. As such, when Erlang can't find a way to have a guard succeed, it will crash: it cannot not return something. As such, we need to add a catch-all branch that will always succeed no matter what. In most languages, this would be called an 'else'. In Erlang, we use 'true' (this explains why the VM has thrown "no true branch found" when it got mad):

oh_god(N) ->

if N =:= 2 -> might_succeed;

true -> always_does  %% this is Erlang's if's 'else!'

end.

And now if we test this new function (the old one will keep spitting warnings, ignore them or take them as a reminder of what not to do):

3> c(what_the_if).

./what_the_if.erl:12: Warning: no clause will ever match

./what_the_if.erl:12: Warning: the guard for this clause evaluates to 'false'

{ok,what_the_if}

4> what_the_if:oh_god(2).

might_succeed

5> what_the_if:oh_god(3).

always_does

Here's another function showing how to use many guards in an if expression. The function also illustrates how any expression must return something: Talk has the result of the if expression bound to it, and is then concatenated in a string, inside a tuple. When reading the code, it's easy to see how the lack of a true branch would mess things up, considering Erlang has no such thing as a null value (ie.: lisp's nil, C's NULL, Python's None, etc):

%% note, this one would be better as a pattern match in function heads!

%% I'm doing it this way for the sake of the example.

help_me(Animal) ->

Talk = if Animal == cat  -> "meow";

Animal == beef -> "mooo";

Animal == dog  -> "bark";

Animal == tree -> "bark";

true -> "fgdadfgna"

end,

{Animal, "says " ++ Talk ++ "!"}.

And now we try it:

6> c(what_the_if).

./what_the_if.erl:12: Warning: no clause will ever match

./what_the_if.erl:12: Warning: the guard for this clause evaluates to 'false'

{ok,what_the_if}

7> what_the_if:help_me(dog).

{dog,"says bark!"}

8> what_the_if:help_me("it hurts!").

{"it hurts!","says fgdadfgna!"}

You might be one of the many Erlang programmers wondering why 'true' was taken over 'else' as an atom to control flow; after all, it's much more familiar. Richard O'Keefe gave the following answer on the Erlang mailing lists. I'm quoting it directly because I couldn't have put it better:

It may be more FAMILIAR, but that doesn't mean 'else' is a good thing. I know that writing '; true ->' is a very easy way to get 'else' in Erlang, but we have a couple of decades of psychology-of-programming results to show that it's a bad idea. I have started to replace:

by

if X > Y -> a() if X > Y  -> a()

; true  -> b()  ; X =< Y -> b()

end       end

if X > Y -> a() if X > Y -> a()

; X < Y -> b()  ; X < Y -> b()

; true  -> c()  ; X ==Y -> c()

end end

which I find mildly annoying when _writing_ the code but enormously helpful when _reading_ it.

'Else' or 'true' branches should be "avoided" altogether: ifs are usually easier to read when you cover all logical ends rather than relying on a "catch all" clause.

As mentioned before, there are only a limited set of functions that can be used in guard expressions (we'll see more of them in Types (or lack thereof)). This is where the real conditional powers of Erlang must be conjured. I present to you: the case expression!

Note: All this horror expressed by the function names in what_the_if.erl is expressed in regards to the if language construct when seen from the perspective of any other languages' if. In Erlang's context, it turns out to be a perfectly logical construct with a confusing name.

In Case ... of

If the if expression is like a guard, a case ... of expression is like the whole function head: you can have the complex pattern matching you can use with each argument, and you can have guards on top of it!

As you're probably getting pretty familiar with the syntax, we won't need too many examples. For this one, we'll write the append function for sets (a collection of unique values) that we will represent as an unordered list. This is possibly the worst implementation possible in terms of efficiency, but what we want here is the syntax:

prepend(X,[]) ->

[X];

prepend(X,Set) ->

case lists:member(X,Set) of

true  -> Set;

false -> [X|Set]

end.

If we send in an empty set (list) and a term X to be added, it returns us a list containing only X. Otherwise, the function lists:member/2 checks whether an element is part of a list and returns true if it is, false if it is not. In the case we already had the element X in the set, we do not need to modify the list. Otherwise, we add X as the list's first element.

In this case, the pattern matching was really simple. It can get more complex (you can compare your code with mine):

beach(Temperature) ->

case Temperature of

{celsius, N} when N >= 20, N =< 45 ->

'favorable';

{kelvin, N} when N >= 293, N =< 318 ->

'scientifically favorable';

{fahrenheit, N} when N >= 68, N =< 113 ->

'favorable in the US';

_ ->

'avoid beach'

end.

Here, the answer of "is it the right time to go to the beach" is given in 3 different temperature systems: Celsius, Kelvins and Fahrenheit degrees. Pattern matching and guards are combined in order to return an answer satisfying all uses. As pointed out earlier, case ... of expressions are pretty much the same thing as a bunch of function heads with guards. In fact we could have written our code the following way:

beachf({celsius, N}) when N >= 20, N =< 45 ->

'favorable';

...

beachf(_) ->

'avoid beach'.

This raises the question: when should we use if, case ... of or functions to do conditional expressions?

Which to use?

Which to use is rather hard to answer. The difference between function calls and case ... of are very minimal: in fact, they are represented the same way at a lower level, and using one or the other effectively has the same cost in terms of performance. One difference between both is when more than one argument needs to be evaluated: function(A,B) -> ... end. can have guards and values to match against A and B, but a case expression would need to be formulated a bit like:

case {A,B} of

Pattern Guards -> ...

end.

This form is rarely seen and might surprise the reader a bit. In similar situations, using a function call might be more appropriate. On the other hand the prepend/2 function we had written earlier is arguably cleaner the way it is rather than having an immediate function call to track down on a simple true or false clause.

Then the other question is why would you ever use if, given cases and functions are flexible enough to even encompass if through guards? The rationale behind if is quite simple: it was added to the language as a short way to have guards without needing to write the whole pattern matching part when it wasn't needed.

Of course, all of this is more about personal preferences and what you may encounter more often. There is no good solid answer. The whole topic is still debated by the Erlang community from time to time. Nobody's going to go try to beat you up because of what you've chosen, as long as it is easy to understand. As Ward Cunningham once put it, "Clean code is when you look at a routine and it's pretty much what you expected."

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Types (or lack thereof)

Dynamite-strong Typing

As you might have noticed when typing in examples from Starting Out (for real), and then modules and functions from Modules and Syntax in Functions, we never needed to write the type of a variable or the type of a function. When pattern matching, the code we had written didn't have to know what it would be matched against. The tuple {X,Y} could be matched with {atom, 123} as well as {"A string", <<"binary stuff!">>}, {2.0, ["strings","and",atoms]} or really anything at all.

When it didn't work, an error was thrown in your face, but only once you ran the code. This is because Erlang is dynamically typed: every error is caught at runtime and the compiler won't always yell at you when compiling modules where things may result in failure, like in Starting Out (for real)'s "llama + 5" example.

One classic friction point between proponents of static and dynamic typing has to do with the safety of the software being written. A frequently suggested idea is that good static type systems with compilers enforcing them with fervor will catch most errors waiting to happen before you can even execute the code. As such, statically typed languages are to be seen as safer than their dynamic counterparts. While this might be true when comparing with many dynamic languages, Erlang begs to differ and certainly has a track record to prove it. The best example is the often reported nine nines (99.9999999%) of availability offered on the Ericsson AXD 301 ATM switches, consisting of over 1 million lines of Erlang code. Please note that this is not an indication that none of the components in an Erlang-based system failed, but that a general switch system was available 99.9999999% of the time, planned outages included. This is partially because Erlang is built on the notion that a failure in one of the components should not affect the whole system. Errors coming from the programmer, hardware failures or [some] network failures are accounted for: the language includes features which will allow to distribute a program over to different nodes, handle unexpected errors, and never stop running.

To make it short, while most languages and type systems aim to make a program error-free, Erlang uses a strategy where it is assumed that errors will happen anyway and makes sure to cover these cases: Erlang's dynamic type system is not a barrier to reliability and safety of programs. This sounds like a lot of prophetic talking, but you'll see how it's done in the later chapters.

Note: Dynamic typing was historically chosen for simple reasons; those who implemented Erlang at first mostly came from dynamically typed languages, and as such, having Erlang dynamic was the most natural option to them.

Erlang is also strongly typed. A weakly typed language would do implicit type conversions between terms. If Erlang were to be weakly typed we could possibly do the operation 6 = 5 + "1". while in practice, an exception for bad arguments will be thrown:

1> 6 + "1".

** exception error: bad argument in an arithmetic expression

in operator  +/2

called as 6 + "1"

Of course, there are times when you could want to convert one kind of data to another one: changing regular strings into bit strings to store them or an integer to a floating point number. The Erlang standard library provides a number of functions to do it.

Type conversions

Erlang, like many languages, changes the type of a term by casting it into another one. This is done with the help of built-in functions, as many of the conversions could not be implemented in Erlang itself. Each of these functions take the form <type>_to_<type> and are implemented in the erlang module. Here are a few of them:

1> erlang:list_to_integer("54").

54

2> erlang:integer_to_list(54).

"54"

3> erlang:list_to_integer("54.32").

** exception error: bad argument

in function  list_to_integer/1

called as list_to_integer("54.32")

4> erlang:list_to_float("54.32").

54.32

5> erlang:atom_to_list(true).

"true"

6> erlang:list_to_bitstring("hi there").

<<"hi there">>

7> erlang:bitstring_to_list(<<"hi there">>).

"hi there"

And so on. We're hitting on a language wart here: because the scheme <type>_to_<type> is used, every time a new type is added to the language, a whole lot of conversion BIFs need to be added! Here's the whole list already there:

atom_to_binary/2, atom_to_list/1, binary_to_atom/2, binary_to_existing_atom/2, binary_to_list/1, bitstring_to_list/1, binary_to_term/1, float_to_list/1, fun_to_list/1, integer_to_list/1, integer_to_list/2, iolist_to_binary/1, iolist_to_atom/1, list_to_atom/1, list_to_binary/1, list_to_bitstring/1, list_to_existing_atom/1, list_to_float/1, list_to_integer/2, list_to_pid/1, list_to_tuple/1, pid_to_list/1, port_to_list/1, ref_to_list/1, term_to_binary/1, term_to_binary/2 and tuple_to_list/1.

That's a lot of conversion functions. We'll see most if not all of these types through this book, although we probably won't need all of these functions.

To Guard a Data Type

Erlang basic data types are easy to spot, visually: tuples have the curly brackets, lists the square brackets, strings are enclosed in double quotation marks, etc. Enforcing a certain data type has thus been possible with pattern matching: a function head/1 taking a list could only accept lists because otherwise, the matching ([H|_]) would have failed.

However, we've had a problem with numeric values because we couldn't specify ranges. Consequently, we used guards in functions about temperature, the age to drive, etc. We're hitting another roadblock now. How could we write a guard that ensures that patterns match against data of a single specific type, like numbers, atoms or bitstrings?

There are functions dedicated to this task. They will take a single argument and return true if the type is right, false otherwise. They are part of the few functions allowed in guard expressions and are named the type test BIFs:

is_atom/1           is_binary/1

is_bitstring/1      is_boolean/1        is_builtin/3

is_float/1          is_function/1       is_function/2

is_integer/1        is_list/1           is_number/1

is_pid/1            is_port/1           is_record/2

is_record/3         is_reference/1      is_tuple/1

They can be used like any other guard expression, wherever guard expressions are allowed. You might be wondering why there is no function just giving the type of the term being evaluated (something akin to type_of(X) -> Type). The answer is pretty simple. Erlang is about programming for the right cases: you only program for what you know will happen and what you expect. Everything else should cause errors as soon as possible. Although this might sound insane, the explanations you'll get in chapter 8 (Error management) will hopefully make things clearer. Until then, just trust me on that.

Note: type test BIFs constitute more than half of the functions allowed in guard expressions. The rest are also BIFs, but do not represent type tests. These are:

abs(Number), bit_size(Bitstring), byte_size(Bitstring), element(N, Tuple), float(Term), hd(List), length(List), node(), node(Pid|Ref|Port), round(Number), self(), size(Tuple|Bitstring), tl(List), trunc(Number), tuple_size(Tuple).

The functions node/1 and self/0 are related to distributed Erlang and processes/actors. We'll eventually use them, but we've still got other topics to cover before then.

It may seem like Erlang data structures are relatively limited, but lists and tuples are usually enough to build other complex structures without worrying about anything. As an example the basic node of a binary tree could be represented as {node, Value, Left, Right}, where Left and Right are either similar nodes or empty tuples. I could also represent myself as:

{person, {name, <<"Fred T-H">>},

{qualities, ["handsome", "smart", "honest", "objective"]},

{faults, ["liar"]},

{skills, ["programming", "bass guitar", "underwater breakdancing"]}}.

Which shows that by nesting tuples and list and filling them with data, we can obtain complex data structures and build functions to operate on them.

Update:

The release R13B04 saw the addition of the BIF binary_to_term/2, which lets you unserialize data the same way binary_to_term/1 would, except the second argument is an option list. If you pass in [safe], the binary won't be decoded if it contains unknown atoms or anonymous functions, which could exhaust memory.

For Type Junkies

This section is meant to be read by programmers who can not live without a static type system for one reason or another. It will include a little bit more advanced theory and everything may not be understood by everyone. I will briefly describe tools used to do static type analysis in Erlang, defining custom types and getting more safety that way. These tools will be described for anyone to understand much later in the book, given that it is not necessary to use any of them to write reliable Erlang programs. Because we'll show them later, I'll give very little details about installing, running them, etc. Again, this section is for those who really can't live without advanced type systems.

Through the years, there were some attempts to build type systems on top of Erlang. One such attempt happened back in 1997, conducted by Simon Marlow, one of the lead developers of the Glasgow Haskell Compiler, and Philip Wadler, who worked on Haskell's design and has contributed to the theory behind monads (Read the paper on said type system). Joe Armstrong later commented on the paper:

One day Phil phoned me up and announced that a) Erlang needed a type system, b) he had written a small prototype of a type system and c) he had a one year’s sabbatical and was going to write a type system for Erlang and “were we interested?” Answer —“Yes.”

Phil Wadler and Simon Marlow worked on a type system for over a year and the results were published in [20]. The results of the project were somewhat disappointing. To start with, only a subset of the language was type-checkable, the major omission being the lack of process types and of type checking inter-process messages.

Processes and messages both being one of the core features of Erlang, it may explain why the system was never added to the language. Other attempts at typing Erlang failed. The efforts of the HiPE project (attempts to make Erlang's performances much better) produced Dialyzer, a static analysis tool still in use today, with its very own type inference mechanism.

The type system that came out of it is based on success typings, a concept different from Hindley-Milner or soft-typing type systems. Success types are simple in concept: the type-inference will not try to find the exact type of every expression, but it will guarantee that the types it infers are right, and that the type errors it finds are really errors.

The best example would come from the implementation of the function and, which will usually take two Boolean values and return 'true' if they're both true, 'false' otherwise. In Haskell's type system, this would be written and :: bool -> bool -> bool. If the and function had to be implemented in Erlang, it could be done the following way:

and(false, _) -> false;

and(_, false) -> false;

and(true,true) -> true.

Under success typing, the infered type of the function would be and(_,_) -> bool(), where _ means 'anything'. The reason for this is simple: when running an Erlang program and calling this function with the arguments false and 42, the result would still be 'false'. The use of the _ wildcard in pattern matching made it that in practice, any argument can be passed as long as one of them is 'false' for the function to work. ML types would have thrown a fit (and its users had a heart attack) if you had called the function this way. Not Erlang. It might make more sense to you if you decide to read the paper on the implementation of success types, which explains the rationale behind the behavior. I really encourage any type junkies out there to read it, it's an interesting and practical implementation definition.

The details about type definitions and function annotations are described in the Erlang Enhancement Proposal 8 (EEP 8). If you're interested in using success typings in Erlang, check out the TypEr application and Dialyzer, both part of the standard distribution. To use them, type in $ typer --help and $ dialyzer --help (typer.exe --help and dialyzer.exe --help for Windows, if they're accessible from the directory you are currently in).

TypEr will be used to generate type annotations for functions. Used on this small FIFO implementation, it spits the following type annotations:

%% File: fifo.erl

%% --------------

-spec new() -> {'fifo',[],[]}.

-spec push({'fifo',_,_},_) -> {'fifo',nonempty_maybe_improper_list(),_}.

-spec pop({'fifo',_,maybe_improper_list()}) -> {_,{'fifo',_,_}}.

-spec empty({'fifo',_,_}) -> bool().

Which is pretty much right. Improper lists should be avoided because lists:reverse/1 doesn't support them, but someone bypassing the module's interface would be able to get through it and submit one. In this case, the functions push/2 and pop/2 might still succeed for a few calls before they cause an exception. This either tells us to add guards or refine our type definitions manually. Suppose we add the signature -spec push({fifo,list(),list()}_) -> {fifo,nonempty_list(),list()}. and a function that passes an improper list to push/2 to the module: when scanning it in Dialyzer (which checks and matches the types), the error message "The call fifo:push({fifo,[1|2],[]},3) breaks the contract '<Type definition here>' is output.

Dialyzer will complain only when code will break other code, and if it does, it'll usually be right (it will complain about more stuff too, like clauses that will never match or general discrepancies). Polymorphic data types are also possible to write and analyze with Dialyzer: the hd() function could be annotated with -spec([A]) -> A. and be analyzed correctly, although Erlang programmers seem to rarely use this type syntax.

Don't drink too much Kool-Aid:

Some of the things you can't expect Dialyzer and TypEr to do is type classes with constructors, first order types and recursive types. The types of Erlang are only annotations without effects or restrictions on actual compiling unless you enforce them yourself. The type checker will never tell you a program that can run right now (or has run for two years) has a type bug when it effectively causes no error when running (although you could have buggy code running correctly...)

While recursive types are something that would be really interesting to have, they're unlikely to ever appear in the current forms of TypEr and Dialyzer (the paper above explains why). Defining your own types to simulate recursive types by adding one or two levels manually is the best you can do at the moment.

It's certainly not a full-blown type system, not as strict or powerful as what languages like Scala, Haskell or Ocaml propose. Its warning and error messages are also usually a bit cryptic and not really user friendly. However, it's still a very good compromise if you really can't live in a dynamic world or wish for additional safety; just expect it to be a tool in your arsenal, not too much more.

Update:

Since version R13B04, recursive types are now available as an experimental feature for Dialyzer. This makes the previous Don't drink too much Kool-aid partially wrong. Shame on me.

Note that the type documentation has also become official (although it remains subject to change) and is more complete than what can be found in EEP8.

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Recursion

Hello recursion!

Some readers accustomed with imperative and object-oriented programming languages might be wondering why loops weren't shown already. The answer to this is "what is a loop?" Truth is, functional programming languages usually do not offer looping constructs like for and while. Instead, functional programmers rely on a silly concept named recursion.

I suppose you remember how invariable variables were explained in the intro chapter. If you don't, you can give them more attention! Recursion can also be explained with the help of mathematical concepts and functions. A basic mathematical function such as the factorial of a value is a good example of a function that can be expressed recursively. The factorial of a number n is the product of the sequence 1 x 2 x 3 x ... x n, or alternatively n x n-1 x n-2 x ... x 1. To give some examples, the factorial of 3 is 3! = 3 x 2 x 1 = 6. The factorial of 4 would be 4! = 4 x 3 x 2 x 1 = 24. Such a function can be expressed the following way in mathematical notation:

What this tells us is that if the value of n we have is 0, we return the result 1. For any value above 0, we return the factorial of n-1, which unfolds until it reaches 1:

4! = 4 x 3!

4! = 4 x 3 x 2!

4! = 4 x 3 x 2 x 1!

4! = 4 x 3 x 2 x 1 x 1

How can such a function be translated from mathematical notation to Erlang? The conversion is simple enough. Take a look at the parts of the notation: n!, 1 and n((n-1)!) and then the ifs. What we've got here is a function name (n!), guards (the ifs) and a the function body (1 and n((n-1)!)). We'll rename n! to fac(N) to restrict our syntax a bit and then we get the following:

-module(recursion).

-export([fac/1]).

fac(N) when N == 0 -> 1;

fac(N) when N > 0  -> N*fac(N-1).

And this factorial function is now done! It's pretty similar to the mathematical definition, really. With the help of pattern matching, we can shorten the definition a bit:

fac(0) -> 1;

fac(N) when N > 0 -> N*fac(N-1).

So that's quick and easy for some mathematical definitions which are recursive in nature. We looped! A definition of recursion could be made short by saying "a function that calls itself." However, we need to have a stopping condition (the real term is base case), because we'd otherwise loop indefinitely. In our case, the stopping condition is when n is equal to 0. At that point we no longer tell our function to call itself and it stops its execution right there.

Length

Let's try to make it slightly more practical. We'll implement a function to count how many elements a list contains. So we know from the beginning that we will need:

- a base case;

- a function that calls itself;

- a list to try our function on.

With most recursive functions, I find the base case easier to write first: what's the simplest input we can have to find a length from? Surely an empty list is the simplest one, with a length of 0. So let's make a mental note that [] = 0 when dealing with lengths. Then the next simplest list has a length of 1: [_] = 1. This sounds like enough to get going with our definition. We can write this down:

len([]) -> 0;

len([_]) -> 1.

Awesome! We can calculate the length of lists, given the length is either 0 or 1! Very useful indeed. Well of course it's useless, because it's not yet recursive, which brings us to the hardest part: extending our function so it calls itself for lists longer than 1 or 0. It was mentioned earlier that lists are defined recursively as [1 | [2| ... [n | []]]]. This means we can use the [H|T] pattern to match against lists of one or more elements, as a list of length one will be defined as [X|[]] and a list of length two will be defined as [X|[Y|[]]]. Note that the second element is a list itself. This means we only need to count the first one and the function can call itself on the second element. Given each value in a list counts as a length of 1, the function can be rewritten the following way:

len([]) -> 0;

len([_|T]) -> 1 + len(T).

And now you've got your own recursive function to calculate the length of a list. To see how len/1 would behave when ran, let's try it on a given list, say [1,2,3,4]:

len([1,2,3,4]) = len([1 | [2,3,4])

= 1 + len([2 | [3,4]])

= 1 + 1 + len([3 | [4]])

= 1 + 1 + 1 + len([4 | []])

= 1 + 1 + 1 + 1 + len([])

= 1 + 1 + 1 + 1 + 0

= 1 + 1 + 1 + 1

= 1 + 1 + 2

= 1 + 3

= 4

Which is the right answer. Congratulations on your first useful recursive function in Erlang!

Length of a Tail Recursion

You might have noticed that for a list of 4 terms, we expanded our function call to a single chain of 5 additions. While this does the job fine for short lists, it can become problematic if your list has a few million values in it. You don't want to keep millions of numbers in memory for such a simple calculation. It's wasteful and there's a better way. Enter tail recursion.

Tail recursion is a way to transform the above linear process (it grows as much as there are elements) to an iterative one (there is not really any growth). To have a function call being tail recursive, it needs to be 'alone'. Let me explain: what made our previous calls grow is how the answer of the first part depended on evaluating the second part. The answer to 1 + len(Rest) needs the answer of len(Rest) to be found. The function len(Rest) itself then needed the result of another function call to be found. The additions would get stacked until the last one is found, and only then would the final result be calculated. Tail recursion aims to eliminate this stacking of operation by reducing them as they happen.

In order to achieve this, we will need to hold an extra temporary variable as a parameter in our function. I'll illustrate the concept with the help of the factorial function, but this time defining it to be tail recursive. The aforementioned temporary variable is sometimes called accumulator and acts as a place to store the results of our computations as they happen in order to limit the growth of our calls:

tail_fac(N) -> tail_fac(N,1).

tail_fac(0,Acc) -> Acc;

tail_fac(N,Acc) when N > 0 -> tail_fac(N-1,N*Acc).

Here, I define both tail_fac/1 and tail_fac/2. The reason for this is that Erlang doesn't allow default arguments in functions (different arity means different function) so we do that manually. In this specific case, tail_fac/1 acts like an abstraction over the tail recursive tail_fac/2 function. The details about the hidden accumulator of tail_fac/2 don't interest anyone, so we would only export tail_fac/1 from our module. When running this function, we can expand it to:

tail_fac(4)    = tail_fac(4,1)

tail_fac(4,1)  = tail_fac(4-1, 4*1)

tail_fac(3,4)  = tail_fac(3-1, 3*4)

tail_fac(2,12) = tail_fac(2-1, 2*12)

tail_fac(1,24) = tail_fac(1-1, 1*24)

tail_fac(0,24) = 24

See the difference? Now we never need to hold more than two terms in memory: the space usage is constant. It will take as much space to calculate the factorial of 4 as it will take space to calculate the factorial of 1 million (if we forget 4! is a smaller number than 1M! in its complete representation, that is).

With an example of tail recursive factorials under your belt, you might be able to see how this pattern could be applied to our len/1 function. What we need is to make our recursive call 'alone'. If you like visual examples, just imagine you're going to put the +1 part inside the function call by adding a parameter:

len([]) -> 0;

len([_|T]) -> 1 + len(T).

becomes:

tail_len(L) -> tail_len(L,0).

tail_len([], Acc) -> Acc;

tail_len([_|T], Acc) -> tail_len(T,Acc+1).

And now your length function is tail recursive.

More recursive functions

We'll write a few more recursive functions, just to get in the habit a bit more. After all, recursion being the only looping construct that exists in Erlang (except list comprehensions), it's one of the most important concepts to understand. It's also useful in every other functional programming language you'll try afterwards, so take notes!

The first function we'll write will be duplicate/2. This function takes an integer as its first parameter and then any other term as its second parameter. It will then create a list of as many copies of the term as specified by the integer. Like before, thinking of the base case first is what might help you get going. For duplicate/2, asking to repeat something 0 time is the most basic thing that can be done. All we have to do is return an empty list, no matter what the term is. Every other case needs to try and get to the base case by calling the function itself. We will also forbid negative values for the integer, because you can't duplicate something -n times:

duplicate(0,_) ->

[];

duplicate(N,Term) when N > 0 ->

[Term|duplicate(N-1,Term)].

Once the basic recursive function is found, it becomes easier to transform it into a tail recursive one by moving the list construction into a temporary variable:

tail_duplicate(N,Term) ->

tail_duplicate(N,Term,[]).

tail_duplicate(0,_,List) ->

List;

tail_duplicate(N,Term,List) when N > 0 ->

tail_duplicate(N-1, Term, [Term|List]).

Success! I want to change the subject a little bit here by drawing a parallel between tail recursion and a while loop. Our tail_duplicate/2 function has all the usual parts of a while loop. If we were to imagine a while loop in a fictional language with Erlang-like syntax, our function could look a bit like this:

function(N, Term) ->

while N > 0 ->

List = [Term|List],

N = N-1

end,

List.

Note that all the elements are there in both the fictional language and in Erlang. Only their position changes. This demonstrates that a proper tail recursive function is similar to an iterative process, like a while loop.

There's also an interesting property that we can 'discover' when we compare recursive and tail recursive functions by writing a reverse/1 function, which will reverse a list of terms. For such a function, the base case is an empty list, for which we have nothing to reverse. We can just return an empty list when that happens. Every other possibility should try to converge to the base case by calling itself, like with duplicate/2. Our function is going to iterate through the list by pattern matching [H|T] and then putting H after the rest of the list:

reverse([]) -> [];

reverse([H|T]) -> reverse(T)++[H].

On long lists, this will be a true nightmare: not only will we stack up all our append operations, but we will need to traverse the whole list for every single of these appends until the last one! For visual readers, the many checks can be represented as:

reverse([1,2,3,4]) = [4]++[3]++[2]++[1]

↑    ↵

= [4,3]++[2]++[1]

↑ ↑    ↵

= [4,3,2]++[1]

↑ ↑ ↑    ↵

= [4,3,2,1]

This is where tail recursion comes to the rescue. Because we will use an accumulator and will add a new head to it every time, our list will automatically be reversed. Let's first see the implementation:

tail_reverse(L) -> tail_reverse(L,[]).

tail_reverse([],Acc) -> Acc;

tail_reverse([H|T],Acc) -> tail_reverse(T, [H|Acc]).

If we represent this one in a similar manner as the normal version, we get:

tail_reverse([1,2,3,4]) = tail_reverse([2,3,4], [1])

= tail_reverse([3,4], [2,1])

= tail_reverse([4], [3,2,1])

= tail_reverse([], [4,3,2,1])

= [4,3,2,1]

Which shows that the number of elements visited to reverse our list is now linear: not only do we avoid growing the stack, we also do our operations in a much more efficient manner!

Another function to implement could be sublist/2, which takes a list L and an integer N, and returns the N first elements of the list. As an example, sublist([1,2,3,4,5,6],3) would return [1,2,3]. Again, the base case is trying to obtain 0 elements from a list. Take care however, because sublist/2 is a bit different. You've got a second base case when the list passed is empty! If we do not check for empty lists, an error would be thrown when calling recursion:sublist([1],2). while we want [1] instead. Once this is defined, the recursive part of the function only has to cycle through the list, keeping elements as it goes, until it hits one of the base cases:

sublist(_,0) -> [];

sublist([],_) -> [];

sublist([H|T],N) when N > 0 -> [H|sublist(T,N-1)].

Which can then be transformed to a tail recursive form in the same manner as before:

tail_sublist(L, N) -> tail_sublist(L, N, []).

tail_sublist(_, 0, SubList) -> SubList;

tail_sublist([], _, SubList) -> SubList;

tail_sublist([H|T], N, SubList) ->

tail_sublist(T, N-1, [H|SubList]).

There's a flaw in this function. A fatal flaw! We use a list as an accumulator in exactly the same manner we did to reverse our list. If you compile this function as is, sublist([1,2,3,4,5,6],3) would not return [1,2,3], but [3,2,1]. The only thing we can do is take the final result and reverse it ourselves. Just change the tail_sublist/2 call and leave all our recursive logic intact:

tail_sublist(L, N) -> reverse(tail_sublist(L, N, [])).

The final result will be ordered correctly. It might seem like reversing our list after a tail recursive call is a waste of time and you would be partially right (we still save memory doing this). On shorter lists, you might find your code is running faster with normal recursive calls than with tail recursive calls for this reason, but as your data sets grow, reversing the list will be comparatively lighter.

Note: instead of writing your own reverse/1 function, you should use lists:reverse/1. It's been used so much for tail recursive calls that the maintainers and developers of Erlang decided to turn it into a BIF. Your lists can now benefit from extremely fast reversal (thanks to functions written in C) which will make the reversal disadvantage a lot less obvious. The rest of the code in this chapter will make use of our own reversal function, but after that you should not use it ever again.

To push things a bit further, we'll write a zipping function. A zipping function will take two lists of same length as parameters and will join them as a list of tuples which all hold two terms. Our own zip/2 function will behave this way:

1> recursive:zip([a,b,c],[1,2,3]).

[{a,1},{b,2},{c,3}]

Given we want our parameters to both have the same length, the base case will be zipping two empty lists:

zip([],[]) -> [];

zip([X|Xs],[Y|Ys]) -> [{X,Y}|zip(Xs,Ys)].

However, if you wanted a more lenient zip function, you could decide to have it finish whenever one of the two list is done. In this scenario, you therefore have two base cases:

lenient_zip([],_) -> [];

lenient_zip(_,[]) -> [];

lenient_zip([X|Xs],[Y|Ys]) -> [{X,Y}|lenient_zip(Xs,Ys)].

Notice that no matter what our base cases are, the recursive part of the function remains the same. I would suggest you try and make your own tail recursive versions of zip/2 and lenient_zip/2, just to make sure you fully understand how to make tail recursive functions: they'll be one of the central concepts of larger applications where our main loops will be made that way.

If you want to check your answers, take a look at my implementation of recursive.erl, more precisely the tail_zip/2 and tail_lenient_zip/3 functions.

Note: tail recursion as seen here is not making the memory grow because when the virtual machine sees a function calling itself in a tail position (the last expression to be evaluated in a function), it eliminates the current stack frame. This is called tail-call optimisation (TCO) and it is a special case of a more general optimisation named Last Call Optimisation (LCO).

LCO is done whenever the last expression to be evaluated in a function body is another function call. When that happens, as with TCO, the Erlang VM avoids storing the stack frame. As such tail recursion is also possible between multiple functions. As an example, the chain of functions a() -> b(). b() -> c(). c() -> a(). will effectively create an infinite loop that won't go out of memory as LCO avoids overflowing the stack. This principle, combined with our use of accumulators is what makes tail recursion useful.

Quick, Sort!

I can (and will) now assume recursion and tail recursion make sense to you, but just to make sure, I'm going to push for a more complex example, quicksort. Yes, the traditional "hey look I can write short functional code" canonical example. A naive implementation of quicksort works by taking the first element of a list, the pivot, and then putting all the elements smaller or equal to the pivot in a new list, and all those larger in another list. We then take each of these lists and do the same thing on them until each list gets smaller and smaller. This goes on until you have nothing but an empty list to sort, which will be our base case. This implementation is said to be naive because smarter versions of quicksort will try to pick optimal pivots to be faster. We don't really care about that for our example though.

We will need two functions for this one: a first function to partition the list into smaller and larger parts and a second function to apply the partition function on each of the new lists and to glue them together. First of all, we'll write the glue function:

quicksort([]) -> [];

quicksort([Pivot|Rest]) ->

{Smaller, Larger} = partition(Pivot,Rest,[],[]),

quicksort(Smaller) ++ [Pivot] ++ quicksort(Larger).

This shows the base case, a list already partitioned in larger and smaller parts by another function, the use of a pivot with both lists quicksorted appended before and after it. So this should take care of assembling lists. Now the partitioning function:

partition(_,[], Smaller, Larger) -> {Smaller, Larger};

partition(Pivot, [H|T], Smaller, Larger) ->

if H =< Pivot -> partition(Pivot, T, [H|Smaller], Larger);

H >  Pivot -> partition(Pivot, T, Smaller, [H|Larger])

end.

And you can now run your quicksort function. If you've looked for Erlang examples on the Internet before, you might have seen another implementation of quicksort, one that is simpler and easier to read, but makes use of list comprehensions. The easy to replace parts are the ones that create new lists, the partition/4 function:

lc_quicksort([]) -> [];

lc_quicksort([Pivot|Rest]) ->

lc_quicksort([Smaller || Smaller <- Rest, Smaller =< Pivot])

++ [Pivot] ++

lc_quicksort([Larger || Larger <- Rest, Larger > Pivot]).

The main differences are that this version is much easier to read, but in exchange, it has to traverse the list to partition it in two parts. This is a fight of clarity against performance, but the real loser here is you, because a function lists:sort/1 already exists. Use that one instead.

Don't drink too much Kool-Aid:

All this conciseness is good for educational purposes, but not for performance. Many functional programming tutorials never mention this! First of all, both implementations here need to process values that are equal to the pivot more than once. We could have decided to instead return 3 lists: elements smaller, larger and equal to the pivot in order to make this more efficient.

Another problem relates to how we need to traverse all the partitioned lists more than once when attaching them to the pivot. It is possible to reduce the overhead a little by doing the concatenation while partitioning the lists in three parts. If you're curious about this, look at the last function (bestest_qsort/1) of recursive.erl for an example.

A nice point about all of these quicksorts is that they will work on lists of any data type you've got, even tuples of lists and whatnot. Try them, they work!

More than lists

By reading this chapter, you might be starting to think recursion in Erlang is mainly a thing concerning lists. While lists are a good example of a data structure that can be defined recursively, there's certainly more than that. For the sake of diversity, we'll see how to build binary trees, and then read data from them.

First of all, it's important to define what a tree is. In our case, it's nodes all the way down. Nodes are tuples that contain a key, a value associated to the key, and then two other nodes. Of these two nodes, we need one that has a smaller and one that has a larger key than the node holding them. So here's recursion! A tree is a node containing nodes, each of which contains nodes, which in turn also contain nodes. This can't keep going forever (we don't have infinite data to store), so we'll say that our nodes can also contain empty nodes.

To represent nodes, tuples are an appropriate data structure. For our implementation, we can then define these tuples as {node, {Key, Value, Smaller, Larger}} (a tagged tuple!), where Smaller and Larger can be another similar node or an empty node ({node, nil}). We won't actually need a concept more complex than that.

Let's start building a module for our very basic tree implementation. The first function, empty/0, returns an empty node. The empty node is the starting point of a new tree, also called the root:

-module(tree).

-export([empty/0, insert/3, lookup/2]).

empty() -> {node, 'nil'}.

By using that function and then encapsulating all representations of nodes the same way, we hide the implementation of the tree so people don't need to know how it's built. All that information can be contained by the module alone. If you ever decide to change the representation of a node, you can then do it without breaking external code.

To add content to a tree, we must first understand how to recursively navigate through it. Let's proceed in the same way as we did for every other recursion example by trying to find the base case. Given that an empty tree is an empty node, our base case is thus logically an empty node. So whenever we'll hit an empty node, that's where we can add our new key/value. The rest of the time, our code has to go through the tree trying to find an empty node where to put content.

To find an empty node starting from the root, we must use the fact that the presence of Smaller and Larger nodes let us navigate by comparing the new key we have to insert to the current node's key. If the new key is smaller than the current node's key, we try to find the empty node inside Smaller, and if it's larger, inside Larger. There is one last case, though: what if the new key is equal to the current node's key? We have two options there: let the program fail or replace the value with the new one. This is the option we'll take here. Put into a function all this logic works the following way:

insert(Key, Val, {node, 'nil'}) ->

{node, {Key, Val, {node, 'nil'}, {node, 'nil'}}};

insert(NewKey, NewVal, {node, {Key, Val, Smaller, Larger}}) when NewKey < Key ->

{node, {Key, Val, insert(NewKey, NewVal, Smaller), Larger}};

insert(NewKey, NewVal, {node, {Key, Val, Smaller, Larger}}) when NewKey > Key ->

{node, {Key, Val, Smaller, insert(NewKey, NewVal, Larger)}};

insert(Key, Val, {node, {Key, _, Smaller, Larger}}) ->

{node, {Key, Val, Smaller, Larger}}.

Note here that the function returns a completely new tree. This is typical of functional languages having only single assignment. While this can be seen as inefficient, most of the underlying structures of two versions of a tree sometimes happen to be the same and are thus shared, copied by the VM only when needed.

What's left to do on this example tree implementation is creating a lookup/2 function that will let you find a value from a tree by giving its key. The logic needed is extremely similar to the one used to add new content to the tree: we step through the nodes, checking if the lookup key is equal, smaller or larger than the current node's key. We have two base cases: one when the node is empty (the key isn't in the tree) and one when the key is found. Because we don't want our program to crash each time we look for a key that doesn't exist, we'll return the atom 'undefined'. Otherwise, we'll return {ok, Value}. The reason for this is that if we only returned Value and the node contained the atom 'undefined', we would have no way to know if the tree did return the right value or failed to find it. By wrapping successful cases in such a tuple, we make it easy to understand which is which. Here's the implemented function:

lookup(_, {node, 'nil'}) ->

undefined;

lookup(Key, {node, {Key, Val, _, _}}) ->

{ok, Val};

lookup(Key, {node, {NodeKey, _, Smaller, _}}) when Key < NodeKey ->

lookup(Key, Smaller);

lookup(Key, {node, {_, _, _, Larger}}) ->

lookup(Key, Larger).

And we're done. Let's test it with by making a little email address book. Compile the file and start the shell:

1> T1 = tree:insert("Jim Woodland", "jim.woodland@gmail.com", tree:empty()).

{node,{"Jim Woodland","jim.woodland@gmail.com",

{node,nil},

{node,nil}}}

2> T2 = tree:insert("Mark Anderson", "i.am.a@hotmail.com", T1).

{node,{"Jim Woodland","jim.woodland@gmail.com",

{node,nil},

{node,{"Mark Anderson","i.am.a@hotmail.com",

{node,nil},

{node,nil}}}}}

3> Addresses = tree:insert("Anita Bath", "abath@someuni.edu", tree:insert("Kevin Robert", "myfairy@yahoo.com", tree:insert("Wilson Longbrow", "longwil@gmail.com", T2))).

{node,{"Jim Woodland","jim.woodland@gmail.com",

{node,{"Anita Bath","abath@someuni.edu",

{node,nil},

{node,nil}}},

{node,{"Mark Anderson","i.am.a@hotmail.com",

{node,{"Kevin Robert","myfairy@yahoo.com",

{node,nil},

{node,nil}}},

{node,{"Wilson Longbrow","longwil@gmail.com",

{node,nil},

{node,nil}}}}}}}

And now you can lookup email addresses with it:

4> tree:lookup("Anita Bath", Addresses).

{ok, "abath@someuni.edu"}

5> tree:lookup("Jacques Requin", Addresses).

undefined

That concludes our functional address book example built from a recursive data structure other than a list! Anita Bath now...

Note: Our tree implementation is very naive: we do not support common operations such as deleting nodes or rebalancing the tree to make the following lookups faster. If you're interested in implementing and/or exploring these, studying the implementation of Erlang's gb_trees module (otp_src_R<version>B<revision>/lib/stdlib/src/gb_trees.erl) is a good idea. This is also the module you should use when dealing with trees in your code, rather than reinventing your own wheel.

Thinking recursively

If you've understood everything in this chapter, thinking recursively is probably becoming more intuitive. A different aspect of recursive definitions when compared to their imperative counterparts (usually in while or for loops) is that instead of taking a step-by-step approach ("do this, then that, then this, then you're done"), our approach is more declarative ("if you get this input, do that, this otherwise"). This property is made more obvious with the help of pattern matching in function heads.

If you still haven't grasped how recursion works, maybe reading this will help you.

Joking aside, recursion coupled with pattern matching is sometimes an optimal solution to the problem of writing concise algorithms that are easy to understand. By subdividing each part of a problem into separate functions until they can no longer be simplified, the algorithm becomes nothing but assembling a bunch of correct answers coming from short routines (that's a bit similar to what we did with quicksort). This kind of mental abstraction is also possible with your everyday loops, but I believe the practice is easier with recursion. Your mileage may vary.

And now ladies and gentlemen, a discussion: the author vs. himself

- — Okay, I think I understand recursion. I get the declarative aspect of it. I get it has mathematical roots, like with invariable variables. I get that you find it easier in some cases. What else?

- — It respects a regular pattern. Find the base cases, write them down, then every other cases should try to converge to these base cases to get your answer. It makes writing functions pretty easy.

- — Yeah, I got that, you repeated it a bunch of times already. My loops can do the same.

- — Yes they can. Can't deny that!

- — Right. A thing I don't get is why you bothered writing all these non-tail recursive versions if they're not as good as tail recursive ones.

- — Oh it's simply to make things easier to grasp. Moving from regular recursion, which is prettier and easier to understand, to tail recursion, which is theoretically more efficient, sounded like a good way to show all options.

- — Right, so they're useless except for educational purposes, I get it.

- — Not exactly. In practice you'll see little difference in the performance between tail recursive and normal recursive calls. The areas to take care of are in functions that are supposed to loop indifinitely, like main loops, then those that operate on large datasets. Finally there's a type of functions that will always generate very large stacks, be slow and possibly crash early if you don't make them tail recursive. The best example of this is the Fibonacci function, which grows exponentially if it's not iterative or tail recursive. You should profile your code (I'll show how to do that at a later point, I promise), see what slows it down, and fix it.

- — But loops are always iterative and make this a non-issue.

- — Yes, but... but... my beautiful Erlang...

- — Well isn't that great? All that learning because there is no 'while' or 'for' in Erlang. Thank you very much I'm going back to programming my toaster in C!

- — Not so fast there! Functional programming languages have other assets! If we've found some base case patterns to make our life easier when writing recursive functions, a bunch of smart people have found many more to the point where you will need to write very few recursive functions yourself. If you stay around, I'll show you how such abstractions can be built. But for this we will need more power. Let me tell you about higher order functions...

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Higher Order Functions

Let's get functional

An important part of all functional programming languages is the ability to take a function you defined and then pass it as a parameter to another function. This in turn binds that function parameter to a variable which can be used like any other variable within the function. A function that can be transported around that way is named a higher order function. Higher order functions are a powerful means of abstraction and one of the best tools to master in Erlang.

Again, this a concept rooted in mathematics, mainly lambda calculus. I won't go into much detail about lambda calculus because some people have a hard time grasping it and it's a bit out of scope. However, I'll define it briefly as a system where everything is a function, even numbers. Because everything is a function, functions must accept other functions as parameters and can operate on them with even more functions!

Alright, this might be a little bit weird, so let's start with an example:

-module(hhfuns).

-compile(export_all).

one() -> 1.

two() -> 2.

add(X,Y) -> X() + Y().

Now open the Erlang shell, compile the module and get going:

1> c(hhfuns).

{ok, hhfuns}

2> hhfuns:add(one,two).

** exception error: bad function one

in function  hhfuns:add/2

3> hhfuns:add(1,2).

** exception error: bad function 1

in function  hhfuns:add/2

4> hhfuns:add(fun hhfuns:one/0, fun hhfuns:two/0).

3

Confusing? Not so much, once you know how it works (isn't that always the case?) In command 2, the atoms one and two are passed to add/2, which then uses both atoms as function names (X() + Y()). If function names are written without a parameter list then those name as interpreted as atoms, and atoms can not be functions, so the call fails. This is the reason why the expression 3 also fails: the values 1 and 2 can not be called as functions either, and functions are what we need!

This is why a new notation has to be added to the language in order to let you pass functions from outside a module. This is what fun Module:Function/Arity is: it tells the VM to use that specific function, and then bind it to a variable.

So what are the gains of using functions in that manner? Well a little example might be needed in order to understand it. We'll add a few functions to hhfuns that work recursively over a list to add or subtract one from each integer of a list:

increment([]) -> [];

increment([H|T]) -> [H+1|increment(T)].

decrement([]) -> [];

decrement([H|T]) -> [H-1|decrement(T)].

See how similar these functions are? They basically do the same thing: they cycle through a list, apply a function on each element (+ or -) and then call themselves again. There is almost nothing changing in that code: only the applied function and the recursive call are different. The core of a recursive call on a list like that is always the same. We'll abstract all the similar parts in a single function (map/2) that will take another function as an argument:

map(_, []) -> [];

map(F, [H|T]) -> [F(H)|map(F,T)].

incr(X) -> X + 1.

decr(X) -> X - 1.

Which can then be tested in the shell:

1> c(hhfuns).

{ok, hhfuns}

2> L = [1,2,3,4,5].

[1,2,3,4,5]

3> hhfuns:increment(L).

[2,3,4,5,6]

4> hhfuns:decrement(L).

[0,1,2,3,4]

5> hhfuns:map(fun hhfuns:incr/1, L).

[2,3,4,5,6]

6> hhfuns:map(fun hhfuns:decr/1, L).

[0,1,2,3,4]

Here the results are the same, but you have just created a very smart abstraction! Every time you will want to apply a function to each element of a list, you only have to call map/2 with your function as a parameter. However, it is a bit annoying to have to put every function we want to pass as a parameter to map/2 in a module, name it, export it, then compile it, etc. In fact it's plainly unpractical. What we need are functions that can be declared on the fly...

Anonymous functions

Anonymous functions, or funs, address that problem by letting you declare a special kind of function inline, without naming them. They can do pretty much everything normal functions can do, except calling themselves recursively (how could they do it if they are anonymous?) Their syntax is:

fun(Args1) ->

Expression1, Exp2, ..., ExpN;

(Args2) ->

Expression1, Exp2, ..., ExpN;

(Args3) ->

Expression1, Exp2, ..., ExpN

end

And can be used the following way:

7> Fn = fun() -> a end.

#Fun<erl_eval.20.67289768>

8> Fn().

a

9> hhfuns:map(fun(X) -> X + 1 end, L).

[2,3,4,5,6]

10> hhfuns:map(fun(X) -> X - 1 end, L).

[0,1,2,3,4]

And now you're seeing one of the things that make people like functional programming so much: the ability to make abstractions on a very low level of code. Basic concepts such as looping can thus be ignored, letting you focus on what is done rather than how to do it.

Anonymous functions are already pretty dandy for such abstractions but they still have more hidden powers:

11> PrepareAlarm = fun(Room) ->

11>                     io:format("Alarm set in ~s.~n",[Room]),

11>                     fun() -> io:format("Alarm tripped in ~s! Call Batman!~n",[Room]) end

11>                   end.

#Fun<erl_eval.20.67289768>

12> AlarmReady = PrepareAlarm("bathroom").

Alarm set in bathroom.

#Fun<erl_eval.6.13229925>

13> AlarmReady().

Alarm tripped in bathroom! Call Batman!

ok

Hold the phone Batman! What's going on here? Well, first of all, we declare an anonymous function assigned to PrepareAlarm. This function has not run yet: it only gets executed when PrepareAlarm("bathroom"). is called. At that point, the call to io:format/2 is evaluated and the "Alarm set" text is output. The second expression (another anonymous function) is returned to the caller and then assigned to AlarmReady. Note that in this function, the variable Room's value is taken from the 'parent' function (PrepareAlarm). This is related to a concept called closures.

To understand closures, one must first understand scope. A function's scope can be imaged as the place where all the variables and their values are stored. In the function base(A) -> B = A + 1., A and B are both defined to be part of base/1's scope. This means that anywhere inside base/1, you can refer to A and B and expect a value to be bound to them. And when I say 'anywhere', I ain't kidding, kid; this includes anonymous functions too:

base(A) ->

B = A + 1,

F = fun() -> A * B end,

F().

B and A are still bound to base/1's scope, so the function F can still access them. This is because F inherits base/1's scope. Like most kinds of real-life inheritance, the parents can't get what the children have:

base(A) ->

B = A + 1,

F = fun() -> C = A * B end,

F(),

C.

In this version of the function, B is still equal to A + 1 and F will still execute fine. However, the variable C is only in the scope of the anonymous function in F. When base/1 tries to access C's value on the last line, it only finds an unbound variable. In fact, had you tried to compile this function, the compiler would have thrown a fit. Inheritance only goes one way.

It is important to note that the inherited scope follows the anonymous function wherever it is, even when it is passed to another function:

a() ->

Secret = "pony",

fun() -> Secret end.

b(F) ->

"a/0's password is "++F().

Then if we compile it:

14> c(hhfuns).

{ok, hhfuns}

15> hffuns:b(hhfuns:a()).

"a/0's password is pony"

Who told a/0's password? Well, a/0 did. While the anonymous function has a/0's scope when it's declared in there, it can still carry it when executed in b/1, as explained above. This is very useful because it lets us carry around parameters and content out of its original context, where the whole context itself are not needed anymore (exactly like we did with Batman in a previous example).

You're most likely to use anonymous functions to carry state around when you have functions defined that take many arguments, but you have a constant one:

16> math:pow(5,2).

25.0

17> Base = 2.

2

18> PowerOfTwo = fun(X) -> math:pow(Base,X) end.

#Fun<erl_eval.6.13229925>

17> hhfuns:map(PowerOfTwo, [1,2,3,4]).

[2.0,4.0,8.0,16.0]

By wrapping the call to math:pow/2 inside an anonymous function with the Base variable bound in its scope, we made it possible to have each of the calls to PowerOfTwo in hhfuns:map/2 use the integers from the list as the exponents of our base.

A little trap you might fall into when writing anonymous functions is when you try to redefine the scope:

base() ->

A = 1,

(fun() -> A = 2 end)().

This will declare an anonymous function and then run it. As the anonymous function inherits base/0's scope, trying to use the = operator compares 2 with the variable A (bound to 1). This is guaranteed to fail. However it is possible to redefine the variable if it's done in the nested function's head:

base() ->

A = 1,

(fun(A) -> A = 2 end)(2).

And this works. If you try to compile it, you'll get a warning about shadowing ("Warning: variable 'A' shadowed in 'fun'"). Shadowing is the term used to describe the act of defining a new variable that has the same name as one that was in the parent scope. This is there to prevent some mistakes (usually rightly so), so you might want to consider renaming your variables in these circumstances.

We'll set the anonymous function theory aside a bit and we'll explore more common abstractions to avoid having to write more recursive functions, like I promised at the end of the previous chapter.

Maps, filters, folds and more

At the beginning of this chapter, I briefly showed how to abstract away two similar functions to get a map/2 function. I also affirmed that such a function could be used for any list where we want to act on each element. The function was the following:

map(_, []) -> [];

map(F, [H|T]) -> [F(H)|map(F,T)].

However, there are many other similar abstractions to build from commonly occurring recursive functions. Let's first take a look at these two functions:

%% only keep even numbers

even(L) -> lists:reverse(even(L,[])).

even([], Acc) -> Acc;

even([H|T], Acc) when H rem 2 == 0 ->

even(T, [H|Acc]);

even([_|T], Acc) ->

even(T, Acc).

%% only keep men older than 60

old_men(L) -> lists:reverse(old_men(L,[])).

old_men([], Acc) -> Acc;

old_men([Person = {male, Age}|People], Acc) when Age > 60 ->

old_men(People, [Person|Acc]);

old_men([_|People], Acc) ->

old_men(People, Acc).

The first one takes a list of numbers and returns only those that are even. The second one goes through a list of people of the form {Gender, Age} and only keeps those that are males over 60. The similarities are a bit harder to find here, but we've got some common points. Both functions operate on lists and have the same objective of keeping elements that succeed some test (also a predicate) and then drop the others. From this generalization we can extract all the useful information we need and abstract them away:

filter(Pred, L) -> lists:reverse(filter(Pred, L,[])).

filter(_, [], Acc) -> Acc;

filter(Pred, [H|T], Acc) ->

case Pred(H) of

true  -> filter(Pred, T, [H|Acc]);

false -> filter(Pred, T, Acc)

end.

To use the filtering function we now only need to get the test outside of the function. Compile the hhfuns module and try it:

1> c(hhfuns).

{ok, hhfuns}

2> Numbers = lists:seq(1,10).

[1,2,3,4,5,6,7,8,9,10]

3> hhfuns:filter(fun(X) -> X rem 2 == 0 end, Numbers).

[2,4,6,8,10]

4> People = [{male,45},{female,67},{male,66},{female,12},{unkown,174},{male,74}].

[{male,45},{female,67},{male,66},{female,12},{unkown,174},{male,74}]

5> hhfuns:filter(fun({Gender,Age}) -> Gender == male andalso Age > 60 end, People).

[{male,66},{male,74}]

These two examples show that with the use of the filter/2 function, the programmer only has to worry about producing the predicate and the list. The act of cycling through the list to throw out unwanted items is no longer necessary to think about. This is one important thing about abstracting functional code: try to get rid of what's always the same and let the programmer supply in the parts that change.

In the previous chapter, another kind of recursive manipulation we applied on lists was to look at every element of a list one after the other and reduce them to a single answer. This is called a fold and can be used on the following functions:

%% find the maximum of a list

max([H|T]) -> max2(T, H).

max2([], Max) -> Max;

max2([H|T], Max) when H > Max -> max2(T, H);

max2([_|T], Max) -> max2(T, Max).

%% find the minimum of a list

min([H|T]) -> min2(T,H).

min2([], Min) -> Min;

min2([H|T], Min) when H < Min -> min2(T,H);

min2([_|T], Min) -> min2(T, Min).

%% sum of all the elements of a list

sum(L) -> sum(L,0).

sum([], Sum) -> Sum;

sum([H|T], Sum) -> sum(T, H+Sum).

To find how the fold should behave, we've got to find all the common points of these actions and then what is different. As mentioned above, we always have a reduction from a list to a single value. Consequently, our fold should only consider iterating while keeping a single item, no list-building needed. Then we need to ignore the guards, because they're not always there: these need to be in the user's function. In this regard, our folding function will probably look a lot like sum.

A subtle element of all three functions that wasn't mentioned yet is that every function needs to have an initial value to start counting with. In the case of sum/2, we use 0 as we're doing addition and given X = X + 0, the value is neutral and we can't mess up the calculation by starting there. If we were doing multiplication we'd use 1 given X = X * 1. The functions min/1 and max/1 can't have a default starting value: if the list was only negative numbers and we started at 0, the answer would be wrong. As such, we need to use the first element of the list as a starting point. Sadly, we can't always decide this way, so we'll leave that decision to the programmer. By taking all these elements, we can build the following abstraction:

fold(_, Start, []) -> Start;

fold(F, Start, [H|T]) -> fold(F, F(H,Start), T).

And when tried:

6> c(hhfuns).

{ok, hhfuns}

7> [H|T] = [1,7,3,5,9,0,2,3].

[1,7,3,5,9,0,2,3]

8> hhfuns:fold(fun(A,B) when A > B -> A; (_,B) -> B end, H, T).

9

9> hhfuns:fold(fun(A,B) when A < B -> A; (_,B) -> B end, H, T).

0

10> hhfuns:fold(fun(A,B) -> A + B end, 0, lists:seq(1,6)).

21

Pretty much any function you can think of that reduces lists to 1 element can be expressed as a fold.

What's funny there is that you can represent an accumulator as a single element (or a single variable), and an accumulator can be a list. Therefore, we can use a fold to build a list. This means fold is universal in the sense that you can implement pretty much any other recursive function on lists with a fold, even map and filter:

reverse(L) ->

fold(fun(X,Acc) -> [X|Acc] end, [], L).

map2(F,L) ->

reverse(fold(fun(X,Acc) -> [F(X)|Acc] end, [], L)).

filter2(Pred, L) ->

F = fun(X,Acc) ->

case Pred(X) of

true  -> [X|Acc];

false -> Acc

end

end,

reverse(fold(F, [], L)).

And they all work the same as those written by hand before. How's that for powerful abstractions?

Map, filters and folds are only one of many abstractions over lists provided by the Erlang standard library (see lists:map/2, lists:filter/2, lists:foldl/3 and lists:foldr/3). Other functions include all/2 and any/2 which both take a predicate and test if all the elements return true or if at least one of them returns true, respectively. Then you have dropwhile/2 that will ignore elements of a list until it finds one that fit a certain predicate, its opposite, takewhile/2, that will keep all elements until there is one that doesn't return true to the predicate. A complimentary function to the two previous ones is partition/2, which will take a list and return two: one that has the terms which satisfy a given predicate, and one list for the others. Other frequently used lists functions include flatten/1, flatlength/1, flatmap/2, merge/1, nth/2, nthtail/2, split/2 and a bunch of others.

You'll also find other functions such as zippers (as seen in last chapter), unzippers, combinations of maps and folds, etc. I encourage you to read the documentation on lists to see what can be done. You'll find yourself rarely needing to write recursive functions by using what's already been abstracted away by smart people.

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Errors and Exceptions

Not so fast!

There's no right place for a chapter such as this one. I've shown enough for one to encounter many errors, but yet not enough to describe all the mechanisms of error handling (or the errors themselves). That's a bit because Erlang has two main paradigms: functional and concurrent. The functional subset is the one I've been explaining since the beginning of the book: referential transparency, recursion, higher order functions, etc. The concurrent subset is the one that makes Erlang famous: actors, thousands and thousands of concurrent processes, supervision trees, etc.

Because I judge the functional part essential to know before moving on to the concurrent part, I'll only cover the functional subset of the language in this chapter. If we are to manage errors, we must first understand them.

Note: Although Erlang includes a few ways to handle errors in functional code, most of the time you'll be told to just let it crash. I hinted to this in the Introduction. The mechanisms that let you program this way are in the concurrent part of the language.

A Compilation of Errors

There are many kinds of errors: compile-time errors, logical errors, run-time errors and generated errors. I'll focus on compile-time errors for this section and go through the others in the next sections.

Compile-time errors are often syntactic mistakes: check your function names, the tokens in the language (brackets, parentheses, periods, comas), the arity of your functions, etc. Here's a list of some of the common compile-time error messages and potential resolutions in case you encounter them:

module.beam: Module name 'madule' does not match file name 'module'

The module name you've entered in the -module attribute doesn't match the filename.

./module.erl:2: Warning: function some_function/0 is unused

You have not exported a function, or the place where it's used has the wrong name or arity. It's also possible you've written a function that is no longer needed. Check your code!

./module.erl:2: function some_function/1 undefined

The function does not exist. You've written the wrong name or arity either in the -export attribute or when declaring the function. This error is also output when the given function could not be compiled, usually because of a syntax error like forgetting to end a function with a period.

./module.erl:5: syntax error before: 'SomeCharacterOrWord'

This happens for a variety of reason, namely unclosed parentheses, tuples or wrong expression termination (like closing the last branch of a case with a comma). Other reasons might include the use of a reserved atom in your code or unicode characters getting converted weird between different encodings (I've seen it happen!)

./module.erl:5: syntax error before:

All right, that one is certainly not as descriptive! This usually comes up when your line termination is not right. This is a specific case of the previous error, so just keep an eye out.

./module.erl:5: Warning: this expression will fail with a 'badarith' exception

Erlang is all about dynamic typing, but remember that the types are strong. In this case, the compiler is smart enough to find that one of your arithmetic expressions will fail (say, llama + 5). It won't find type error much more complex than that, though.

./module.erl:5: Warning: variable 'Var' is unused

You declared a variable and never use it afterwards. This might be a bug with your code, so double-check what you have written. Otherwise, you might want to switch the variable name to _ or just prefix it with an underscore if you feel the name helps make the code readable.

./module.erl:5: Warning: a term is constructed, but never used

In one of your functions, you're doing something such as building a list, declaring a tuple or an anonymous function without ever binding it to a variable or returning it. This warning tells you you're doing something useless or that you have made some mistake.

./module.erl:5: head mismatch

It's possible your function has more than one head, and each of them has a different arity. Don't forget that different arity means different functions, and you can't interleave function declarations that way. Similarly, this error is raised when you insert a function definition between the head clauses of another function.

./module.erl:5: Warning: this clause cannot match because a previous clause at line 4 always matches

A function defined in the module has a specific clause defined after a catch-all one. As such, the compiler can warn you that you'll never even need to go to the other branch.

./module.erl:9: variable 'A' unsafe in 'case' (line 5)

You're using a variable declared within one of the branches of a case ... of outside of it. This is considered unsafe. If you want to use such variables, you'd be better of doing MyVar = case ... of...

This should cover most errors you get at compile-time at this point. There aren't too many and most of the time the hardest part is finding which error caused a huge cascade of errors listed against other functions. It is better to resolve compiler errors in the order they were reported to avoid being misled by errors which may not actually be errors at all. Other kinds of errors sometimes appear and if you've got one I haven't included, send me an email and I'll add it along with an explanation as soon as possible.

No, YOUR logic is wrong!

Logical errors are the hardest kind of errors to find and debug. They're most likely errors coming from the programmer: branches of conditional statements such as 'if's and 'case's that don't consider all the cases, mixing up a multiplication for a division, etc. They do not make your programs crash but just end up giving you unseen bad data or having your program work in unintended manner.

You're most likely on your own when it comes to this, but Erlang has many facilities to help you there, including test frameworks, TypEr and Dialyzer (as described in the types chapter), a debugger and tracing module, etc. Testing your code is likely your best defence. Sadly, there are enough of this kind of errors in every programmer's career to write a few dozen books about so I'll avoid losing too much time here. It's easier to focus on those that make your programs crash, because it happens right there and won't bubble up 50 levels from now. Note that this is pretty much the origin of the 'let it crash' ideal I mentioned a few times already.

Run-time Errors

Run-time errors are pretty destructive in the sense that it crashes your code. While Erlang has ways to deal with them, recognizing these errors is always helpful. As such, I've made a little list of common run-time errors with an explanation and example code that could generate them.

function_clause

1> lists:sort([3,2,1]).

[1,2,3]

2> lists:sort(fffffff).

** exception error: no function clause matching lists:sort(fffffff)

The most likely reason you'll have this error is when you fail all guard clauses of a function, or will just fail all pattern matches.

case_clause

3> case "Unexpected Value" of

3>    expected_value -> ok;

3>    other_expected_value -> 'also ok'

3> end.

** exception error: no case clause matching "Unexpected Value"

Looks like someone has forgotten a specific case, sends in the wrong kind of data, or needs a catch-all clause!

if_clause

4> if 2 > 4 -> ok;

4>    0 > 1 -> ok

4> end.

** exception error: no true branch found when evaluating an if expression

This is pretty similar to case_clause: it can not find a branch that evaluates to true. Making sure you consider all cases or add the catch-all true clause might be what you need.

badmatch

5> [X,Y] = {4,5}.

** exception error: no match of right hand side value {4,5}

Badmatch errors happen whenever pattern matching fails. This most likely means you're trying to do impossible pattern matches (such as above), trying to bind a variable for the second time, or just anything that isn't equal on both sides of the = operator (which is pretty much what makes rebinding a variable fail!). Note that this error sometimes happen because the programmer believes that a variable of the form _MyVar is the same as _. Variables with an underscore are normal variables, except the compiler won't complain if they're not used. It is not possible to bind them more than once.

badarg

6> erlang:binary_to_list("heh, already a list").

** exception error: bad argument

in function  binary_to_list/1

called as binary_to_list("heh, already a list")

This one is really similar to function_clause as it's about calling functions with incorrect arguments. The main difference here is that this error is usually triggered by the programmer after validating the arguments from within the function, outside of the guard clauses. I'll show how to throw such errors this later in this chapter.

undef

7> lists:random([1,2,3]).

** exception error: undefined function lists:random/1

This happens when you call a function that doesn't exist. Make sure the function is exported from the module with the right arity (if you're calling it from outside the module) and double check that you didn't typo the name of the function or the module either. Another reason to get the message is when the module is not in Erlang's search path. By default, Erlang's search path is set to be in the current directory. You can add paths by using code:add_patha/1 or code:add_pathz/1. If this still doesn't work, make sure you compiled the module to begin with!

badarith

8> 5 + llama.

** exception error: bad argument in an arithmetic expression

in operator  +/2

called as 5 + llama

This happens when you try to do arithmetic that doesn't exist, like divisions by zero or between atoms and numbers.

badfun

9> hhfuns:add(one,two).

** exception error: bad function one

in function  hhfuns:add/2

The most frequent reason why this error occurs is when you use variables as functions, but the variable's value is not a function. In the example above, I'm using the hhfuns function from the previous chapter and using two atoms as functions. This doesn't work and badfun is thrown.

badarity

10> F = fun(_) -> ok end.

#Fun<erl_eval.6.13229925>

11> F(a,b).

** exception error: interpreted function with arity 1 called with two arguments

The badarity error is a specific case of badfun: it happens when you use higher order functions, but you pass them more or less arguments than they can handle.

system_limit

There are many reasons why a system_limit error can be thrown: too many processes (we'll get there), atoms that are too long, too many arguments in a function, number of atoms too large, too many nodes connected, etc. Top get a full list in details, read the Erlang Efficiency Guide on system limits. Note that some of these errors are serious enough to crash the whole VM.

Raising Exceptions

In trying to monitor code's execution and protect against logical errors, it's often a good idea to provoke run-time crashes so problems will be spotted early.

There are three kinds of exceptions in Erlang: errors, throws and exits. They all have different uses (kind of):

Error

Calling erlang:error(Reason) will end the execution in the current process and include a stack trace of the last functions called with their arguments when you catch it. This is the kind of exceptions that provoke the run-time errors above.

Errors are the means for a function to stop its execution when you can't expect the calling code to handle what just happened. If you get an if_clause error, what can you do? Change the code and recompile, that's what you can do (other than just displaying a pretty error message). An example of when not to use errors could be our tree module from the recursion chapter. That module might not always be able to find a specific key in a tree when doing a lookup. In this case, it makes sense to expect the user to deal with unknown results: they could use a default value, check to insert a new one, delete the tree, etc. This is when it's appropriate to return a tuple of the form {ok, Value} or an atom like none rather than raising errors.

Now, errors aren't limited to the examples above. You can define your own kind of errors too:

1> erlang:error(badarith).

** exception error: bad argument in an arithmetic expression

2> erlang:error(custom_error).

** exception error: custom_error

Here, custom_error is not recognized by the Erlang shell and it has no custom translation such as "bad argument in ...", but it's usable in the same way and can be handled by the programmer in an identical manner (we'll see how to do that soon).

Exits

There are two kinds of exits: 'internal' exits and 'external' exits. Internal exits are triggered by calling the function exit/1 and make the current process stop its execution. External exits are called with exit/2 and have to do with multiple processes in the concurrent aspect of Erlang; as such, we'll mainly focus on internal exits and will visit the external kind later on.

Internal exits are pretty similar to errors. In fact, historically speaking, they were the same and only exit/1 existed. They've got roughly the same use cases. So how to choose one? Well the choice is not obvious. To understand when to use one or the other, there's no choice but to start looking at the concepts of actors and processes from far away.

In the introduction, I've compared processes as people communicating by mail. There's not a lot to add to the analogy, so I'll go to diagrams and bubbles.

Processes here can send each other messages. This is pretty much all you need to do the rest. A process can also listen for messages, wait for them. You can also choose what messages to listen to, discard some, ignore others, give up listening after a certain time etc.

These basic concepts let the implementors of Erlang use a special kind of message to communicate exceptions between processes. They act a bit like a process' last breathe; they're sent right before a process dies and the code it contains stops executing. Other processes that were listening for that specific kind of message can then know about the event and do whatever they please with it. This includes logging, restarting the process that died, etc.

With this concept explained, the difference in using erlang:error/1 and exit/1 is easier to understand. While both can be used in an extremely similar manner, the real difference is intent. You can then choose if what you've got is 'simply' an error or a condition worthy of killing the current process. This point is made stronger by the fact erlang:error/1 returns a stack trace and exit/1 doesn't. If you were to have a pretty large stack trace or lots of arguments to the current function, copying the exit message to every listening process would mean copying the data. In some cases, this could become unpractical.

Throws

A throw is a class of exceptions used for cases that the programmer can be expected to handle. To compare them with exits and errors, they don't really carry any 'crash that process!' intent behind them, but rather control flow. As you use throws while expecting the programmer to handle them, it's usually a good idea to document their use within a module using them.

To throw an exception, the syntax is:

1> throw(permission_denied).

** exception throw: permission_denied

Where you can replace permission_denied by anything you want (including 'everything is fine', but that is not helpful and you will lose friends).

Throws can also be used for non-local returns when in deep recursion. An example of that is the ssl module which uses throw/1 as a way to push {error, Reason} tuples back to a top-level function. This function then simply returns that tuple to the user. This lets the implementer only write for the successful cases and have one function deal with the exceptions on top of it all.

Another example could be the array module, where there is a lookup function that can return a user-supplied default value if it can't find the element needed. When the element can't be found, the value default is thrown as an exception, and the top-level function handles that and substitutes it by the user-supplied default value. This keeps the programmer of the module from needing to pass the default value as a parameter of every function of the lookup algorithm, again, focusing only on the successful cases.

As a rule of thumb, try to limit the use of your throws for non-local returns to a single module in order make it easier to debug your code. It will also let you change the innards of your module without requiring changes in its interface.

Dealing with Exceptions

I've mentioned quite a few times already that throws, errors and exits can be handled. The way to do this is by using a try ... catch expression.

A try ... catch is a way to evaluate an expression while letting you handle the successful case as well as the errors encountered. The general syntax for such an expression is:

try Expression of

SuccessfulPattern1 [Guards] ->

Expression1;

SuccessfulPattern2 [Guards] ->

Expression2

catch

TypeOfError:ExceptionPattern1 ->

Expression3;

TypeOfError:ExceptionPattern2 ->

Expression4

end.

The Expression in between try and of is said to be protected. This means that any kind of exception happening within that call will be caught. The patterns and expressions in between the try ... of and catch are behaving in exactly the same manner as a case ... of. Finally, the catch part. Here, you can replace TypeOfError by either error, throw or exit, for each respective type we've seen in this chapter. If no type is provided, a throw is assumed. So let's put this in practice.

First of all, let's start a module named exceptions. We're going for simple here:

-module(exceptions).

-compile(export_all).

throws(F) ->

try F() of

_ -> ok

catch

Throw -> {throw, caught, Throw}

end.

We can compile it and try it with different kinds of exceptions:

1> c(exceptions).

{ok,exceptions}

2> exceptions:throws(fun() -> throw(thrown) end).

{throw,caught,thrown}

3> exceptions:throws(fun() -> erlang:error(pang) end).

** exception error: pang

As you can see, this try ... catch is only receiving throws. As said earlier, this is because when no type is mentioned, a throw is assumed. Then we have functions with catch clauses of each type:

errors(F) ->

try F() of

_ -> ok

catch

error:Error -> {error, caught, Error}

end.

exits(F) ->

try F() of

_ -> ok

catch

exit:Exit -> {exit, caught, Exit}

end.

And to try them:

4> c(exceptions).

{ok,exceptions}

5> exceptions:errors(fun() -> erlang:error("Die!") end).

{error,caught,"Die!"}

6> exceptions:exits(fun() -> exit(goodbye) end).

{exit,caught,goodbye}

The next example on the menu shows how to combine all types of exceptions in a single try ... catch. We'll first declare a function to generate all the exceptions we need:

sword(1) -> throw(slice);

sword(2) -> erlang:error(cut_arm);

sword(3) -> exit(cut_leg);

sword(4) -> throw(punch);

sword(5) -> exit(cross_bridge).

black_knight(Attack) when is_function(Attack, 0) ->

try Attack() of

_ -> "None shall pass."

catch

throw:slice -> "It is but a scratch.";

error:cut_arm -> "I've had worse.";

exit:cut_leg -> "Come on you pansy!";

_:_ -> "Just a flesh wound."

end.

Where is_function/2 is a BIF which makes sure the variable Attack is a function of arity 0. Then we add this one for good measure:

talk() -> "blah blah".

And now for something completely different:

7> c(exceptions).

{ok,exceptions}

8> exceptions:talk().

"blah blah"

9> exceptions:black_knight(fun exceptions:talk/0).

"None shall pass."

10> exceptions:black_knight(fun() -> exceptions:sword(1) end).

"It is but a scratch."

11> exceptions:black_knight(fun() -> exceptions:sword(2) end).

"I've had worse."

12> exceptions:black_knight(fun() -> exceptions:sword(3) end).

"Come on you pansy!"

13> exceptions:black_knight(fun() -> exceptions:sword(4) end).

"Just a flesh wound."

14> exceptions:black_knight(fun() -> exceptions:sword(5) end).

"Just a flesh wound."

The expression on line 9 demonstrates normal behavior for the black knight, when normal function execution happens. Each line that follows that one demonstrates pattern matching on exceptions according to their class (throw, error, exit) and the reason associated with them (slice, cut_arm, cut_leg).

One thing shown here on expressions 13 and 14 is a catch-all clause for exceptions. The _:_ pattern is what you need to use to make sure to catch any exception from any type. In practice, you should be careful when using the catch-all patterns: try to protect your code from what you can handle, but not more. Erlang has other facilities in place to take care of the rest.

There's also an additional clause that can be added after a try ... catch that will always be executed. This is equivalent to the 'finally' block in many other languages:

try Expr of

Pattern -> Expr1

catch

Type:Exception -> Expr2

after % this always gets executed

Expr3

end

No matter if there are errors or not, the expressions inside the after part are guaranteed to run. However, you can not get any return value out of the after construct. Therefore, after is mostly used to run code with side effects. The canonical use of this is when you want to make sure a file you were reading gets closed whether exceptions are raised or not.

We now know how to handle the 3 classes of exceptions in Erlang with catch blocks. However, I've hidden information from you: it's actually possible to have more than one expression between the try and the of!

whoa() ->

try

talk(),

_Knight = "None shall Pass!",

_Doubles = [N*2 || N <- lists:seq(1,100)],

throw(up),

_WillReturnThis = tequila

of

tequila -> "hey this worked!"

catch

Exception:Reason -> {caught, Exception, Reason}

end.

By calling exceptions:whoa(), we'll get the obvious {caught, throw, up}, because of throw(up). So yeah, it's possible to have more than one expression between try and of...

What exceptions:whoa/0 highlighted that you might have not noticed is that when we use many expressions in that manner, we might not always care about what the return value is. The of part thus becomes a bit useless. Well good news, you can just give it up:

im_impressed() ->

try

talk(),

_Knight = "None shall Pass!",

_Doubles = [N*2 || N <- lists:seq(1,100)],

throw(up),

_WillReturnThis = tequila

catch

Exception:Reason -> {caught, Exception, Reason}

end.

And now it's a bit leaner!

Note: It is important to know that the protected part of an exception can't be tail recursive. The VM must always keep a reference there in case there's an exception popping up.

Because the try ... catch construct without the of part has nothing but a protected part, calling a recursive function from there might be dangerous for programs supposed to run for a long time (which is Erlang's niche). After enough iterations, you'll go out of memory or your program will get slower without really knowing why. By putting your recursive calls between the of and catch, you are not in a protected part and you will benefit from Last Call Optimisation.

Some people use try ... of ... catch rather than try ... catch by default to avoid unexpected errors of that kind, except for obviously non-recursive code with a result they don't care about. You're most likely able to make your own decision on what to do!

Wait, there's more!

As if it wasn't enough to be on par with most languages already, Erlang's got yet another error handling structure. That structure is defined as the keyword catch and basically captures all types of exceptions on top of the good results. It's a bit of a weird one because it displays a different representation of exceptions:

1> catch throw(whoa).

whoa

2> catch exit(die).

{'EXIT',die}

3> catch 1/0.

{'EXIT',{badarith,[{erlang,'/',[1,0]},

{erl_eval,do_apply,5},

{erl_eval,expr,5},

{shell,exprs,6},

{shell,eval_exprs,6},

{shell,eval_loop,3}]}}

4> catch 2+2.

4

What we can see from this is that throws remain the same, but that exits and errors are both represented as {'EXIT', Reason}. That's consequent to errors being bolted to the language after exits (they kept a similar representation for backwards compatibility).

The way to read the stack trace is as follows:

5> catch doesnt:exist(a,4).

{'EXIT',{undef,[{doesnt,exist,[a,4]},

{erl_eval,do_apply,5},

{erl_eval,expr,5},

{shell,exprs,6},

{shell,eval_exprs,6},

{shell,eval_loop,3}]}}

- The type of error is undef, which means the function you called is not defined (see the list at the beginning of this chapter)

- The list right after the type of error is a stack trace

- The tuple on top of the stack trace represents the last function to be called ({Module, Function, Arguments}). That's your undefined function.

- The tuples after that are the functions called before the error. This time they're of the form {Module, Function, Arity}.

- That's all there is to it, really.

You can also manually get a stack trace by calling erlang:get_stacktrace/0 in the process that crashed.

You'll often see catch written in the following manner (we're still in exceptions.erl):

catcher(X,Y) ->

case catch X/Y of

{'EXIT', {badarith,_}} -> "uh oh";

N -> N

end.

And as expected:

6> c(exceptions).

{ok,exceptions}

7> exceptions:catcher(3,3).

1.0

8> exceptions:catcher(6,3).

2.0

9> exceptions:catcher(6,0).

"uh oh"

This sounds compact and easy to catch exceptions, but there are a few problems with catch. The first of it is operator precedence:

10> X = catch 4+2.

* 1: syntax error before: 'catch'

10> X = (catch 4+2).

6

That's not exactly intuitive given most expressions do not need to be wrapped in parentheses this way. Another problem with catch is that you can't see the difference between what looks like the underlying representation of an exception and a real exception:

11> catch erlang:boat().

{'EXIT',{undef,[{erlang,boat,[]},

{erl_eval,do_apply,5},

{erl_eval,expr,5},

{shell,exprs,6},

{shell,eval_exprs,6},

{shell,eval_loop,3}]}}

12> catch exit({undef, [{erlang,boat,[]}, {erl_eval,do_apply,5}, {erl_eval,expr,5}, {shell,exprs,6}, {shell,eval_exprs,6}, {shell,eval_loop,3}]}).

{'EXIT',{undef,[{erlang,boat,[]},

{erl_eval,do_apply,5},

{erl_eval,expr,5},

{shell,exprs,6},

{shell,eval_exprs,6},

{shell,eval_loop,3}]}}

And you can't know the difference between an error and an actual exit. You could also have used throw/1 to generate the above exception. In fact, a throw/1 in a catch might also be problematic in another scenario:

one_or_two(1) -> return;

one_or_two(2) -> throw(return).

And now the killer problem:

13> c(exceptions).

{ok,exceptions}

14> catch exceptions:one_or_two(1).

return

15> catch exceptions:one_or_two(2).

return

Because we're behind a catch, we can never know if the function threw an exception of if it returned an actual value! This might not really happen a whole lot in practice, but it's still a wart big enough to have warranted the addition of the try ... catch construct in the R10B release.

Try a try in a tree

To put exceptions in practice, we'll do a little exercise requiring us to dig for our tree module. We're going to add a function that lets us do a lookup in the tree to find out whether a value is already present in there or not. Because the tree is ordered by its keys and in this case we do not care about the keys, we'll need to traverse the whole thing until we find the value.

The traversal of the tree will be roughly similar to what we did in tree:lookup/2, except this time we will always search down both the left branch and then the right branch. To write the function, you'll just need to remember that a tree node is either {node, {Key, Value, NodeLeft, NodeRight}} or {node, 'nil'} when empty. With this in hand, we can write a basic implementation without exceptions:

has_value(_, {node, 'nil'}) ->

false;

has_value(Val, {node, {_, Val, _, _}}) ->

true;

has_value(Val, {node, {_, _, Left, Right}}) ->

case has_value(Val, Left) of

true -> true;

false -> has_value(Val, Right)

end.

The problem with this implementation is that every node of the tree we branch at has to test for the result of the previous branch:

This is a bit annoying. With the help of throws, we can make something that will require less comparisons:

has_value(Val, Tree) ->

try has_value1(Val, Tree) of

false -> false

catch

true -> true

end.

has_value1(_, {node, 'nil'}) ->

false;

has_value1(Val, {node, {_, Val, _, _}}) ->

throw(true);

has_value1(Val, {node, {_, _, Left, Right}}) ->

has_value1(Val, Left),

has_value1(Val, Right).

The execution of the code above is similar to the previous version, except that we never need to check for the return value: we don't care about it at all. In this version, only a throw means the value was found. When this happens, the tree evaluation stops and it falls back to the catch on top. Otherwise, the execution keeps going until the last false is returned and that's what the user sees:

Of course, the implementation above is longer than the previous one. However, it is possible to gain in speed and in clarity by using non-local returns with a throw, depending on the operations you're doing. The current example is a simple comparison and there's not much to see, but the practice still makes sense with more complex data structures and operations.

That being said, we're probably ready to solve real problems in sequential Erlang.

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Functionally Solving Problems

Sounds like we're ready to do something practical with all that Erlang juice we drank. Nothing new is going to be shown but how to apply bits of what we've seen before. The problems in this chapter were taken from Miran's Learn You a Haskell. I decided to take the same solutions so curious readers can compare solutions in Erlang and Haskell as they wish. If you do so, you might find the final results to be pretty similar for two languages with such different syntaxes. This is because once you know functional concepts, they're relatively easy to carry over to other functional languages.

Reverse Polish Notation Calculator

Most people have learned to write arithmetic expressions with the operators in-between the numbers ((2 + 2) / 5). This is how most calculators let you insert equations and probably the notation you were taught to count with in school. This notation has the downside of needing you to know about operator precedence: multiplication and division are more important (have a higher precedence) than addition and subtraction.

Another notation exists, called prefix notation or Polish notation, where the operator comes before the operands. Under this notation, (2 + 2) / 5 would become (/ (+ 2 2) 5). If we decide to say + and / always take two arguments, then (/ (+ 2 2) 5) can simply be written as / + 2 2 5.

However, we will instead focus on Reverse Polish notation (or just RPN), which is the opposite of prefix notation: the operator follows the operands. The same example as above in RPN would be written 2 2 + 5 /. Other example equations could be 9 * 5 + 7 or 10 * 2 * (3 + 4) / 2 which get translated to 9 5 * 7 + and 10 2 * 3 4 + * 2 /, respectively. This notation was used a whole lot in early models of calculators as it would take little memory to use. In fact some people still carry RPN calculators around. We'll write one of these.

First of all, it might be good to understand how to read RPN equations. One way to do it is to find the operators one by one and then regroup them with their operands by arity:

10 4 3 + 2 * -

10 (4 3 +) 2 * -

10 ((4 3 +) 2 *) -

(10 ((4 3 +) 2 *) -)

(10 (7 2 *) -)

(10 14 -)

-4

However, in the context of a computer or a calculator, a simpler way to do it is to make a stack of all the operands as we see them. Taking the equation 10 4 3 + 2 * -, the first operand we see is 10. We add that to the stack. Then there's 4, so we also push that on top of the stack. In third place, we have 3; let's push that one on the stack too. Our stack should now look like this:

The next character to parse is a +. That one is a function of arity 2. In order to use it we will need to feed it two operands, which will be taken from the stack:

So we take that 7 and push it back on top of the stack (yuck, we don't want to keep these filthy numbers floating around!) The stack is now [7,10] and what's left of the equation is 2 * -. We can take the 2 and push it on top of the stack. We then see *, which needs two operands to work. Again, we take them from the stack:

And push 14 back on top of our stack. All that remains is -, which also needs two operands. O Glorious luck! There are two operands left in our stack. Use them!

And so we have our result. This stack-based approach is relatively fool-proof and the low amount of parsing needed to be done before starting to calculate results explains why it was a good idea for old calculators to use this. There are other reasons to use RPN, but this is a bit out of the scope of this guide, so you might want to hit the Wikipedia article instead.

Writing this solution in Erlang is not too hard once we've done the complex stuff. It turns out the tough part is figuring out what steps need to be done in order to get our end result and we just did that. Neat. Open a file named calc.erl.

The first part to worry about is how we're going to represent an equation. To make things simple, we'll probably input them as a string: "10 4 3 + 2 * -". This string has whitespace, which isn't part of our problem-solving process, but is necessary in order to use a simple tokenizer. What would be usable then is a list of terms of the form ["10","4","3","+","2","*","-"] after going through the tokenizer. Turns out the function string:tokens/2 does just that:

1> string:tokens("10 4 3 + 2 * -", " ").

["10","4","3","+","2","*","-"]

That will be a good representation for our equation. The next part to define is the stack. How are we going to do that? You might have noticed that Erlang's lists act a lot like a stack. Using the cons (|) operator in [Head|Tail] effectively behaves the same as pushing Head on top of a stack (Tail, in this case). Using a list for a stack will be good enough.

To read the equation, we just have to do the same as we did when solving the problem by hand. Read each value from the equation, if it's a number, put it on the stack. If it's a function, pop all the values it needs from the stack, then push the result back in. To generalize, all we need to do is go over the whole equation as a loop only once and accumulate the results. Sounds like the perfect job for a fold!

What we need to plan for is the function that lists:foldl/3 will apply on every operator and operand of the equation. This function, because it will be run in a fold, will need to take two arguments: the first one will be the element of the equation to work with and the second one will be the stack.

We can start writing our code in the calc.erl file. We'll write the function responsible for all the looping and also the removal of spaces in the equation:

-module(calc).

-export([rpn/1]).

rpn(L) when is_list(L) ->

[Res] = lists:foldl(fun rpn/2, [], string:tokens(L, " ")),

Res.

We'll implement rpn/2 next. Note that because each operator and operand from the equation ends up being put on top of the stack, the solved equation's result will be on that stack. We need to get that last value out of there before returning it to the user. This is why we pattern match over [Res] and only return Res.

Alright, now to the harder part. Our rpn/2 function will need to handle the stack for all values passed to it. The head of the function will probably look like rpn(Op,Stack) and its return value like [NewVal|Stack]. When we get regular numbers, the operation will be:

rpn(X, Stack) -> [read(X)|Stack].

Here, read/1 is a function that converts a string to an integer or floating point value. Sadly, there is no built-in function to do this in Erlang (only one or the other). We'll add it ourselves:

read(N) ->

case string:to_float(N) of

{error,no_float} -> list_to_integer(N);

{F,_} -> F

end.

Where string:to_float/1 does the conversion from a string such as "13.37" to its numeric equivalent. However, if there is no way to read a floating point value, it returns {error,no_float}. When that happens, we need to call list_to_integer/1 instead.

Now back to rpn/2. The numbers we encounter all get added to the stack. However, because our pattern matches on anything (see Pattern Matching), operators will also get pushed on the stack. To avoid this, we'll put them all in preceding clauses. The first one we'll try this with is the addition:

rpn("+", [N1,N2|S]) -> [N1+N2|S];

rpn(X, Stack) -> [read(X)|Stack].

We can see that whenever we encounter the "+" string, we take two numbers from the top of the stack (N1,N2) and add them before pushing the result back onto that stack. This is exactly the same logic we applied when solving the problem by hand. Trying the program we can see that it works:

1> c(calc).

{ok,calc}

2> calc:rpn("3 5 +").

8

3> calc:rpn("7 3 + 5 +").

15

The rest is trivial, as you just need to add all the other operators:

rpn("+", [N1,N2|S]) -> [N2+N1|S];

rpn("-", [N1,N2|S]) -> [N2-N1|S];

rpn("*", [N1,N2|S]) -> [N2*N1|S];

rpn("/", [N1,N2|S]) -> [N2/N1|S];

rpn("^", [N1,N2|S]) -> [math:pow(N2,N1)|S];

rpn("ln", [N|S])    -> [math:log(N)|S];

rpn("log10", [N|S]) -> [math:log10(N)|S];

rpn(X, Stack) -> [read(X)|Stack].

Note that functions that take only one argument such as logarithms only need to pop one element from the stack. It is left as an exercise to the reader to add functions such as 'sum' or 'prod' which return the sum of all the elements read so far or the products of them all. To help you out, they are implemented in my version of calc.erl already.

To make sure this all works fine, we'll write very simple unit tests. Erlang's = operator can act as an assertion function. Assertions should crash whenever they encounter unexpected values, which is exactly what we need. Of course, there are more advanced testing frameworks for Erlang, including Common Test and EUnit. We'll check them out later, but for now the basic = will do the job:

rpn_test() ->

5 = rpn("2 3 +"),

87 = rpn("90 3 -"),

-4 = rpn("10 4 3 + 2 * -"),

-2.0 = rpn("10 4 3 + 2 * - 2 /"),

ok = try

rpn("90 34 12 33 55 66 + * - +")

catch

error:{badmatch,[_|_]} -> ok

end,

4037 = rpn("90 34 12 33 55 66 + * - + -"),

8.0 = rpn("2 3 ^"),

true = math:sqrt(2) == rpn("2 0.5 ^"),

true = math:log(2.7) == rpn("2.7 ln"),

true = math:log10(2.7) == rpn("2.7 log10"),

50 = rpn("10 10 10 20 sum"),

10.0 = rpn("10 10 10 20 sum 5 /"),

1000.0 = rpn("10 10 20 0.5 prod"),

ok.

The test function tries all operations; if there's no exception raised, the tests are considered successful. The first four tests check that the basic arithmetic functions work right. The fifth test specifies behaviour I have not explained yet. The try ... catch expects a badmatch error to be thrown because the equation can't work:

90 34 12 33 55 66 + * - +

90 (34 (12 (33 (55 66 +) *) -) +)

At the end of rpn/1, the values -3947 and 90 are left on the stack because there is no operator to work on the 90 that hangs there. Two ways to handle this problem are possible: either ignore it and only take the value on top of the stack (which would be the last result calculated) or crash because the arithmetic is wrong. Given Erlang's policy is to let it crash, it's what was chosen here. The part that actually crashes is the [Res] in rpn/1. That one makes sure only one element, the result, is left in the stack.

The few tests that are of the form true = FunctionCall1 == FunctionCall2 are there because you can't have a function call on the left hand side of =. It still works like an assert because we compare the comparison's result to true.

I've also added the test cases for the sum and prod operators so you can exercise yourselves implementing them. If all tests are successful, you should see the following:

1> c(calc).

{ok,calc}

2> calc:rpn_test().

ok

3> calc:rpn("1 2 ^ 2 2 ^ 3 2 ^ 4 2 ^ sum 2 -").

28.0

Where 28 is indeed equal to sum(1² + 2² + 3² + 4²) - 2. Try as many of them as you wish.

One thing that could be done to make our calculator better would be to make sure it raises badarith errors when it crashes because of unknown operators or values left on the stack, rather than our current badmatch error. It would certainly make debugging easier for the user of the calc module.

Heathrow to London

Our next problem is also taken from Learn You a Haskell. You're on a plane due to land at Heathrow airport in the next hours. You have to get to London as fast as possible; your rich uncle is dying and you want to be the first there to claim dibs on his estate.

There are two roads going from Heathrow to London and a bunch of smaller streets linking them together. Because of speed limits and usual traffic, some parts of the roads and smaller streets take longer to drive on than others. Before you land, you decide to maximize your chances by finding the optimal path to his house. Here's the map you've found on your laptop:

Having become a huge fan of Erlang after reading online books, you decide to solve the problem using that language. To make it easier to work with the map, you enter data the following way in a file named road.txt:

50

10

30

5

90

20

40

2

25

10

8

0

The road is laid in the pattern: A1, B1, X1, A2, B2, X2, ..., An, Bn, Xn, where X is one of the roads joining the A side to the B side of the map. We insert a 0 as the last X segment, because no matter what we do we're at our destination already. Data can probably be organized in tuples of 3 elements (triples) of the form {A,B,X}.

The next thing you realize is that it's worth nothing to try to solve this problem in Erlang when you don't know how to solve it by hand to begin with. In order to do this, we'll use what recursion taught us.

When writing a recursive function, the first thing to do is to find our base case. For our problem at hand, this would be if we had only one tuple to analyze, that is, if we only had to choose between A, B (and crossing X, which in this case is useless because we're at destination):

Then the choice is only between picking which of path A or path B is the shortest. If you've learned your recursion right, you know that we ought to try and converge towards the base case. This means that on each step we'll take, we'll want to reduce the problem to choosing between A and B for the next step.

Let's extend our map and start over:

Ah! It gets interesting! How can we reduce the triple {5,1,3} to a strict choice between A and B? Let's see how many options are possible for A. To get to the intersection of A1 and A2 (I'll call this the point A1), I can either take road A1 directly (5), or come from B1 (1) and then cross over X1 (3). In this case, The first option (5) is longer than the second one (4). For the option A, the shortest path is [B, X]. So what are the options for B? You can either proceed from A1 (5) then cross over X1 (3), or strictly take the path B1 (1).

Alright! What we've got is a length 4 with the path [B, X] towards the first intersection A and a length 1 with the path [B] towards the intersection of B1 and B2. We then have to decide what to pick between going to the second point A (the intersection of A2 and the endpoint or X2) and the second point B (intersection of B2 and X2). To make a decision, I suggest we do the same as before. Now you don't have much choice but to obey, given I'm the guy writing this text. Here we go!

All possible paths to take in this case can be found in the same way as the previous one. We can get to the next point A by either taking the path A2 from [B, X], which gives us a length of 14 (14 = 4 + 10), or by taking B2 then X2 from [B], which gives us a length of 16 (16 = 1 + 15 + 0). In this case, the path [B, X, A] is better than [B, B, X].

We can also get to the next point B by either taking the path A2 from [B, X] and then crossing over X2 for a length of 14 (14 = 4 + 10 + 0), or by taking the road B2 from [B] for a length of 16 (16 = 1 + 15). Here, the best path is to pick the first option, [B, X, A, X].

So when this whole process is done, we're left with two paths, A or B, both of length 14. Either of them is the right one. The last selection will always have two paths of the same length, given the last X segment has a length 0. By solving our problem recursively, we've made sure to always get the shortest path at the end. Not too bad, eh?

Subtly enough, we've given ourselves the basic logical parts we need to build a recursive function. You can implement it if you want, but I promised we would have very few recursive functions to write ourselves. We'll use a fold.

Note: while I have shown folds being used and constructed with lists, folds represent a broader concept of iterating over a data structure with an accumulator. As such, folds can be implemented over trees, dictionaries, arrays, database tables, etc.

It is sometimes useful when experimenting to use abstractions like maps and folds; they make it easier to later change the data structure you use to work with your own logic.

So where were we? Ah, yes! We had the file we're going to feed as input ready. To do file manipulations, the file module is our best tool. It contains many functions common to many programming languages in order to deal with files themselves (setting permissions, moving files around, renaming and deleting them, etc.)

It also contains the usual functions to read and/or write from files such as: file:open/2 and file:close/1 to do as their names say (opening and closing files!), file:read/2 to get the content a file (either as string or a binary), file:read_line/1 to read a single line, file:position/3 to move the pointer of an open file to a given position, etc.

There's a bunch of shortcut functions in there too, such as file:read_file/1 (opens and reads the contents as a binary), file:consult/1 (opens and parses a file as Erlang terms) or file:pread/2 (changes a position and then reads) and pwrite/2 (changes the position and writes content).

With all these choices available, it's going to be easy to find a function to read our road.txt file. Because we know our road is relatively small, we're going to call file:read_file("road.txt").':

1> {ok, Binary} = file:read_file("road.txt").

{ok,<<"50\r\n10\r\n30\r\n5\r\n90\r\n20\r\n40\r\n2\r\n25\r\n10\r\n8\r\n0\r\n">>}

2> S = string:tokens(binary_to_list(Binary), "\r\n\t ").

["50","10","30","5","90","20","40","2","25","10","8","0"]

Note that in this case, I added a space (" ") and a tab ("\t") to the valid tokens so the file could have been written in the form "50 10 30 5 90 20 40 2 25 10 8 0" too. Given that list, we'll need to transform the strings into integers. We'll use a similar manner to what we used in our RPN calculator:

3> [list_to_integer(X) || X <- S].

[50,10,30,5,90,20,40,2,25,10,8,0]

Let's start a new module called road.erl and write this logic down:

-module(road).

-compile(export_all).

main() ->

File = "road.txt",

{ok, Bin} = file:read_file(File),

parse_map(Bin).

parse_map(Bin) when is_binary(Bin) ->

parse_map(binary_to_list(Bin));

parse_map(Str) when is_list(Str) ->

[list_to_integer(X) || X <- string:tokens(Str,"\r\n\t ")].

The function main/0 is here responsible from reading the content of the file and passing it on to parse_map/1. Because we use the function file:read_file/1 to get the contents out of road.txt, the result we obtain is a binary. For this reason, I've made the function parse_map/1 match on both lists and binaries. In the case of a binary, we just call the function again with the string being converted to a list (our function to split the string works on lists only.)

The next step in parsing the map would be to regroup the data into the {A,B,X} form described earlier. Sadly, there's no simple generic way to pull elements from a list 3 at a time, so we'll have to pattern match our way in a recursive function in order to do it:

group_vals([], Acc) ->

lists:reverse(Acc);

group_vals([A,B,X|Rest], Acc) ->

group_vals(Rest, [{A,B,X} | Acc]).

That function works in a standard tail-recursive manner; there's nothing too complex going on here. We'll just need to call it by modifying parse_map/1 a bit:

parse_map(Bin) when is_binary(Bin) ->

parse_map(binary_to_list(Bin));

parse_map(Str) when is_list(Str) ->

Values = [list_to_integer(X) || X <- string:tokens(Str,"\r\n\t ")],

group_vals(Values, []).

If we try and compile it all, we should now have a road that makes sense:

1> c(road).

{ok,road}

2> road:main().

[{50,10,30},{5,90,20},{40,2,25},{10,8,0}]

Ah yes, that looks right. We get the blocks we need to write our function that will then fit in a fold. For this to work, finding a good accumulator is necessary.

To decide what to use as an accumulator, the method I find the easiest to use is to imagine myself in the middle of the algorithm while it runs. For this specific problem, I'll imagine that I'm currently trying to find the shortest path of the second triple ({5,90,20}). To decide on which path is the best, I need to have the result from the previous triple. Luckily, we know how to do it, because we don't need an accumulator and we got all that logic out already. So for A:

And take the shortest of these two paths. For B, it was similar:

So now we know that the current best path coming from A is [B, X]. We also know it has a length of 40. For B, the path is simply [B] and the length is 10. We can use this information to find the next best paths for A and B by reapplying the same logic, but counting the previous ones in the equation. The other data we need is the path traveled so we can show it to the user. Given we need two paths (one for A and one for B) and two accumulated lengths, our accumulator can take the form {{DistanceA, PathA}, {DistanceB, PathB}}. That way, each iteration of the fold has access to all the state and we build it up to show it to the user in the end.

This gives us all the parameters our function will need: the {A,B,X} triples and an accumulator of the form {{DistanceA,PathA}, {DistanceB,PathB}}.

Putting this into code in order to get our accumulator can be done the following way:

shortest_step({A,B,X}, {{DistA,PathA}, {DistB,PathB}}) ->

OptA1 = {DistA + A, [{a,A}|PathA]},

OptA2 = {DistB + B + X, [{x,X}, {b,B}|PathB]},

OptB1 = {DistB + B, [{b,B}|PathB]},

OptB2 = {DistA + A + X, [{x,X}, {a,A}|PathA]},

{erlang:min(OptA1, OptA2), erlang:min(OptB1, OptB2)}.

Here, OptA1 gets the first option for A (going through A), OptA2 the second one (going through B then X). The variables OptB1 and OptB2 get the similar treatment for point B. Finally, we return the accumulator with the paths obtained.

About the paths saved in the code above, note that I decided to use the form [{x,X}] rather than [x] for the simple reason that it might be nice for the user to know the length of each segment. The other thing I'm doing is that I'm accumulating the paths backwards ({x,X} comes before {b,B}.) This is because we're in a fold, which is tail recursive: the whole list is reversed, so it is necessary to put the last one traversed before the others.

Finally, I use erlang:min/2 to find the shortest path. It might sound weird to use such a comparison function on tuples, but remember that every Erlang term can be compared to any other! Because the length is the first element of the tuple, we can sort them that way.

What's left to do is to stick that function into a fold:

optimal_path(Map) ->

{A,B} = lists:foldl(fun shortest_step/2, {{0,[]}, {0,[]}}, Map),

{_Dist,Path} = if hd(element(2,A)) =/= {x,0} -> A;

hd(element(2,B)) =/= {x,0} -> B

end,

lists:reverse(Path).

At the end of the fold, both paths should end up having the same distance, except one's going through the final {x,0} segment. The if looks at the last visited element of both paths and returns the one that doesn't go through {x,0}. Picking the path with the fewest steps (compare with length/1) would also work. Once the shortest one has been selected, it is reversed (it was built in a tail-recursive manner; you must reverse it). You can then display it to the world, or keep it secret and get your rich uncle's estate. To do that, you have to modify the main function to call optimal_path/1. Then it can be compiled.

main() ->

File = "road.txt",

{ok, Bin} = file:read_file(File),

optimal_path(parse_map(Bin)).

Oh, look! We've got the right answer! Great Job!

1> c(road).

{ok,road}

2> road:main().

[{b,10},{x,30},{a,5},{x,20},{b,2},{b,8}]

Or, to put it in a visual way:

But you know what would be really useful? Being able to run our program from outside the Erlang shell. We'll need to change our main function again:

main([FileName]) ->

{ok, Bin} = file:read_file(FileName),

Map = parse_map(Bin),

io:format("~p~n",[optimal_path(Map)]),

erlang:halt().

The main function now has an arity of 1, needed to receive parameters from the command line. I've also added the function erlang:halt/0, which will shut down the Erlang VM after being called. I've also wrapped the call to optimal_path/1 into io:format/2 because that's the only way to have the text visible outside the Erlang shell.

With all of this, your road.erl file should now look like this (minus comments):

-module(road).

-compile(export_all).

main([FileName]) ->

{ok, Bin} = file:read_file(FileName),

Map = parse_map(Bin),

io:format("~p~n",[optimal_path(Map)]),

erlang:halt(0).

%% Transform a string into a readable map of triples

parse_map(Bin) when is_binary(Bin) ->

parse_map(binary_to_list(Bin));

parse_map(Str) when is_list(Str) ->

Values = [list_to_integer(X) || X <- string:tokens(Str,"\r\n\t ")],

group_vals(Values, []).

group_vals([], Acc) ->

lists:reverse(Acc);

group_vals([A,B,X|Rest], Acc) ->

group_vals(Rest, [{A,B,X} | Acc]).

%% Picks the best of all paths, woo!

optimal_path(Map) ->

{A,B} = lists:foldl(fun shortest_step/2, {{0,[]}, {0,[]}}, Map),

{_Dist,Path} = if hd(element(2,A)) =/= {x,0} -> A;

hd(element(2,B)) =/= {x,0} -> B

end,

lists:reverse(Path).

%% actual problem solving

%% change triples of the form {A,B,X}

%% where A,B,X are distances and a,b,x are possible paths

%% to the form {DistanceSum, PathList}.

shortest_step({A,B,X}, {{DistA,PathA}, {DistB,PathB}}) ->

OptA1 = {DistA + A, [{a,A}|PathA]},

OptA2 = {DistB + B + X, [{x,X}, {b,B}|PathB]},

OptB1 = {DistB + B, [{b,B}|PathB]},

OptB2 = {DistA + A + X, [{x,X}, {a,A}|PathA]},

{erlang:min(OptA1, OptA2), erlang:min(OptB1, OptB2)}.

And running the code:

$ erlc road.erl

$ erl -noshell -run road main road.txt

[{b,10},{x,30},{a,5},{x,20},{b,2},{b,8}]

And yep, it's right! It's pretty much all you need to do to get things to work. You could make yourself a bash/batch file to wrap the line into a single executable, or you could check out escript to get similar results.

As we've seen with these two exercises, solving problems is much easier when you break them off in small parts that you can solve individually before piecing everything together. It's also not worth much to go ahead and program something without understanding it. Finally, a few tests are always appreciated. They'll let you make sure everything works fine and will let you change the code without changing the results at the end.

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

A Short Visit to Common Data Structures

Won't be too long, promised!

Chances are you now understand the functional subset of Erlang pretty well and could read many programs without a problem. However, I could bet it's still a bit hard to think about how to build a real useful program even though the last chapter was about solving problems in a functional manner. I'm saying this because it's how I felt like at about that point in my learning, but if you're doing better, congratulations!

Anyway, the point I'm coming to is that we've seen a bunch of things: most basic data types, the shell, how to write modules and functions (with recursion), different ways to compile, control the flow of the program, handle exceptions, abstract away some common operations, etc. We've also seen how to store data with tuples, lists and an incomplete implementation of a binary search tree. What we haven't seen is the other data structures provided to the programmer in the Erlang standard library.

Records

Records are, first of all, a hack. They are more or less an afterthought to the language and can have their share of inconveniences. I'll cover that later. They're still pretty useful whenever you have a small data structure where you want to access the attributes by name directly. As such, Erlang records are a lot like structs in C (if you know C.)

They're declared as module attributes in the following manner:

-module(records).

-compile(export_all).

-record(robot, {name,

type=industrial,

hobbies,

details=[]}).

So here we have a record representing robots with 4 fields: name, type, hobbies and details. There is also a default value for type and details, industrial and [], respectively. Here's how to declare a record in the module records:

first_robot() ->

#robot{name="Mechatron",

type=handmade,

details=["Moved by a small man inside"]}.

And running the code:

1> c(records).

{ok,records}

2> records:first_robot().

{robot,"Mechatron",handmade,undefined,

["Moved by a small man inside"]

Woops! Here comes the hack! Erlang records are just syntactic sugar on top of tuples. Fortunately, there's a way to make it better. The Erlang shell has a command rr(Module) that lets you load record definitions from Module:

3> rr(records).

[robot]

4> records:first_robot().

#robot{name = "Mechatron",type = handmade,

hobbies = undefined,

details = ["Moved by a small man inside"]}

Ah there! This makes it much easier to work with records that way. You'll notice that in first_robot/0, we had not defined the hobbies field and it had no default value in its declaration. Erlang, by defaults, sets the value to undefined for you.

To see the behavior of the defaults we set in the robot definition, let's compile the following function:

car_factory(CorpName) ->

#robot{name=CorpName, hobbies="building cars"}.

And run it:

5> c(records).

{ok,records}

6> records:car_factory("Jokeswagen").

#robot{name = "Jokeswagen",type = industrial,

hobbies = "building cars",details = []}

And we have an industrial robot that likes to spend time building cars.

Note: The function rr() can take more than a module name: it can take a wildcard (like rr("*")) and also a list as a second argument to specify which records to load.

There are a few other functions to deal with records in the shell: rd(Name, Definition) lets you define a record in a manner similar to the -record(Name, Definition) used in our module. You can use rf() to 'unload' all records, or rf(Name) or rf([Names]) to get rid of specific definitions.

You can use rl() to print all record definitions in a way you could copy-paste into the module or use rl(Name) or rl([Names]) to restrict it to specific records.

Finally, rp(Term) lets you convert a tuple to a record (given the definition exists).

Writing records alone won't do much. We need a way to extract values from them. There are basically two ways to do this. The first one is with a special 'dot syntax'. Assuming you have the record definition for robots loaded:

5> Crusher = #robot{name="Crusher", hobbies=["Crushing people","petting cats"]}.

#robot{name = "Crusher",type = industrial,

hobbies = ["Crushing people","petting cats"],

details = []}

6> Crusher#robot.hobbies.

["Crushing people","petting cats"]

Ugh, not a pretty syntax. This is due to the nature of records as tuples. Because they're just some kind of compiler trick, you have to keep keywords around defining what record goes with what variable, hence the #robot part of Crusher#robot.hobbies. It's sad, but there's no way out of it. Worse than that, nested records get pretty ugly:

7> NestedBot = #robot{details=#robot{name="erNest"}}.

#robot{name = undefined,type = industrial,

hobbies = undefined,

details = #robot{name = "erNest",type = industrial,

hobbies = undefined,details = []}}

8> (NestedBot#robot.details)#robot.name.

"erNest"

And yes, the parentheses are mandatory.

Update:

Starting with revision R14A, it is now possible to nest records without the parentheses. The NestedBot example above could also be written as NestedRobot#robot.details#robot.name and work the same.

To further show the dependence of records on tuples, see the following:

9> #robot.type.

3

What this outputs is which element of the underlying tuple it is.

One saving feature of records is the possibility to use them in function heads to pattern match and also in guards. Declare a new record as follows on top of the file, and then add the functions under:

-record(user, {id, name, group, age}).

%% use pattern matching to filter

admin_panel(#user{name=Name, group=admin}) ->

Name ++ " is allowed!";

admin_panel(#user{name=Name}) ->

Name ++ " is not allowed".

%% can extend user without problem

adult_section(U = #user{}) when U#user.age >= 18 ->

%% Show stuff that can't be written in such a text

allowed;

adult_section(_) ->

%% redirect to sesame street site

forbidden.

The syntax to bind a variable to any field of a record is demonstrated in the admin_panel/1 function (it's possible to bind variables to more than one field). An important thing to note about the adult_section/1 function is that you need to do SomeVar = #some_record{} in order to bind the whole record to a variable. Then we do the compiling as usual:

10> c(records).

{ok,records}

11> rr(records).

[robot,user]

12> records:admin_panel(#user{id=1, name="ferd", group=admin, age=96}).

"ferd is allowed!"

13> records:admin_panel(#user{id=2, name="you", group=users, age=66}).

"you is not allowed"

14> records:adult_section(#user{id=21, name="Bill", group=users, age=72}).

allowed

15> records:adult_section(#user{id=22, name="Noah", group=users, age=13}).

forbidden

What this lets us see is how it is not necessary to match on all parts of the tuple or even know how many there are when writing the function: we can only match on the age or the group if that's what's needed and forget about all the rest of the structure. If we were to use a normal tuple, the function definition might need to look a bit like function({record, _, _, ICareAboutThis, _, _) -> .... Then, whenever someone decides to add an element to the tuple, someone else (probably angry about it all) would need to go around and update all functions where that tuple is used.

The following function illustrates how to update a record (they wouldn't be very useful otherwise):

repairman(Rob) ->

Details = Rob#robot.details,

NewRob = Rob#robot{details=["Repaired by repairman"|Details]},

{repaired, NewRob}.

And then:

16> c(records).

{ok,records}

17> records:repairman(#robot{name="Ulbert", hobbies=["trying to have feelings"]}).

{repaired,#robot{name = "Ulbert",type = industrial,

hobbies = ["trying to have feelings"],

details = ["Repaired by repairman"]}}

And you can see my robot has been repaired. The syntax to update records is a bit special here. It looks like we're updating the record in place (Rob#robot{Field=NewValue}) but it's all compiler trickery to call the underlying erlang:setelement/3 function.

One last thing about records. Because they're pretty useful and code duplication is annoying, Erlang programmers frequently share records across modules with the help of header files. Erlang header files are pretty similar to their C counter-part: they're nothing but a snippet of code that gets added to the module as if it were written there in the first place. Create a file named records.hrl with the following content:

%% this is a .hrl (header) file.

-record(included, {some_field,

some_default = "yeah!",

unimaginative_name}).

To include it in records.erl, just add the following line to the module:

-include("records.hrl").

And then the following function to try it:

included() -> #included{some_field="Some value"}.

Now, try it as usual:

18> c(records).

{ok,records}

19> rr(records).

[included,robot,user]

20> records:included().

#included{some_field = "Some value",some_default = "yeah!",

unimaginative_name = undefined}

Hooray! That's about it for records; they're ugly but useful. Their syntax is not pretty, they're not much but a hack, but they're relatively important for the maintainability of your code.

Key-Value Stores

I've had you build a tree back a few chapters, and the use was to use it as a key-value store for an address book. That book sucked: we couldn't delete or convert it to anything useful. It was a good demonstration of recursion, but not much more. Now is the time to introduce you to a bunch of useful data structures and modules to store data under a certain key. I won't define what every function does nor go through all the modules. I will simply link to the doc pages. Consider me as someone responsible about 'raising awareness about key-value stores in Erlang' or something. Sounds like a good title. I just need one of these ribbons.

For small amounts of data, there are basically two data structures that can be used. The first one is called a proplist. A proplist is any list of tuples of the form [{Key,Value}]. They're a weird kind of structure because there is no other rule than that. In fact the rules are so relax than the list can also contain boolean values, integers and whatever you want. We're rather interested by the idea of a tuple with a key and a value in a list here, though. To work with proplists, you can use the proplists module. It contains functions such as proplists:delete/2, proplists:get_value/2, proplists:get_all_values/2, proplists:lookup/2 and proplists:lookup_all/2.

You'll notice there is no function to add or update an element of the list. This shows how loosely defined proplists are as a data structure. To get these functionalities, you must cons your element manually ([NewElement|OldList]) and use functions such as lists:keyreplace/4. Using two modules for one small data structure is not the cleanest thing, but because proplists are so loosely defined, they're often used to deal with configuration lists, and general description of a given item. Proplists are not exactly complete data structures. They're more of a common pattern that appears when using lists and tuples to represent some object or item; the proplists module is a bit of a toolbox over such a pattern.

If you do want a more complete key-value store for small amounts of data, the orddict module is what you need. Orddicts (ordered dictionaries) are proplists with a taste for formality. Each key can be there once, the whole list is sorted for faster average lookup, etc. Common functions for the CRUD usage include orddict:store/3, orddict:find/2 (when you do not know whether they key is in the dictionaries), orddict:fetch/2 (when you know it is there or that it must be there) and orddict:erase/2.

Orddicts are a generally good compromise between complexity and efficiency up to about 75 elements (see my benchmark). After that amount, you should switch to different key-value stores.

There are basically two key-value structures/modules to deal with larger amounts of data: dicts and gb_trees. Dictionaries have the same interface as orddicts: dict:store/3, dict:find/2, dict:fetch/2, dict:erase/2 and every other function, such as dict:map/2 and dict:fold/2 (pretty useful to work on the whole data structure!) Dicts are thus very good choices to scale orddicts up whenever it is needed.

General Balanced Trees, on the other hand, have a bunch more functions leaving you more direct control over how the structure is to be used. There are basically two modes for gb_trees: the mode where you know your structure in and out (I call this the 'smart mode', and the mode where you can't assume much about it (I call this one the 'naive mode'). In naive mode, the functions are gb_trees:enter/2, gb_trees:lookup/2 and gb_trees:delete_any/2. The related smart functions are gb_trees:insert/3, gb_trees:get/2, gb_trees:update/3 and gb_trees:delete/2. There is also gb_trees:map/2, which is always a nice thing when you need it.

The disadvantage of 'naive' functions over 'smart' ones is that because gb_trees are balanced trees, whenever you insert a new element (or delete a bunch), it might be possible that the tree will need to balance itself. This can take time and memory (even in useless checks just to make sure). The 'smart' function all assume that the key is present in the tree: this lets you skip all the safety checks and results in faster times.

When should you use gb_trees over dicts? Well, it's not a clear decision. As the keyval_benchmark.erl benchmark module I have written will show, gb_trees and dicts have somewhat similar performances in many respects. However, the benchmark demonstrates that dicts have the best read speeds while the gb_trees tend to be a little quicker on other operations. You can judge based on your own needs which one would be the best.

Oh and also note that while dicts have a fold function, gb_trees don't: they instead have an iterator function, which returns a bit of the tree on which you can call gb_trees:next(Iterator) to get the following values in order. What this means is that you need to write your own recursive functions on top of gb_trees rather than use a generic fold. On the other hand, gb_trees let you have quick access to the smallest and largest elements of the structure with gb_trees:smallest/1 and gb_trees:largest/1.

I would therefore say that your application's needs is what should govern which key-value store to choose. Different factors such as how much data you've got to store, what you need to do with it and whatnot all have their importance. Measure, profile and benchmark to make sure.

Note: some special key-value stores exist to deal with resources of different size. Such stores are ETS tables, DETS tables and the mnesia database. However, their use is strongly related to the concepts of multiple processes and distribution. Because of this, they'll only be approached later on. I'm leaving this as a reference to pique your curiosity and for those interested.

Arrays

But what about code that requires data structures with nothing but numeric keys? Well for that, there are arrays. They allow you to access elements with numerical indices and to fold over the whole structure while possibly ignoring undefined slots.

Don't drink too much kool-aid:

Erlang arrays, at the opposite of their imperative counterparts, are not able to have such things as constant-time insertion or lookup. Because they're usually slower than those in languages which support destructive assignment and that the style of programming done with Erlang doesn't necessary lend itself too well to arrays and matrices, they are rarely used in practice.

Generally, Erlang programmers who need to do matrix manipulations and other uses requiring arrays tend to use concepts called Ports to let other languages do the heavy lifting, or C-Nodes, Linked in drivers and NIFs (Experimental, R13B03+).

Arrays are also weird in the sense that they're one of the few data structures to be 0-indexed (at the opposite of tuples or lists), along with indexing in the regular expressions module. Be careful with them.

A Set of Sets

If you've ever studied set theory in whatever mathematics class you have an idea about what sets can do. If you haven't, you might want to skip over this. However, I'll just say that sets are groups of unique elements that you can compare and operate on: find which elements are in two groups, in none of them, only in one or the other, etc. There are more advanced operations letting you define relations and operate on these relations and much more. I'm not going to dive into the theory (again, it's out of the scope of this book) so I'll just describe them as it is.

There are 4 main modules to deal with sets in Erlang. This is a bit weird at first, but it makes more sense once you realize that it's because it was agreed by implementers that there was no 'best' way to build a set. The four modules are ordsets, sets, gb_sets and sofs (sets of sets):

ordsets

Ordsets are implemented as a sorted list. They're mainly useful for small sets, are the slowest kind of set, but they have the simplest and most readable representation of all sets. There are standard functions for them such as ordsets:new/0, ordsets:is_element/2, ordsets:add_element/2, ordsets:del_element/2, ordsets:union/1, ordsets:intersection/1, and a bunch more.

sets

Sets (the module) is implemented on top of a structure really similar to the one used in dict. They implement the same interface as ordsets, but they're going to scale much better. Like dictionaries, they're especially good for read-intensive manipulations, like checking whether some element is part of the set or not.

gb_sets

Gb_sets themselves are constructed above a General Balanced Tree structure similar to the one used in the gb_trees module. gb_sets are to sets what gb_tree is to dict; an implementation that is faster when considering operations different than reading, leaving you with more control. While gb_sets implement the same interface as sets and ordsets, they also add more functions. Like gb_trees, you have smart vs. naive functions, iterators, quick access to the smallest and largest values, etc.

sofs

Sets of sets (sofs) are implemented with sorted lists, stuck inside a tuple with some metadata. They're the module to use if you want to have full control over relationships between sets, families, enforce set types, etc. They're really what you want if you need mathematics concept rather than 'just' groups of unique elements.

Don't drink too much kool-aid:

While such a variety can be seen as something great, some implementation details can be downright frustrating. As an example, gb_sets, ordsets and sofs all use the == operator to compare values: if you have the numbers 2 and 2.0, they'll both end up seen as the same one.

However, sets (the module) uses the =:= operator, which means you can't necessarily switch over every implementation as you wish. There are cases where you need one precise behavior and at that point, you might lose the benefit of having multiple implementations.

It's a bit confusing to have that many options available. Björn Gustavsson, from the Erlang/OTP team and programmer of Wings3D mainly suggests using gb_sets in most circumstances, using orddict when you need a clear representation that you want to process with your own code and 'sets' when you need the =:= operator (source.)

In any case, like for key-value stores, the best solution is usually to benchmark and see what fits your application better.

Directed Graphs

There is one other data structure that I want to mention here (not that there are not more than what's mentioned in this chapter, on the contrary): directed graphs. Again, this data structure is more for readers who already know the mathematical theory that goes with it.

Directed graphs in Erlang are implemented as two modules, digraph and digraph_utils. The digraph module basically allows the construction and modification of a directed graph: manipulating edges and vertices, finding paths and cycles, etc. On the other hand, digraph_utils allows you to navigate a graph (postorder, preorder), testing for cycles, arborescences or trees, finding neighbors, and so on.

Because directed graphs are closely related to set theory, the 'sofs' module contains a few functions letting you convert families to digraphs and digraphs to families.

Queues

The queue module implements a double-ended FIFO (First In, First Out) queue:

They're implemented a bit as illustrated above: two lists (in this context, stacks) that allow to both append and prepend elements rapidly.

The queue module basically has different functions in a mental separation into 3 interfaces (or APIs) of varying complexity, called 'Original API', 'Extended API' and 'Okasaki API':

Original API

The original API contains the functions at the base of the queue concept, including: new/0, for creating empty queues, in/2, for inserting new elements, out/1, for removing elements, and then functions to convert to lists, reverse the queue, look if a particular value is part of it, etc.

Extended API

The extended API mainly adds some introspection power and flexibility: it lets you do things such as looking at the front of the queue without removing the first element (see get/1 or peek/1), removing elements without caring about them (drop/1), etc. These functions are not essential to the concept of queues, but they're still useful in general.

Okasaki API

The Okasaki API is a bit weird. It's derived from Chris Okasaki's Purely Functional Data Structures. The API provides operations similar to what was available in the two previous APIs, but some of the function names are written backwards and the whole thing is relatively peculiar. Unless you do know you want this API, I wouldn't bother with it.

You'll generally want to use queues when you'll need to ensure that the first item ordered is indeed the first one processed. So far, the examples I've shown mainly used lists as a accumulators that would then be reversed. In cases where you can't just do all the reversing at once and elements are frequently added, the queue module is what you want (well, you should test and measure first! Always test and measure first!)

End of the short visit

That's about it for the data structures trip of Erlang. Thank you for having kept your arms inside the vehicles the whole time. Of course, there are a few more data structures available than that to solve different problems. I've only covered those that you're likely to encounter or need the most given the strengths of general use cases of Erlang. I encourage you to explore the standard library and the extended one too to find more information.

You might be glad to learn that this completes our trip into sequential (functional) Erlang. I know a lot of people get in Erlang to see all the concurrency and processes and whatnot. It's understandable, given it's really where Erlang shines. Supervision trees, fancy error management, distribution, and more. I know I've been very impatient to write about these subjects, so I guess some readers were very impatient to read about them.

However, I judged it made more sense to be comfortable with functional Erlang before moving on to concurrent Erlang. It will be easier to move on afterwards and focus on all the new concepts. Here we go!

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

The Hitchhiker's Guide to Concurrency

Far out in the uncharted backwaters of the unfashionable beginning of the 21st century lies a small subset of human knowledge.

Within this subset of human knowledge is an utterly insignificant little discipline whose Von Neumann-descended architecture is so amazingly primitive that it is still thought that RPN calculators are a pretty neat idea.

This discipline has — or rather had — a problem, which was this: most of the people studying it were unhappy for pretty much of the time when trying to write parallel software. Many solutions were suggested for this problem, but most of these were largely concerned with the handling of little pieces of logic called locks and mutexes and whatnot, which is odd because on the whole it wasn't the small pieces of logic that needed parallelism.

And so the problem remained; lots of people were mean, and most of them were miserable, even those with RPN calculators.

Many were increasingly of the opinion that they'd all made a big mistake in trying to add parallelism to their programming languages, and that no program should have ever left its initial thread.

Note: parodying The Hitchhiker's Guide to the Galaxy is fun. Read the book if you haven't already. It's good!

Don't Panic

Hi. Today (or whatever day you are reading this, even tomorrow), I'm going to tell you about concurrent Erlang. Chances are you've read about or dealt with concurrency before. You might also be curious about the emergence of multi-core programming. Anyway, the probabilities are high that you're reading this book because of all this talk about concurrency going on these days.

A warning though; this chapter is mostly theoric. If you have a headache, a distaste for programming language history or just want to program, you might be better off skipping to the end of the chapter or skip to the next one (where more practical knowledge is shown.)

I've already explained in the book's intro that Erlang's concurrency was based on message passing and the actor model, with the example of people communicating with nothing but letters. I'll explain it more in details again later, but first of all, I believe it is important to define the difference between concurrency and parallelism.

In many places both words refer to the same concept. They are often used as two different ideas in the context of Erlang. For many Erlangers, concurrency refers to the idea of having many actors running independently, but not necessarily all at the same time. Parallelism is having actors running exactly at the same time. I will say that there doesn't seem to be any consensus on such definitions around various areas of computer science, but I will use them in this manner in this text. Don't be surprised if other sources or people use the same terms to mean different things.

This is to say Erlang had concurrency from the beginning, even when everything was done on a single core processor in the '80s. Each Erlang process would have its own slice of time to run, much like desktop applications did before multi-core systems.

Parallelism was still possible back then; all you needed to do was to have a second computer running the code and communicating with the first one. Even then, only two actors could be run in parallel in this setup. Nowadays, multi-core systems allows for parallelism on a single computer (with some industrial chips having many dozens of cores) and Erlang takes full advantage of this possibility.

Don't drink too much Kool-Aid:

The distinction between concurrency and parallelism is important to make, because many programmers hold the belief that Erlang was ready for multi-core computers years before it actually was. Erlang was only adapted to true symmetric multiprocessing in the mid 2000s and only got most of the implementation right with the R13B release of the language in 2009. Before that, SMP often had to be disabled to avoid performance losses. To get parallelism on a multicore computer without SMP, you'd start many instances of the VM instead.

An interesting fact is that because Erlang concurrency is all about isolated processes, it took no conceptual change at the language level to bring true parallelism to the language. All the changes were transparently done in the VM, away from the eyes of the programmers.

Concepts of Concurrency

Back in the day, Erlang's development as a language was extremely quick with frequent feedback from engineers working on telephone switches in Erlang itself. These interactions proved processes-based concurrency and asynchronous message passing to be a good way to model the problems they faced. Moreover, the telephony world already had a certain culture going towards concurrency before Erlang came to be. This was inherited from PLEX, a language created earlier at Ericsson, and AXE, a switch developed with it. Erlang followed this tendency and attempted to improve on previous tools available.

Erlang had a few requirements to satisfy before being considered good. The main ones were being able to scale up and support many thousands of users across many switches, and then to achieve high reliability—to the point of never stopping the code.

Scalability

I'll focus on the scaling first. Some properties were seen as necessary to achieve scalability. Because users would be represented as processes which only reacted upon certain events (i.e.: receiving a call, hanging up, etc.), an ideal system would support processes doing small computations, switching between them very quickly as events came through. To make it efficient, it made sense for processes to be started very quickly, to be destroyed very quickly and to be able to switch them really fast. Having them lightweight was mandatory to achieve this. It was also mandatory because you didn't want to have things like process pools (a fixed amount of processes you split the work between.) Instead, it would be much easier to design programs that could use as many processes as they need.

Another important aspect of scalability is to be able to bypass your hardware's limitations. There are two ways to do this: make the hardware better, or add more hardware. The first option is useful up to a certain point, after which it becomes extremely expensive (i.e.: buying a super computer). The second option is usually cheaper and requires you to add more computers to do the job. This is where distribution can be useful to have as a part of your language.

Anyway, to get back to small processes, because telephony applications needed a lot of reliability, it was decided that the cleanest way to do things was to forbid processes from sharing memory. Shared memory could leave things in an inconsistent state after some crashes (especially on data shared across different nodes) and had some complications. Instead, processes should communicate by sending messages where all the data is copied. This would risk being slower, but safer.

Fault-tolerance

This leads us on the second type of requirements for Erlang: reliability. The first writers of Erlang always kept in mind that failure is common. You can try to prevent bugs all you want, but most of the time some of them will still happen. In the eventuality bugs don't happen, nothing can stop hardware failures all the time. The idea is thus to find good ways to handle errors and problems rather than trying to prevent them all.

It turns out that taking the design approach of multiple processes with message passing was a good idea, because error handling could be grafted onto it relatively easily. Take lightweight processes (made for quick restarts and shutdowns) as an example. Some studies proved that the main sources of downtime in large scale software systems are intermittent or transient bugs (source). Then, there's a principle that says that errors which corrupt data should cause the faulty part of the system to die as fast as possible in order to avoid propagating errors and bad data to the rest of the system. Another concept here is that there exist many different ways for a system to terminate, two of which are clean shutdowns and crashes (terminating with an unexpected error).

Here the worst case is obviously the crash. A safe solution would be to make sure all crashes are the same as clean shutdowns: this can be done through practices such as shared-nothing and single assignment (which isolates a process' memory), avoiding locks (a lock could happen to not be unlocked during a crash, keeping other processes from accessing the data or leaving data in an inconsistent state) and other stuff I won't cover more, but were all part of Erlang's design. Your ideal solution in Erlang is thus to kill processes as fast as possible to avoid data corruption and transient bugs. Lightweight processes are a key element in this. Further error handling mechanisms are also part of the language to allow processes to monitor other processes (which are described in the Errors and Processes chapter), in order to know when processes die and to decide what to do about it.

Supposing restarting processes real fast is enough to deal with crashes, the next problem you get is hardware failures. How do you make sure your program keeps running when someone kicks the computer it's running on? Although a fancy defense mechanism comprising laser detection and strategically placed cacti could do the job for a while, it would not last forever. The hint is simply to have your program running on more than one computer at once, something that was needed for scaling anyway. This is another advantage of independent processes with no communication channel outside message passing. You can have them working the same way whether they're local or on a different computer, making fault tolerance through distribution nearly transparent to the programmer.

Being distributed has direct consequences on how processes can communicate with each other. One of the biggest hurdles of distribution is that you can't assume that because a node (a remote computer) was there when you made a function call, it will still be there for the whole transmission of the call or that it will even execute it correctly. Someone tripping over a cable or unplugging the machine would leave your application hanging. Or maybe it would make it crash. Who knows?

Well it turns out the choice of asynchronous message passing was a good design pick there too. Under the processes-with-asynchronous-messages model, messages are sent from one process to a second one and stored in a mailbox inside the receiving process until they are taken out to be read. It's important to mention that messages are sent without even checking if the receiving process exists or not because it would not be useful to do so. As implied in the previous paragraph, it's impossible to know if a process will crash between the time a message is sent and received. And if it's received, it's impossible to know if it will be acted upon or again if the receiving process will die before that. Asynchronous messages allow safe remote function calls because there is no assumption about what will happen; the programmer is the one to know. If you need to have a confirmation of delivery, you have to send a second message as a reply to the original process. This message will have the same safe semantics, and so will any program or library you build on this principle.

Implementation

Alright, so it was decided that lightweight processes with asynchronous message passing were the approach to take for Erlang. How to make this work? Well, first of all, the operating system can't be trusted to handle the processes. Operating systems have many different ways to handle processes, and their performance varies a lot. Most if not all of them are too slow or too heavy for what is needed by standard Erlang applications. By doing this in the VM, the Erlang implementers keep control of optimization and reliability. Nowadays, Erlang's processes take about 300 words of memory each and can be created in a matter of microseconds—not something doable on major operating systems these days.

To handle all these potential processes your programs could create, the VM starts one thread per core which acts as a scheduler. Each of these schedulers has a run queue, or a list of Erlang processes on which to spend a slice of time. When one of the schedulers has too many tasks in its run queue, some are migrated to another one. This is to say each Erlang VM takes care of doing all the load-balancing and the programmer doesn't need to worry about it. There are some other optimizations that are done, such as limiting the rate at which messages can be sent on overloaded processes in order to regulate and distribute the load.

All the hard stuff is in there, managed for you. That is what makes it easy to go parallel with Erlang. Going parallel means your program should go twice as fast if you add a second core, four times faster if there are 4 more and so on, right? It depends. Such a phenomenon is named linear scaling in relation to speed gain vs. the number of cores or processors (see the graph below.) In real life, there is no such thing as a free lunch (well, there are at funerals, but someone still has to pay, somewhere).

Not Entirely Unlike Linear Scaling

The difficulty of obtaining linear scaling is not due to the language itself, but rather to the nature of the problems to solve. Problems that scale very well are often said to be embarrassingly parallel. If you look for embarrassingly parallel problems on the Internet, you're likely to find examples such as ray-tracing (a method to create 3D images), brute-forcing searches in cryptography, weather prediction, etc.

From time to time, people then pop up in IRC channels, forums or mailing lists asking if Erlang could be used to solve that kind of problem, or if it could be used to program on a GPU. The answer is almost always 'no'. The reason is relatively simple: all these problems are usually about numerical algorithms with lots of data crunching. Erlang is not very good at this.

Erlang's embarrassingly parallel problems are present at a higher level. Usually, they have to do with concepts such as chat servers, phone switches, web servers, message queues, web crawlers or any other application where the work done can be represented as independent logical entities (actors, anyone?). This kind of problem can be solved efficiently with close-to-linear scaling.

Many problems will never show such scaling properties. In fact, you only need one centralized sequence of operations to lose it all. Your parallel program only goes as fast as its slowest sequential part. An example of that phenomenon is observable any time you go to a mall. Hundreds of people can be shopping at once, rarely interfering with each other. Then once it's time to pay, queues form as soon as there are fewer cashiers than there are customers ready to leave.

It would be possible to add cashiers until there's one for each customer, but then you would need a door for each customer because they couldn't get inside or outside the mall all at once.

To put this another way, even though customers could pick each of their items in parallel and basically take as much time to shop whether they're alone or a thousand in the store, they would still have to wait to pay. Therefore their shopping experience can never be shorter than the time it takes them to wait in the queue and pay.

A generalisation of this principle is called Amdahl's Law. It indicates how much of a speedup you can expect your system to have whenever you add parallelism to it, and in what proportion:

According to Amdahl's law, code that is 50% parallel can never get faster than twice what it was before, and code that is 95% parallel can theoretically be expected to be about 20 times faster if you add enough processors. What's interesting to see on this graph is how getting rid of the last few sequential parts of a program allows a relatively huge theoretical speedup compared to removing as much sequential code in a program that is not very parallel to begin with.

Don't drink too much Kool-Aid:

Parallelism is not the answer to every problem. In some cases, going parallel will even slow down your application. This can happen whenever your program is 100% sequential, but still uses multiple processes.

One of the best examples of this is the ring benchmark. A ring benchmark is a test where many thousands of processes will pass a piece of data to one after the other in a circular manner. Think of it as a game of telephone if you want. In this benchmark, only one process at a time does something useful, but the Erlang VM still spends time distributing the load accross cores and giving every process its share of time.

This plays against many common hardware optimizations and makes the VM spend time doing useless stuff. This often makes purely sequential applications run much slower on many cores than on a single one. In this case, disabling symmetric multiprocessing ($ erl -smp disable) might be a good idea.

So long and thanks for all the fish!

Of course, this chapter would not be complete if it wouldn't show the three primitives required for concurrency in Erlang: spawning new processes, sending messages, and receiving messages. In practice there are more mechanisms required for making really reliable applications, but for now this will suffice.

I've skipped around the issue a whole lot and I have yet to explain what a process really is. It's in fact nothing but a function. That's it. It runs a function and once it's done, it disappears. Technically, a process also has some hidden state (such as a mailbox for messages), but functions are enough for now.

To start a new process, Erlang provides the function spawn/1, which takes a single function and runs it:

1> F = fun() -> 2 + 2 end.

#Fun<erl_eval.20.67289768>

2> spawn(F).

<0.44.0>

The result of spawn/1 (<0.44.0>) is called a Process Identifier, often just written PID, Pid, or pid by the community. The process identifier is an arbitrary value representing any process that exists (or might have existed) at some point in the VM's life. It is used as an address to communicate with the process.

You'll notice that we can't see the result of the function F. We only get its pid. That's because processes do not return anything.

How can we see the result of F then? Well, there are two ways. The easiest one is to just output whatever we get:

3> spawn(fun() -> io:format("~p~n",[2 + 2]) end).

4

<0.46.0>

This isn't practical for a real program, but it is useful for seeing how Erlang dispatches processes. Fortunately, using io:format/2 is enough to let us experiment. We'll start 10 processes real quick and pause each of them for a while with the help of the function timer:sleep/1, which takes an integer value N and waits for N milliseconds before resuming code. After the delay, the value present in the process is output.

4> G = fun(X) -> timer:sleep(10), io:format("~p~n", [X]) end.

#Fun<erl_eval.6.13229925>

5> [spawn(fun() -> G(X) end) || X <- lists:seq(1,10)].

[<0.273.0>,<0.274.0>,<0.275.0>,<0.276.0>,<0.277.0>,

<0.278.0>,<0.279.0>,<0.280.0>,<0.281.0>,<0.282.0>]

2

1

4

3

5

8

7

6

10

9

The order doesn't make sense. Welcome to parallelism. Because the processes are running at the same time, the ordering of events isn't guaranteed anymore. That's because the Erlang VM uses many tricks to decide when to run a process or another one, making sure each gets a good share of time. Many Erlang services are implemented as processes, including the shell you're typing in. Your processes must be balanced with those the system itself needs and this might be the cause of the weird ordering.

Note: the results are similar whether symmetric multiprocessing is enabled or not. To prove it, you can just test it out by starting the Erlang VM with $ erl -smp disable.

To see if your Erlang VM runs with or without SMP support in the first place, start a new VM without any options and look for the first line output. If you can spot the text [smp:2:2] [rq:2], it means you're running with SMP enabled, and that you have 2 run queues (rq, or schedulers) running on two cores. If you only see [rq:1], it means you're running with SMP disabled.

If you wanted to know, [smp:2:2] means there are two cores available, with two schedulers. [rq:2] means there are two run queues active. In earlier versions of Erlang, you could have multiple schedulers, but with only one shared run queue. Since R13B, there is one run queue per scheduler by default; this allows for better parallelism.

To prove the shell itself is implemented as a regular process, I'll use the BIF self/0, which returns the pid of the current process:

6> self().

<0.41.0>

7> exit(self()).

** exception exit: <0.41.0>

8> self().

<0.285.0>

And the pid changes because the process has been restarted. The details of how this works will be seen later. For now, there's more basic stuff to cover. The most important one right now is to figure out how to send messages around, because nobody wants to be stuck with outputting the resulting values of processes all the time, and then entering them by hand in other processes (at least I know I don't.)

The next primitive required to do message passing is the operator !, also known as the bang symbol. On the left-hand side it takes a pid and on the right-hand side it takes any Erlang term. The term is then sent to the process represented by the pid, which can access it:

9> self() ! hello.

hello

The message has been put in the process' mailbox, but it hasn't been read yet. The second hello shown here is the return value of the send operation. This means it is possible to send the same message to many processes by doing:

10> self() ! self() ! double.

double

Which is equivalent to self() ! (self() ! double). A thing to note about a process' mailbox is that the messages are kept in the order they are received. Every time a message is read it is taken out the mailbox. Again, this is a bit similar to the introduction's example with people writing letters.

To see the contents of the current mailbox, you can use the flush() command while in the shell:

11> flush().

Shell got hello

Shell got double

Shell got double

ok

This function is just a shortcut that outputs received messages. This means we still can't bind the result of a process to a variable, but at least we know how to send it from a process to another one and check if it's been received.

Sending messages that nobody will read is as useful as writing emo poetry; not a whole lot. This is why we need the receive statement. Rather than playing for too long in the shell, we'll write a short program about dolphins to learn about it:

-module(dolphins).

-compile(export_all).

dolphin1() ->

receive

do_a_flip ->

io:format("How about no?~n");

fish ->

io:format("So long and thanks for all the fish!~n");

_ ->

io:format("Heh, we're smarter than you humans.~n")

end.

As you can see, receive is syntactically similar to case ... of. In fact, the patterns work exactly the same way except they bind variables coming from messages rather than the expression between case and of. Receives can also have guards:

receive

Pattern1 when Guard1 -> Expr1;

Pattern2 when Guard2 -> Expr2;

Pattern3 -> Expr3

end

We can now compile the above module, run it, and start communicating with dolphins:

11> c(dolphins).

{ok,dolphins}

12> Dolphin = spawn(dolphins, dolphin1, []).

<0.40.0>

13> Dolphin ! "oh, hello dolphin!".

Heh, we're smarter than you humans.

"oh, hello dolphin!"

14> Dolphin ! fish.

fish

15>

Here we introduce a new way of spawning with spawn/3. Rather than taking a single function, spawn/3 takes the module, function and its arguments as its own arguments. Once the function is running, the following events take place:

1. The function hits the receive statement. Given the process' mailbox is empty, our dolphin waits until it gets a message;

2. The message "oh, hello dolphin!" is received. The function tries to pattern match against do_a_flip. This fails, and so the pattern fish is tried and also fails. Finally, the message meets the catch-all clause (_) and matches.

3. The process outputs the message "Heh, we're smarter than you humans."

Then it should be noted that if the first message we sent worked, the second provoked no reaction whatsoever from the process <0.40.0>. This is due to the fact once our function output "Heh, we're smarter than you humans.", it terminated and so did the process. We'll need to restart the dolphin:

8> f(Dolphin).

ok

9> Dolphin = spawn(dolphins, dolphin1, []).

<0.53.0>

10> Dolphin ! fish.

So long and thanks for all the fish!

fish

And this time the fish message works. Wouldn't it be useful to be able to receive a reply from the dolphin rather than having to use io:format/2? Of course it would (why am I even asking?) I've mentioned earlier in this chapter that the only manner to know if a process had received a message is to send a reply. Our dolphin process will need to know who to reply to. This works like it does with the postal service. If we want someone to know answer our letter, we need to add our address. In Erlang terms, this is done by packaging a process' pid in a tuple. The end result is a message that looks a bit like {Pid, Message}. Let's create a new dolphin function that will accept such messages:

dolphin2() ->

receive

{From, do_a_flip} ->

From ! "How about no?";

{From, fish} ->

From ! "So long and thanks for all the fish!";

_ ->

io:format("Heh, we're smarter than you humans.~n")

end.

As you can see, rather than accepting do_a_flip and fish for messages, we now require a variable From. That's where the process identifier will go.

11> c(dolphins).

{ok,dolphins}

12> Dolphin2 = spawn(dolphins, dolphin2, []).

<0.65.0>

13> Dolphin2 ! {self(), do_a_flip}.

{<0.32.0>,do_a_flip}

14> flush().

Shell got "How about no?"

ok

It seems to work pretty well. We can receive replies to messages we sent (we need to add an address to each message), but we still need to start a new process for each call. Recursion is the way to solve this problem. We just need the function to call itself so it never ends and always expects more messages. Here's a function dolphin3/0 that puts this in practice:

dolphin3() ->

receive

{From, do_a_flip} ->

From ! "How about no?",

dolphin3();

{From, fish} ->

From ! "So long and thanks for all the fish!";

_ ->

io:format("Heh, we're smarter than you humans.~n"),

dolphin3()

end.

Here the catch-all clause and the do_a_flip clause both loop with the help of dolphin/3. As long as only these messages are sent, the dolphin process will loop indefinitely. However, if we send the fish message, the process will stop:

15> Dolphin3 = spawn(dolphins, dolphin3, []).

<0.75.0>

16> Dolphin3 ! Dolphin3 ! {self(), do_a_flip}.

{<0.32.0>,do_a_flip}

17> flush().

Shell got "How about no?"

Shell got "How about no?"

ok

18> Dolphin3 ! {self(), unknown_message}.

Heh, we're smarter than you humans.

{<0.32.0>,unknown_message}

19> Dolphin3 ! Dolphin3 ! {self(), fish}.

{<0.32.0>,fish}

20> flush().

Shell got "So long and thanks for all the fish!"

ok

And that should be it for dolphins.erl. As you see, it does respect our expected behavior of replying once for every message and keep going afterwards, except for the fish call. The dolphin got fed up with our crazy human antics and left us for good.

There you have it. This is the core of all of Erlang's concurrency. We've seen processes and basic message passing. There are more concepts to see in order to make truly useful and reliable programs. We'll see some of them in the next chapter, and more in the chapters after that.

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Errors and Processes

Links

A link is a specific kind of relationship that can be created between two processes. When that relationship is set up and one of the processes dies from an unexpected throw, error or exit (see Errors and Exceptions), the other linked process also dies.

This is a useful concept from the perspective of failing as soon as possible to stop errors: if the process that has an error crashes but those that depend on it don't, then all these depending processes now have to deal with a dependency disappearing. Letting them die and then restarting the whole group is usually an acceptable alternative. Links let us do exactly this.

To set a link between two processes, Erlang has the primitive function link/1, which takes a Pid as an argument. When called, the function will create a link between the current process and the one identified by Pid. To get rid of a link, use unlink/1. When one of the linked process crashes, a special kind of message is sent, with information relative to what happened. No such message is sent if the process dies of natural causes (read: is done running its functions.) I'll first introduce this new function as part of linkmon.erl:

myproc() ->

timer:sleep(5000),

exit(reason).

If you try the next following calls (and wait 5 seconds between each spawn command), you should see the shell crashing for 'reason' only when a link has been set between the two processes.

1> c(linkmon).

{ok,linkmon}

2> spawn(fun linkmon:myproc/0).

<0.52.0>

3> link(spawn(fun linkmon:myproc/0)).

true

** exception error: reason

Or, to put it in picture:

However, this {'EXIT', B, Reason} message can not be caught with a try ... catch as usual. Other mechanisms need to be used to do this. We'll see them later.

It's important to note that links are used to establish larger groups of processes that should all die together:

chain(0) ->

receive

_ -> ok

after 2000 ->

exit("chain dies here")

end;

chain(N) ->

Pid = spawn(fun() -> chain(N-1) end),

link(Pid),

receive

_ -> ok

end.

This function will take an integer N, start N processes linked one to the other. In order to be able to pass the N-1 argument to the next 'chain' process (which calls spawn/1), I wrap the call inside an anonymous function so it doesn't need arguments anymore. Calling spawn(?MODULE, chain, [N-1]) would have done a similar job.

Here, I'll have many processes linked together, dying as each of their successors exits:

4> c(linkmon).

{ok,linkmon}

5> link(spawn(linkmon, chain, [3])).

true

** exception error: "chain dies here"

And as you can see, the shell does receive the death signal from some other process. Here's a drawn representation of the spawned processes and links going down:

[shell] == [3] == [2] == [1] == [0]

[shell] == [3] == [2] == [1] == *dead*

[shell] == [3] == [2] == *dead*

[shell] == [3] == *dead*

[shell] == *dead*

*dead, error message shown*

[shell] <-- restarted

After the process running linkmon:chain(0) dies, the error is propagated down the chain of links until the shell process itself dies because of it. The crash could have happened in any of the linked processes; because links are bidirectional, you only need one of them to die for the others to follow suit.

Note: If you wanted to kill another process from the shell, you could use the function exit/2, which is called this way: exit(Pid, Reason). Try it if you wish.

Note: Links can not be stacked. If you call link/1 15 times for the same two processes, only one link will still exist between them and a single call to unlink/1 will be enough to tear it down.

Its important to note that link(spawn(Function)) or link(spawn(M,F,A)) happens in more than one step. In some cases, it is possible for a process to die before the link has been set up and then provoke unexpected behavior. For this reason, the function spawn_link/1-3 has been added to the language. It takes the same arguments as spawn/1-3, creates a process and links it as if link/1 had been there, except it's all done as an atomic operation (the operations are combined as a single one, which can either fail or succeed, but nothing else). This is generally considered safer and you save a set of parentheses too.

It's a Trap!

Now to get back to links and processes dying. Error propagation across processes is done through a process similar to message passing, but with a special type of message called signals. Exit signals are 'secret' messages that automatically act on processes, killing them in the action.

I have mentioned many times already that in order to be reliable, an application needs to be able to both kill and restart a process quickly. Right now, links are alright to do the killing part. What's missing is the restarting.

In order to do restart a process, we need a way to first know that it died. This can be done by adding a layer on top of links (the delicious frosting on the cake) with a concept called system processes. System processes are basically normal processes, except they can convert exit signals to regular messages. This is done by calling process_flag(trap_exit, true) in a running process. Nothing speaks as much as an example, so we'll go with that. I'll just redo the chain example with a system process at the beginning:

1> process_flag(trap_exit, true).

true

2> spawn_link(fun() -> linkmon:chain(3) end).

<0.49.0>

3> receive X -> X end.

{'EXIT',<0.49.0>,"chain dies here"}

Ah! Now things get interesting. To get back to our drawings, what happens is now more like this:

[shell] == [3] == [2] == [1] == [0]

[shell] == [3] == [2] == [1] == *dead*

[shell] == [3] == [2] == *dead*

[shell] == [3] == *dead*

[shell] <-- {'EXIT,Pid,"chain dies here"} -- *dead*

[shell] <-- still alive!

And this is the mechanism allowing for a quick restart of processes. By writing programs using system processes, it is be easy to create a process whose only role is to check if something dies and then restart it whenever it fails. We'll cover more of this in the next chapter, when we really apply these techniques.

For now, I want to come back to the exception functions seen in the exceptions chapter and show how they behave around processes that trap exits. Let's first set the bases to experiment without a system process. I'll successively show the results of uncaught throws, errors and exits in neighboring processes:

Exception source: spawn_link(fun() -> ok end)

Untrapped Result: - nothing -

Trapped Result: {'EXIT', <0.61.0>, normal}

The process exited normally, without a problem. Note that this looks a bit like the result of catch exit(normal), except a PID is added to the tuple to know what processed failed.

Exception source: spawn_link(fun() -> exit(reason) end)

Untrapped Result: ** exception exit: reason

Trapped Result: {'EXIT', <0.55.0>, reason}

The process has terminated for a custom reason. In this case, if there is no trapped exit, the process crashes. Otherwise, you get the above message.

Exception source: spawn_link(fun() -> exit(normal) end)

Untrapped Result: - nothing -

Trapped Result: {'EXIT', <0.58.0>, normal}

This successfully emulates a process terminating normally. In some cases, you might want to kill a process as part of the normal flow of a program, without anything exceptional going on. This is the way to do it.

Exception source: spawn_link(fun() -> 1/0 end)

Untrapped Result: Error in process <0.44.0> with exit value: {badarith, [{erlang, '/', [1,0]}]}

Trapped Result: {'EXIT', <0.52.0>, {badarith, [{erlang, '/', [1,0]}]}}

The error ({badarith, Reason}) is never caught by a try ... catch block and bubbles up into an 'EXIT'. At this point, it behaves exactly the same as exit(reason) did, but with a stack trace giving more details about what happened.

Exception source: spawn_link(fun() -> erlang:error(reason) end)

Untrapped Result: Error in process <0.47.0> with exit value: {reason, [{erlang, apply, 2}]}

Trapped Result: {'EXIT', <0.74.0>, {reason, [{erlang, apply, 2}]}}

Pretty much the same as with 1/0. That's normal, erlang:error/1 is meant to allow you to do just that.

Exception source: spawn_link(fun() -> throw(rocks) end)

Untrapped Result: Error in process <0.51.0> with exit value: {{nocatch, rocks}, [{erlang, apply, 2}]}

Trapped Result: {'EXIT', <0.79.0>, {{nocatch, rocks}, [{erlang, apply, 2}]}}

Because the throw is never caught by a try ... catch, it bubbles up into an error, which in turn bubbles up into an EXIT. Without trapping exit, the process fails. Otherwise it deals with it fine.

And that's about it for usual exceptions. Things are normal: everything goes fine. Exceptional stuff happens: processes die, different signals are sent around.

Then there's exit/2. This one is the Erlang process equivalent of a gun. It allows a process to kill another one from a distance, safely. Here are some of the possible calls:

Exception source: exit(self(), normal)

Untrapped Result: ** exception exit: normal

Trapped Result: {'EXIT', <0.31.0>, normal}

When not trapping exits, exit(self(), normal) acts the same as exit(normal). Otherwise, you receive a message with the same format you would have had by listening to links from foreign processes dying.

Exception source: exit(spawn_link(fun() -> timer:sleep(50000) end), normal)

Untrapped Result: - nothing -

Trapped Result: - nothing -

This basically is a call to exit(Pid, normal). This command doesn't do anything useful, because a process can not be remotely killed with the reason normal as an argument.

Exception source: exit(spawn_link(fun() -> timer:sleep(50000) end), reason)

Untrapped Result: ** exception exit: reason

Trapped Result: {'EXIT', <0.52.0>, reason}

This is the foreign process terminating for reason itself. Looks the same as if the foreign process called exit(reason) on itself.

Exception source: exit(spawn_link(fun() -> timer:sleep(50000) end), kill)

Untrapped Result: ** exception exit: killed

Trapped Result: {'EXIT', <0.58.0>, killed}

Surprisingly, the message gets changed from the dying process to the spawner. The spawner now receives killed instead of kill. That's because kill is a special exit signal. More details on this later.

Exception source: exit(self(), kill)

Untrapped Result: ** exception exit: killed

Trapped Result: ** exception exit: killed

Oops, look at that. It seems like this one is actually impossible to trap. Let's check something.

Exception source: spawn_link(fun() -> exit(kill) end)

Untrapped Result: ** exception exit: killed

Trapped Result: {'EXIT', <0.67.0>, kill}

Now that's getting confusing. When another process kills itself with exit(kill) and we don't trap exits, our own process dies with the reason killed. However, when we trap exits, things don't happen that way.

While you can trap most exit reasons, there are situations where you might want to brutally murder a process: maybe one of them is trapping exits but is also stuck in an infinite loop, never reading any message. The kill reason acts as a special signal that can't be trapped. This ensures any process you terminate with it will really be dead. Usually, kill is a bit of a last resort, when everything else has failed.

As the kill reason can never be trapped, it needs to be changed to killed when other processes receive the message. If it weren't changed in that manner, every other process linked to it would in turn die for the same kill reason and would in turn kill its neighbors, and so on. A death cascade would ensue.

This also explains why exit(kill) looks like killed when received from another linked process (the signal is modified so it doesn't cascade), but still looks like kill when trapped locally.

If you find this all confusing, don't worry. Many programmers feel the same. Exit signals are a bit of a funny beast. Luckily there aren't many more special cases than the ones described above. Once you understand those, you can understand most of Erlang's concurrent error management without a problem.

Monitors

So yeah. Maybe murdering processes isn't what you want. Maybe you don't feel like taking the world down with you once you're gone. Maybe you're more of a stalker. In that case, monitors might be what you want.

More seriously, monitors are a special type of link with two differences:

- they are unidirectional;

- they can be stacked.

Monitors are what you want when a process wants to know what's going on with a second process, but neither of them really are vital to each other.

Another reason, as listed above, is stacking the references. Now this might seem useless from a quick look, but it is great for writing libraries which need to know what's going on with other processes.

You see, links are more of an organizational construct. When you design the architecture of your application, you determine which process will do which jobs, and what will depend on what. Some processes will supervise others, some couldn't live without a twin process, etc. This structure is usually something fixed, known in advance. Links are useful for that and should not necessarily be used outside of it.

But what happens if you have 2 or 3 different libraries that you call and they all need to know whether a process is alive or not? If you were to use links for this, you would quickly hit a problem whenever you needed to unlink a process. Now, links aren't stackable, so the moment you unlink one, you unlink them all and mess up all the assumptions put up by the other libraries. That's pretty bad. So you need stackable links, and monitors are your solution. They can be removed individually. Plus, being unidirectional is handy in libraries because other processes shouldn't have to be aware of said libraries.

So what does a monitor look like? Easy enough, let's set one up. The function is erlang:monitor/2, where the first argument is the atom process and the second one is the pid:

1> erlang:monitor(process, spawn(fun() -> timer:sleep(500) end)).

#Ref<0.0.0.77>

2> flush().

Shell got {'DOWN',#Ref<0.0.0.77>,process,<0.63.0>,normal}

ok

Every time a process you monitor goes down, you will receive such a message. The message is {'DOWN', MonitorReference, process, Pid, Reason}. The reference is there to allow you to demonitor the process. Remember, monitors are stackable, so it's possible to take more than one down. References allow you to track each of them in a unique manner. Also note that as with links, there is an atomic function to spawn a process while monitoring it, spawn_monitor/1-3:

3> {Pid, Ref} = spawn_monitor(fun() -> receive _ -> exit(boom) end end).

{<0.73.0>,#Ref<0.0.0.100>}

4> erlang:demonitor(Ref).

true

5> Pid ! die.

die

6> flush().

ok

In this case, we demonitored the other process before it crashed and as such we had no trace of it dying. The function demonitor/2 also exists and gives a little bit more information. The second parameter can be a list of options. Only two exist, info and flush:

7> f().

ok

8> {Pid, Ref} = spawn_monitor(fun() -> receive _ -> exit(boom) end end).

{<0.35.0>,#Ref<0.0.0.35>}

9> Pid ! die.

die

10> erlang:demonitor(Ref, [flush, info]).

false

11> flush().

ok

The info option tells you if a monitor existed or not when you tried to remove it. This is why the expression 10 returned false. Using flush as an option will remove the DOWN message from the mailbox if it existed, resulting in flush() finding nothing in the current process' mailbox.

Naming Processes

With links and monitors understood, there is another problem still left to be solved. Let's use the following functions of the linkmon.erl module:

start_critic() ->

spawn(?MODULE, critic, []).

judge(Pid, Band, Album) ->

Pid ! {self(), {Band, Album}},

receive

{Pid, Criticism} -> Criticisim

after 2000 ->

timeout

end.

critic() ->

receive

{From, {"Rage Against the Turing Machine", "Unit Testify"}} ->

From ! {self(), "They are great!"};

{From, {"System of a Downtime", "Memoize"}} ->

From ! {self(), "They're not Johnny Crash but they're good."};

{From, {"Johnny Crash", "The Token Ring of Fire"}} ->

From ! {self(), "Simply incredible."};

{From, {_Band, _Album}} ->

From ! {self(), "They are terrible!"}

end,

critic().

Now we'll just pretend we're going around stores, shopping for music. There are a few albums that sound interesting, but we're never quite sure. You decide to call your friend, the critic.

1> c(linkmon).

{ok,linkmon}

2> Critic = linkmon:start_critic().

<0.47.0>

3> linkmon:judge(Critic, "Genesis", "The Lambda Lies Down on Broadway").

"They are terrible!"

Because of a solar storm (I'm trying to find something realistic here), the connection is dropped:

4> exit(Critic, solar_storm).

true

5> linkmon:judge(Critic, "Genesis", "A trick of the Tail Recursion").

timeout

Annoying. We can no longer get criticism for the albums. To keep the critic alive, we'll write a basic 'supervisor' process whose only role is to restart it when it goes down:

start_critic2() ->

spawn(?MODULE, restarter, []).

restarter() ->

process_flag(trap_exit, true),

Pid = spawn_link(?MODULE, critic, []),

receive

{'EXIT', Pid, normal} -> % not a crash

ok;

{'EXIT', Pid, shutdown} -> % manual termination, not a crash

ok;

{'EXIT', Pid, _} ->

restarter()

end.

Here, the restarter will be its own process. It will in turn start the critic's process and if it ever dies of abnormal cause, restarter/0 will loop and create a new critic. Note that I added a clause for {'EXIT', Pid, shutdown} as a way to manually kill the critic if we ever need to.

The problem with our approach is that there is no way to find the Pid of the critic, and thus we can't call him to have his opinion. One of the solutions Erlang has to solve this is to give names to processes.

The act of giving a name to a process allows you to replace the unpredictable pid by an atom. This atom can then be used exactly as a Pid when sending messages. To give a process a name, the function erlang:register/2 is used. If the process dies, it will automatically lose its name or you can also use unregister/1 to do it manually. You can get a list of all registered processes with registered/0 or a more detailed one with the shell command regs(). Here we can rewrite the restarter/0 function as follows:

restarter() ->

process_flag(trap_exit, true),

Pid = spawn_link(?MODULE, critic, []),

register(critic, Pid),

receive

{'EXIT', Pid, normal} -> % not a crash

ok;

{'EXIT', Pid, shutdown} -> % manual termination, not a crash

ok;

{'EXIT', Pid, _} ->

restarter()

end.

So as you can see, register/2 will always give our critic the name 'critic', no matter what the Pid is. What we need to do is then remove the need to pass in a Pid from the abstraction functions. Let's try this one:

judge2(Band, Album) ->

critic ! {self(), {Band, Album}},

Pid = whereis(critic),

receive

{Pid, Criticism} -> Criticism

after 2000 ->

timeout

end.

Here, the line Pid = whereis(critic) is used to find the critic's process identifier in order to pattern match against it in the receive expression. We want to match with this pid, because it makes sure we will match on the right message (there could be 500 of them in the mailbox as we speak!) This can be the source of a problem though. The code above assumes that the critic's pid will remain the same between the first two lines of the function. However, it is completely plausible the following will happen:

1. critic ! Message

2. critic receives

3. critic replies

4. critic dies

5. whereis fails

6. critic is restarted

7. code crashes

Or yet, this is also a possibility:

1. critic ! Message

2. critic receives

3. critic replies

4. critic dies

5. critic is restarted

6. whereis picks up

wrong pid

7. message never matches

The possibility that things go wrong in a different process can make another one go wrong if we don't do things right. In this case, the value of the critic atom can be seen from multiple processes. This is known as shared state. The problem here is that the value of critic can be accessed and modified by different processes at virtually the same time, resulting in inconsistent information and software errors. The common term for such things is a race condition. Race conditions are particularly dangerous because they depend on the timing of events. In pretty much every concurrent and parallel language out there, this timing depends on unpredictable factors such as how busy the processor is, where the processes go, and what data is being processed by your program.

Don't drink too much kool-aid:

You might have heard that Erlang is usually free of race conditions or deadlocks and makes parallel code safe. This is true in many circumstances, but never assume your code is really that safe. Named processes are only one example of the multiple ways in which parallel code can go wrong.

Other examples include access to files on the computer (to modify them), updating the same database records from many different processes, etc.

Luckily for us, it's relatively easy to fix the code above if we don't assume the named process remains the same. Instead, we'll use references (created with make_ref()) as unique values to identify messages. We'll need to rewrite the critic/0 function into critic2/0 and judge/3 into judge2/2:

judge2(Band, Album) ->

Ref = make_ref(),

critic ! {self(), Ref, {Band, Album}},

receive

{Ref, Criticism} -> Criticism

after 2000 ->

timeout

end.

critic2() ->

receive

{From, Ref, {"Rage Against the Turing Machine", "Unit Testify"}} ->

From ! {Ref, "They are great!"};

{From, Ref, {"System of a Downtime", "Memoize"}} ->

From ! {Ref, "They're not Johnny Crash but they're good."};

{From, Ref, {"Johnny Crash", "The Token Ring of Fire"}} ->

From ! {Ref, "Simply incredible."};

{From, Ref, {_Band, _Album}} ->

From ! {Ref, "They are terrible!"}

end,

critic().

And then change restarter/0 to fit by making it spawn critic2/0 rather than critic/0. Now the other functions should keep working fine. The user won't see a difference. Well, they will because we renamed functions and changed the number of parameters, but they won't know what implementation details were changed and why it was important. All they'll see is that their code got simpler and they no longer need to send a pid around function calls:

6> c(linkmon).

{ok,linkmon}

7> linkmon:start_critic2().

<0.55.0>

8> linkmon:judge2("The Doors", "Light my Firewall").

"They are terrible!"

9> exit(whereis(critic), kill).

true

10> linkmon:judge2("Rage Against the Turing Machine", "Unit Testify").

"They are great!"

And now, even though we killed the critic, a new one instantly came back to solve our problems. That's the usefulness of named processes. Had you tried to call linkmon:judge/2 without a registered process, a bad argument error would have been thrown by the ! operator inside the function, making sure that processes that depend on named ones can't run without them.

Note: If you remember earlier texts, atoms can be used in a limited (though high) number. You shouldn't ever create dynamic atoms. This means naming processes should be reserved to important services unique to an instance of the VM and processes that should be there for the whole time your application runs.

If you need named processes but they are transient or there isn't any of them which can be unique to the VM, it may mean they need need to be represented as a group instead. Linking and restarting them together if they crash might be the sane option, rather than trying to use dynamic names.

In the next chapter, we'll put the recent knowledge we gained on concurrent programming with Erlang to practice by writing a real application.

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

More On Multiprocessing

State Your State

The examples shown in the previous chapter were all right for demonstrative purposes, but you won't go far with only that in your toolkit. It's not that the examples were bad, it's mostly that there is not a huge advantage to processes and actors if they're just functions with messages. To fix this, we have to be able to hold state in a process.

Let's first create a function in a new kitchen.erl module that will let a process act like a fridge. The process will allow two operations: storing food in the fridge and taking food from the fridge. It should only be possible to take food that has been stored beforehand. The following function can act as the base for our process:

-module(kitchen).

-compile(export_all).

fridge1() ->

receive

{From, {store, _Food}} ->

From ! {self(), ok},

fridge1();

{From, {take, _Food}} ->

%% uh....

From ! {self(), not_found},

fridge1();

terminate ->

ok

end.

Something's wrong with it. When we ask to store the food, the process should reply with ok, but there is nothing actually storing the food; fridge1() is called and then the function starts from scratch, without state. You can also see that when we call the process to take food from the fridge, there is no state to take it from and so the only thing to reply is not_found. In order to store and take food items, we'll need to add state to the function.

With the help of recursion, the state to a process can then be held entirely in the parameters of the function. In the case of our fridge process, a possibility would be to store all the food as a list, and then look in that list when someone needs to eat something:

fridge2(FoodList) ->

receive

{From, {store, Food}} ->

From ! {self(), ok},

fridge2([Food|FoodList]);

{From, {take, Food}} ->

case lists:member(Food, FoodList) of

true ->

From ! {self(), {ok, Food}},

fridge2(lists:delete(Food, FoodList));

false ->

From ! {self(), not_found},

fridge2(FoodList)

end;

terminate ->

ok

end.

The first thing to notice is that fridge2/1 takes one argument, FoodList. You can see that when we send a message that matches {From, {store, Food}}, the function will add Food to FoodList before going. Once that recursive call is made, it will then be possible to retrieve the same item. In fact, I implemented it there. The function uses lists:member/2 to check whether Food is part of FoodList or not. Depending on the result, the item is sent back to the calling process (and removed from FoodList) or not_found is sent back otherwise:

1> c(kitchen).

{ok,kitchen}

2> Pid = spawn(kitchen, fridge2, [[baking_soda]]).

<0.51.0>

3> Pid ! {self(), {store, milk}}.

{<0.33.0>,{store,milk}}

4> flush().

Shell got {<0.51.0>,ok}

ok

Storing items in the fridge seems to work. We'll try with some more stuff and then try to take it from the fridge.

5> Pid ! {self(), {store, bacon}}.

{<0.33.0>,{store,bacon}}

6> Pid ! {self(), {take, bacon}}.

{<0.33.0>,{take,bacon}}

7> Pid ! {self(), {take, turkey}}.

{<0.33.0>,{take,turkey}}

8> flush().

Shell got {<0.51.0>,ok}

Shell got {<0.51.0>,{ok,bacon}}

Shell got {<0.51.0>,not_found}

ok

As expected, we can take bacon from the fridge because we have put it in there first (along with the milk and baking soda), but the fridge process has no turkey to find when we request some. This is why we get the last {<0.51.0>,not_found} message.

We love messages, but we keep them secret

Something annoying with the previous example is that the programmer who's going to use the fridge has to know about the protocol that's been invented for that process. That's a useless burden. A good way to solve this is to abstract messages away with the help of functions dealing with receiving and sending them:

store(Pid, Food) ->

Pid ! {self(), {store, Food}},

receive

{Pid, Msg} -> Msg

end.

take(Pid, Food) ->

Pid ! {self(), {take, Food}},

receive

{Pid, Msg} -> Msg

end.

Now the interaction with the process is much cleaner:

9> c(kitchen).

{ok,kitchen}

10> f().

ok

11> Pid = spawn(kitchen, fridge2, [[baking_soda]]).

<0.73.0>

12> kitchen:store(Pid, water).

ok

13> kitchen:take(Pid, water).

{ok,water}

14> kitchen:take(Pid, juice).

not_found

We don't have to care about how the messages work anymore, if sending self() or a precise atom like take or store is needed: all that's needed is a pid and knowing what functions to call. This hides all of the dirty work and makes it easier to build on the fridge process.

One thing left to do would be to hide that whole part about needing to spawn a process. We dealt with hiding messages, but then we still expect the user to handle the creation of the process. I'll add the following start/1 function:

start(FoodList) ->

spawn(?MODULE, fridge2, [FoodList]).

Here, ?MODULE is a macro returning the current module's name. It doesn't look like there are any advantages to writing such a function, but there really are some. The essential part of it would be consistency with the calls to take/2 and store/2: everything about the fridge process is now handled by the kitchen module. If you were to add logging when the fridge process is started or start a second process (say a freezer), it would be really easy to do inside our start/1 function. However if the spawning is left for the user to do through spawn/3, then every place that starts a fridge now needs to add the new calls. That's prone to errors and errors suck.

Let's see this function put to use:

15> f().

ok

16> c(kitchen).

{ok,kitchen}

17> Pid = kitchen:start([rhubarb, dog, hotdog]).

<0.84.0>

18> kitchen:take(Pid, dog).

{ok,dog}

19> kitchen:take(Pid, dog).

not_found

Yay! The dog has got out of the fridge and our abstraction is complete!

Time Out

Let's try a little something with the help of the command pid(A,B,C), which lets us change the 3 integers A, B and C into a pid. Here we'll deliberately feed kitchen:take/2 a fake one:

20> kitchen:take(pid(0,250,0), dog).

Woops. The shell is frozen. This happened because of how take/2 was implemented. To understand what goes on, let's first revise what happens in the normal case:

1. A message to store food is sent from you (the shell) to the fridge process;

2. Your process switches to receive mode and waits for a new message;

3. The fridge stores the item and sends 'ok' to your process;

4. Your process receives it and moves on with its life.

And here's what happens when the shell freezes:

1. A message to store food is sent from you (the shell) to an unknown process;

2. Your process switches to receive mode and waits for a new message;

3. The unknown process either doesn't exist or doesn't expect such a message and does nothing with it;

4. Your shell process is stuck in receive mode.

That's annoying, especially because there is no error handling possible here. Nothing illegal happened, the program is just waiting. In general, anything dealing with asynchronous operations (which is how message passing is done in Erlang) needs a way to give up after a certain period of time if it gets no sign of receiving data. A web browser does it when a page or image takes too long to load, you do it when someone takes too long before answering the phone or is late at a meeting. Erlang certainly has an appropriate mechanism for that, and it's part of the receive construct:

receive

Match -> Expression1

after Delay ->

Expression2

end.

The part in between receive and after is exactly the same that we already know. The after part will be triggered if as much time as Delay (an integer representing milliseconds) has been spent without receiving a message that matches the Match pattern. When this happens, Expression2 is executed.

We'll write two new interface functions, store2/2 and take2/2, which will act exactly like store/2 and take/2 with the exception that they will stop waiting after 3 seconds:

store2(Pid, Food) ->

Pid ! {self(), {store, Food}},

receive

{Pid, Msg} -> Msg

after 3000 ->

timeout

end.

take2(Pid, Food) ->

Pid ! {self(), {take, Food}},

receive

{Pid, Msg} -> Msg

after 3000 ->

timeout

end.

Now you can unfreeze the shell with ^G and try the new interface functions:

User switch command

--> k

--> s

--> c

Eshell V5.7.5  (abort with ^G)

1> c(kitchen).

{ok,kitchen}

2> kitchen:take2(pid(0,250,0), dog).

timeout

And now it works.

Note: I said that after only takes milliseconds as a value, but it is actually possible to use the atom infinity. While this is not useful in many cases (you might just remove the after clause altogether), it is sometimes used when the programmer can submit the wait time to a function where receiving a result is expected. That way, if the programmer really wants to wait forever, he can.

There are uses to such timers other than giving up after too long. One very simple example is how the timer:sleep/1 function we've used before works. Here's how it is implemented (let's put it in a new multiproc.erl module):

sleep(T) ->

receive

after T -> ok

end.

In this specific case, no message will ever be matched in the receive part of the construct because there is no pattern. Instead, the after part of the construct will be called once the delay T has passed.

Another special case is when the timeout is at 0:

flush() ->

receive

_ -> flush()

after 0 ->

ok

end.

When that happens, the Erlang VM will try and find a message that fits one of the available patterns. In the case above, anything matches. As long as there are messages, the flush/0 function will recursively call itself until the mailbox is empty. Once this is done, the after 0 -> ok part of the code is executed and the function returns.

Selective Receives

This 'flushing' concept makes it possible to implement a selective receive which can give a priority to the messages you receive by nesting calls:

important() ->

receive

{Priority, Message} when Priority > 10 ->

[Message | important()]

after 0 ->

normal()

end.

normal() ->

receive

{_, Message} ->

[Message | normal()]

after 0 ->

[]

end.

This function will build a list of all messages with those with a priority above 10 coming first:

1> c(multiproc).

{ok,multiproc}

2> self() ! {15, high}, self() ! {7, low}, self() ! {1, low}, self() ! {17, high}.

{17,high}

3> multiproc:important().

[high,high,low,low]

Because I used the after 0 bit, every message will be obtained until none is left, but the process will try to grab all those with a priority above 10 before even considering the other messages, which are accumulated in the normal/0 call.

If this practice looks interesting, be aware that is is sometimes unsafe due to the way selective receives work in Erlang.

When messages are sent to a process, they're stored in the mailbox until the process reads them and they match a pattern there. As said in the previous chapter, the messages are stored in the order they were received. This means every time you match a message, it begins by the oldest one.

That oldest message is then tried against every pattern of the receive until one of them matches. When it does, the message is removed from the mailbox and the code for the process executes normally until the next receive. When this next receive is evaluated, the VM will look for the oldest message currently in the mailbox (the one after the one we removed), and so on.

When there is no way to match a given message, it is put in a save queue and the next message is tried. If the second message matches, the first message is put back on top of the mailbox to be retried later.

This lets you only care about the messages that are useful. Ignoring some messages to handle them later in the manner described above is the essence of selective receives. While they're useful, the problem with them is that if your process has a lot of messages you never care about, reading useful messages will actually take longer and longer (and the processes will grow in size too).

In the drawing above, imagine we want the 367th message, but the first 366 are junk ignored by our code. To get the 367th message, the process needs to try to match the 366 first ones. Once it's done and they've all been put in the queue, the 367th message is taken out and the first 366 are put back on top of the mailbox. The next useful message could be burrowed much deeper and take even longer to be found.

This kind of receive is a frequent cause of performance problems in Erlang. If your application is running slow and you know there are lots of messages going around, this could be the cause.

If such selective receives are effectively causing a massive slowdown in your code, the first thing to do is to ask yourself is why you are getting messages you do not want. Are the messages sent to the right processes? Are the patterns correct? Are the messages formatted incorrectly? Are you using one process where there should be many? Answering one or many of these questions could solve your problem.

Because of the risks of having useless messages polluting a process' mailbox, Erlang programmers sometimes take a defensive measure against such events. A standard way to do it might look like this:

receive

Pattern1 -> Expression1;

Pattern2 -> Expression2;

Pattern3 -> Expression3;

...

PatternN -> ExpressionN;

Unexpected ->

io:format("unexpected message ~p~n", [Unexpected])

end.

What this does is make sure any message will match at least one clause. The Unexpected variable will match anything, take the unexpected message out of the mailbox and show a warning. Depending on your application, you might want to store the message into some kind of logging facility where you will be able to find information about it later on: if the messages are going to the wrong process, it'd be a shame to lose them for good and have a hard time finding why that other process doesn't receive what it should.

In the case you do need to work with a priority in your messages and can't use such a catch-all clause, a smarter way to do it would be to implement a min-heap or use the gb_trees module and dump every received message in it (make sure to put the priority number first in the key so it gets used for sorting the messages). Then you can just search for the smallest or largest element in the data structure according to your needs.

In most cases, this technique should let you receive messages with a priority more efficiently than selective receives. However, it could slow you down if most messages you receive have the highest priority possible. As usual, the trick is to profile and measure before optimizing.

Note: Since R14A, a new optimization has been added to Erlang's compiler. It simplifies selective receives in very specific cases of back-and-forth communications between processes. An example of such a function is optimized/1 in multiproc.erl.

To make it work, a reference (make_ref()) has to be created in a function and then sent in a message. In the same function, a selective receive is then made. If no message can match unless it contains the same reference, the compiler automatically makes sure the VM will skip messages received before the creation of that reference.

Note that you shouldn't try to coerce your code to fit such optimizations. The Erlang developers only look for patterns that are frequently used and then make them faster. If you write idiomatic code, optimizations should come to you. Not the other way around.

With these concepts understood, the next step will be to do error handling with multiple processes.

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Designing a Concurrent Application

All fine and dandy. You understand the concepts, but then again, all we've had since the beginning of the book were toy examples: calculators, trees, Heathrow to London, etc. It's time for something more fun and more educational. We'll write a small application in concurrent Erlang. The application's going to be small and line-based, but still useful and moderately extensible.

I'm a somewhat unorganized person. I'm lost with homework, things to do around the apartment, this book, work, meetings, appointments, etc. I end up having a dozen of lists everywhere with tasks I still forget to do or look over. Hopefully you still need reminders of what to do (but you don't have a mind that wanders as much as mine does), because we're going to write one of these event reminder applications that prompt you to do stuff and remind you about appointments.

Understanding the Problem

The first step is to know what the hell we're doing. "A reminder app," you say. "Of course," I say. But there's more. How do we plan on interacting with the software? What do we want it to do for us? How do we represent the program with processes? How do we know what messages to send?

As the quote goes, "Walking on water and developing software from a specification are easy if both are frozen." So let's get a spec and stick to it. Our little piece of software will allow us to do the following:

- Add an event. Events contain a deadline (the time to warn at), an event name and a description.

- Show a warning when the time has come for it.

- Cancel an event by name.

- No persistent disk storage. It's not needed to show the architectural concepts we'll see. It will suck for a real app, but I'll instead just show where it could be inserted if you wanted to do it and also point to a few helpful functions.

- Given we have no persistent storage, we have to be able to update the code while it is running.

- The interaction with the software will be done via the command line, but it should be possible to later extend it so other means could be used (say a GUI, web page access, instant messaging software, email, etc.)

Here's the structure of the program I picked to do it:

Where the client, event server and x, y and z are all processes. Here's what each of them can do:

Event Server

- Accepts subscriptions from clients

- Forwards notifications from event processes to each of the subscribers

- Accepts messages to add events (and start the x, y, z processes needed)

- Can accept messages to cancel an event and subsequently kill the event processes

- Can be terminated by a client

- Can have its code reloaded via the shell.

client

- Subscribes to the event server and receive notifications as messages. As such it should be easy to design a bunch of clients all subscribing to the event server. Each of these could potentially be a gateway to the different interaction points mentioned above (GUI, web page, instant messaging software, email, etc.)

- Asks the server to add an event with all its details

- Asks the server to cancel an event

- Monitors the server (to know if it goes down)

- Shuts down the event server if needed

x, y and z:

- Represent a notification waiting to fire (they're basically just timers linked to the event server)

- Send a message to the event server when the time is up

- Receive a cancellation message and die

Note that all clients (IM, mail, etc. which are not implemented in this book) are notified about all events, and a cancellation is not something to warn the clients about. Here the software is written for you and me, and it's assumed only one user will run it.

Here's a more complex graph with all the possible messages:

This represents every process we'll have. By drawing all the arrows there and saying they're messages, we've written a high level protocol, or at least its skeleton.

It should be noted that using one process per event to be reminded of is likely going to be overkill and hard to scale in a real world application. However, for an application you are going to be the sole user of, this is good enough. A different approach could be using functions such as timer:send_after/2-3 to avoid spawning too many processes.

Defining the Protocol

Now that we know what each component has to do and to communicate, a good idea would be to make a list of all messages that will be sent and specify what they will look like. Let's first start with the communication between the client and the event server:

Here I chose to use two monitors because there is no obvious dependency between the client and the server. I mean, of course the client doesn't work without the server, but the server can live without a client. A link could have done the job right here, but because we want our system to be extensible with many clients, we can't assume other clients will all want to crash when the server dies. And neither can we assume the client can really be turned into a system process and trap exits in case the server dies. Now to the next message set:

This adds an event to the event server. A confirmation is sent back under the form of the ok atom, unless something goes wrong (maybe the TimeOut is in the wrong format.) The inverse operation, removing events, can be done as follows:

The event server can then later send a notification when the event is due:

Then we only need the two following special cases for when we want to shut the server down or when it crashes:

No direct confirmation is sent when the server dies because the monitor will already warn us of that. That's pretty much all that will happen between the client and the event server. Now for the messages between the event server and the event processes themselves.

A thing to note here before we start is that it would be very useful to have the event server linked to the events. The reason for this is that we want all events to die if the server does: they make no sense without it.

Ok, so back to the events. When the event server starts them, it gives each of them a special identifier (the event's name). Once one of these events' time has come, it needs to send a message saying so:

On the other hand, the event has to watch for cancel calls from the event server:

And that should be it. One last message will be needed for our protocol, the one that lets us upgrade the server:

No reply is necessary. We'll see why when we actually program that feature and you'll see it makes sense.

Having both the protocol defined and the general idea of how our process hierarchy will look in place, we can actually start working on the project.

Lay Them Foundations

To begin with it all, we should lay down a standard Erlang directory structure, which looks like this:

ebin/

include/

priv/

src/

The ebin/ directory is where files will go once they are compiled. The include/ directory is used to store .hrl files that are to be included by other applications; the private .hrl files are usually just kept inside the src/ directory. The priv/ directory is used for executables that might have to interact with Erlang, such as specific drivers and whatnot. We won't actually use that directory for this project. Then the last one is the src/ directory, where all .erl files stay.

In standard Erlang projects, this directory structure can vary a little. A conf/ directory can be added for specific configuration files, doc/ for documentation and lib/ for third party libraries required for your application to run. Different Erlang product on the market often use different names than these, but the four ones mentioned above usually stay the same given they're part of the standard OTP practices.

An Event Module

Get into the src/ directory and start an event.erl module, which will implement the x, y and z events in the earlier drawings. I'm starting with this module because it's the one with the fewest dependencies: we'll be able to try to run it without having implement the event server or client functions.

Before really writing code, I have to mention that the protocol is incomplete. It helps represent what data will be sent from process to process, but not the intricacies of it: how the addressing works, whether we use references or names, etc. Most messages will be wrapped under the form {Pid, Ref, Message}, where Pid is the sender and Ref is a unique message identifier to help know what reply came from who. If we were to send many messages before looking for replies, we would not know what reply went with what message without a reference.

So here we go. The core of the processes that will run event.erl's code will be the function loop/1, which will look a bit like the following skeleton if you remember the protocol:

loop(State) ->

receive

{Server, Ref, cancel} ->

...

after Delay ->

...

end.

This shows the timeout we have to support to announce an event has come to term and the way a server can call for the cancellation of an event. You'll notice a variable State in the loop. The State variable will have to contain data such as the timeout value (in seconds) and the name of the event (in order to send the message {done, Id}.) It will also need to know the event server's pid in order to send it notifications.

This is all stuff that's fit to be held in the loop's state. So let's declare a state record on the top of the file:

-module(event).

-compile(export_all).

-record(state, {server,

name="",

to_go=0}).

With this state defined, it should be possible to refine the loop a bit more:

loop(S = #state{server=Server}) ->

receive

{Server, Ref, cancel} ->

Server ! {Ref, ok}

after S#state.to_go*1000 ->

Server ! {done, S#state.name}

end.

Here, the multiplication by a thousand is to change the to_go value from seconds to milliseconds.

Don't drink too much Kool-Aid:

Language wart ahead! The reason why I bind the variable 'Server' in the function head is because it's used in pattern matching in the receive section. Remember, records are hacks! The expression S#state.server is secretly expanded to element(S, 2), which isn't a valid pattern to match on.

This still works fine for S#state.to_go after the after part, because that one can be an expression left to be evaluated later.

Now to test the loop:

6> c(event).

{ok,event}

7> rr(event, state).

[state]

8> spawn(event, loop, [#state{server=self(), name="test", to_go=5}]).

<0.60.0>

9> flush().

ok

10> flush().

Shell got {done,"test"}

ok

11> Pid = spawn(event, loop, [#state{server=self(), name="test", to_go=500}]).

<0.64.0>

12> ReplyRef = make_ref().

#Ref<0.0.0.210>

13> Pid ! {self(), ReplyRef, cancel}.

{<0.50.0>,#Ref<0.0.0.210>,cancel}

14> flush().

Shell got {#Ref<0.0.0.210>,ok}

ok

Lots of stuff to see here. Well first of all, we import the record from the event module with rr(Mod). Then, we spawn the event loop with the shell as the server (self()). This event should fire after 5 seconds. The 9th expression was run after 3 seconds, and the 10th one after 6 seconds. You can see we did receive the {done, "test"} message on the second try.

Right after that, I try the cancel feature (with an ample 500 seconds to type it). You can see I created the reference, sent the message and got a reply with the same reference so I know the ok I received was coming from this process and not any other on the system.

The reason why the cancel message is wrapped with a reference but the done message isn't is simply because we don't expect it to come from anywhere specific (any place will do, we won't match on the receive) nor should we want to reply to it. There's another test I want to do beforehand. What about an event happening next year?

15> spawn(event, loop, [#state{server=self(), name="test", to_go=365*24*60*60}]).

<0.69.0>

16>

=ERROR REPORT==== DD-MM-YYYY::HH:mm:SS ===

Error in process <0.69.0> with exit value: {timeout_value,[{event,loop,1}]}

Ouch. It seems like we hit an implementation limit. It turns out Erlang's timeout value is limited to about 50 days in milliseconds. It might not be significant, but I'm showing this error for three reasons:

1. It bit me in the ass when writing the module and testing it, halfway through the chapter.

2. Erlang is certainly not perfect for every task and what we're seeing here is the consequences of using timers in ways not intended by the implementers.

3. That's not really a problem; let's work around it.

The fix I decided to apply for this one was to write a function that would split the timeout value into many parts if turns out to be too long. This will request some support from the loop/1 function too. So yeah, the way to split the time is basically divide it in equal parts of 49 days (because the limit is about 50), and then put the remainder with all these equal parts. The sum of the list of seconds should now be the original time:

%% Because Erlang is limited to about 49 days (49*24*60*60*1000) in

%% milliseconds, the following function is used

normalize(N) ->

Limit = 49*24*60*60,

[N rem Limit | lists:duplicate(N div Limit, Limit)].

The function lists:duplicate/2 will take a given expression as a first argument and reproduce it as many times as the value of the second argument. If we were to send normalize/1 the value 98*24*60*60+4, it would return [4,4233600,4233600]. The loop/1 function should now look like this to accommodate the new format:

%% Loop uses a list for times in order to go around the ~49 days limit

%% on timeouts.

loop(S = #state{server=Server, to_go=[T|Next]}) ->

receive

{Server, Ref, cancel} ->

Server ! {Ref, ok}

after T*1000 ->

if Next =:= [] ->

Server ! {done, S#state.name};

Next =/= [] ->

loop(S#state{to_go=Next})

end

end.

You can try it, it should work as normal, but now support years and years of timeout. How this works is that it takes the first element of the to_go list and waits for its whole duration. When this is done, the next element of the timeout list is verified. If it's empty, the timeout is over and the server is notified of it. Otherwise, the loop keeps going with the rest of the list until it's done.

It would be very annoying to have to manually call something like event:normalize(N) every time an event process is started, especially since our workaround shouldn't be of concern to programmers using our code. The standard way to do this is to instead have an init function handling all initialization of data required for the loop function to work well. While we're at it, we'll add the standard start and start_link functions:

start(EventName, Delay) ->

spawn(?MODULE, init, [self(), EventName, Delay]).

start_link(EventName, Delay) ->

spawn_link(?MODULE, init, [self(), EventName, Delay]).

%%% Event's innards

init(Server, EventName, Delay) ->

loop(#state{server=Server,

name=EventName,

to_go=normalize(Delay)}).

The interface is now much cleaner. Before testing, though, it would be nice to have the only message we can send, cancel, also have its own interface function:

cancel(Pid) ->

%% Monitor in case the process is already dead

Ref = erlang:monitor(process, Pid),

Pid ! {self(), Ref, cancel},

receive

{Ref, ok} ->

erlang:demonitor(Ref, [flush]),

ok;

{'DOWN', Ref, process, Pid, _Reason} ->

ok

end.

Oh! A new trick! Here I'm using a monitor to see if the process is there or not. If the process is already dead, I avoid useless waiting time and return ok as specified in the protocol. If the process replies with the reference, then I know it will soon die: I remove the reference to avoid receiving them when I no longer care about them. Note that I also supply the flush option, which will purge the DOWN message if it was sent before we had the time to demonitor.

Let's test these:

17> c(event).

{ok,event}

18> f().

ok

19> event:start("Event", 0).

<0.103.0>

20> flush().

Shell got {done,"Event"}

ok

21> Pid = event:start("Event", 500).

<0.106.0>

22> event:cancel(Pid).

ok

And it works! The last thing annoying with the event module is that we have to input the time left in seconds. It would be much better if we could use a standard format such as Erlang's datetime ({{Year, Day, Month}, {Hour, Minute, Second}}). Just add the following function that will calculate the difference between the current time on your computer and the delay you inserted:

time_to_go(TimeOut={{_,_,_}, {_,_,_}}) ->

Now = calendar:local_time(),

ToGo = calendar:datetime_to_gregorian_seconds(TimeOut) -

calendar:datetime_to_gregorian_seconds(Now),

Secs = if ToGo > 0  -> ToGo;

ToGo =< 0 -> 0

end,

normalize(Secs).

Oh, yeah. The calendar module has pretty funky function names. As noted above, this calculate the number of seconds between now and when the event is supposed to fire. If the event is in the past, we instead return 0 so it will nnotify the server as soon as it can. Now fix the init function to call this one instead of normalize/1. You can also rename Delay variables to say DateTime if you want the names to be more descriptive:

init(Server, EventName, DateTime) ->

loop(#state{server=Server,

name=EventName,

to_go=time_to_go(DateTime)}).

Now that this is done, we can take a break. Start a new event, go drink a pint (half-litre) of milk/beer and come back just in time to see the event message coming in.

The Event Server

Let's deal with the event server. According to the protocol, the skeleton for that one should look a bit like this:

-module(evserv).

-compile(export_all).

loop(State) ->

receive

{Pid, MsgRef, {subscribe, Client}} ->

...

{Pid, MsgRef, {add, Name, Description, TimeOut}} ->

...

{Pid, MsgRef, {cancel, Name}} ->

...

{done, Name} ->

...

shutdown ->

...

{'DOWN', Ref, process, _Pid, _Reason} ->

...

code_change ->

...

Unknown ->

io:format("Unknown message: ~p~n",[Unknown]),

loop(State)

end.

You'll notice I have wrapped calls that require replies with the same {Pid, Ref, Message} format as earlier. Now, the server will need to keep two things in its state: a list of subscribing clients and a list of all the event processes it spawned. If you have noticed, the protocol says that when an event is done, the event server should receive {done, Name}, but send {done, Name, Description}. The idea here is to have as little traffic as necessary and only have the event processes care about what is strictly necessary. So yeah, list of clients and list of events:

-record(state, {events,    %% list of #event{} records

clients}). %% list of Pids

-record(event, {name="",

description="",

pid,

timeout={{1970,1,1},{0,0,0}}}).

And the loop now has the record definition in its head:

loop(S = #state{}) ->

receive

...

end.

It would be nice if both events and clients were orddicts. We're unlikely to have many hundreds of them at once. If you recall the chapter on data structures, orddicts fit that need very well. We'll write an init function to handle this:

init() ->

%% Loading events from a static file could be done here.

%% You would need to pass an argument to init telling where the

%% resource to find the events is. Then load it from here.

%% Another option is to just pass the events straight to the server

%% through this function.

loop(#state{events=orddict:new(),

clients=orddict:new()}).

With the skeleton and initialization done, I'll implement each message one by one. The first message is the one about subscriptions. We want to keep a list of all subscribers because when an event is done, we have to notify them. Also, the protocol above mentions we should monitor them. It makes sense because we don't want to hold onto crashed clients and send useless messages for no reason. Anyway, it should look like this:

{Pid, MsgRef, {subscribe, Client}} ->

Ref = erlang:monitor(process, Client),

NewClients = orddict:store(Ref, Client, S#state.clients),

Pid ! {MsgRef, ok},

loop(S#state{clients=NewClients});

So what this section of loop/1 does is start a monitor, and store the client info in the orddict under the key Ref. The reason for this is simple: the only other time we'll need to fetch the client ID will be if we receive a monitor's EXIT message, which will contain the reference (which will let us get rid of the orddict's entry).

The next message to care about is the one where we add events. Now, it is possible to return an error status. The only validation we'll do is check the timestamps we accept. While it's easy to subscribe to the {{Year,Month,Day}, {Hour,Minute,seconds}} layout, we have to make sure we don't do things like accept events on February 29 when we're not in a leap year, or any other date that doesn't exist. Moreover, we don't want to accept impossible date values such as "5 hours, minus 1 minute and 75 seconds". A single function can take care of validating all of that.

The first building block we'll use is the function calendar:valid_date/1. This one, as the name says, checks if the date is valid or not. Sadly, the weirdness of the calendar module doesn't stop at funky names: there is actually no function to confirm that {H,M,S} has valid values. We'll have to implement that one too, following the funky naming scheme:

valid_datetime({Date,Time}) ->

try

calendar:valid_date(Date) andalso valid_time(Time)

catch

error:function_clause -> %% not in {{D,M,Y},{H,Min,S}} format

false

end;

valid_datetime(_) ->

false.

valid_time({H,M,S}) -> valid_time(H,M,S).

valid_time(H,M,S) when H >= 0, H < 24,

M >= 0, M < 60,

S >= 0, S < 60 -> true;

valid_time(_,_,_) -> false.

The valid_datetime/1 function can now be used in the part where we try to add the message:

{Pid, MsgRef, {add, Name, Description, TimeOut}} ->

case valid_datetime(TimeOut) of

true ->

EventPid = event:start_link(Name, TimeOut),

NewEvents = orddict:store(Name,

#event{name=Name,

description=Description,

pid=EventPid,

timeout=TimeOut},

S#state.events),

Pid ! {MsgRef, ok},

loop(S#state{events=NewEvents});

false ->

Pid ! {MsgRef, {error, bad_timeout}},

loop(S)

end;

If the time is valid, we spawn a new event process, then store its data in the event server's state before sending a confirmation to the caller. If the timeout is wrong, we notify the client rather than having the error pass silently or crashing the server. Additional checks could be added for name clashes or other restrictions (just remember to update the protocol documentation!)

The next message defined in our protocol is the one where we cancel an event. Canceling an event never fails on the client side, so the code is simpler there. Just check whether the event is in the process' state record. If it is, use the event:cancel/1 function we defined to kill it and send ok. If it's not found, just tell the user everything went right anyway -- the event is not running and that's what the user wanted.

{Pid, MsgRef, {cancel, Name}} ->

Events = case orddict:find(Name, S#state.events) of

{ok, E} ->

event:cancel(E#event.pid),

orddict:erase(Name, S#state.events);

error ->

S#state.events

end,

Pid ! {MsgRef, ok},

loop(S#state{events=Events});

Good, good. So now all voluntary interaction coming from the client to the event server is covered. Let's deal with the stuff that's going between the server and the events themselves. There are two messages to handle: canceling the events (which is done), and the events timing out. That message is simply {done, Name}:

{done, Name} ->

E = orddict:fetch(Name, S#state.events),

send_to_clients({done, E#event.name, E#event.description},

S#state.clients),

NewEvents = orddict:erase(Name, S#state.events),

loop(S#state{events=NewEvents});

And the function send_to_clients/2 does as its name says and is defined as follows:

send_to_clients(Msg, ClientDict) ->

orddict:map(fun(_Ref, Pid) -> Pid ! Msg end, ClientDict).

That should be it for most of the loop code. What's left is the set different status messages: clients going down, shutdown, code upgrades, etc. Here they come:

shutdown ->

exit(shutdown);

{'DOWN', Ref, process, _Pid, _Reason} ->

loop(S#state{clients=orddict:erase(Ref, S#state.clients)});

code_change ->

?MODULE:loop(S);

Unknown ->

io:format("Unknown message: ~p~n",[Unknown]),

loop(S)

The first case (shutdown) is pretty explicit. You get the kill message, let the process die. If you wanted to save state to disk, that could be a possible place to do it. If you wanted safer save/exit semantics, this could be done on every add, cancel or done message. Loading events from disk could then be done in the init function, spawning them as they come.

The 'DOWN' message's actions are also simple enough. It means a client died, so we remove it from the client list in the state.

Unknown messages will just be shown with io:format/2 for debugging purposes, although a real production application would likely use a dedicated logging module

And here comes the code change message. This one is interesting enough for me to give it its own section.

Hot Code Loving

In order to do hot code loading, Erlang has a thing called the code server. The code server is basically a VM process in charge of an ETS table (in-memory database table, native to the VM.) The code server can hold two versions of a single module in memory, and both versions can run at once. A new version of a module is automatically loaded when compiling it with c(Module), loading with l(Module) or loading it with one of the many functions of the code module.

A concept to understand is that Erlang has both local and external calls. Local calls are those function calls you can make with functions that might not be exported. They're just of the format Atom(Args). An external call, on the other hand, can only be done with exported functions and has the form Module:Function(Args).

When there are two versions of a module loaded in the VM, all local calls are done through the currently running version in a process. However, external calls are always done on the newest version of the code available in the code server. Then, if local calls are made from within the external one, they are in the new version of the code.

Given that every process/actor in Erlang needs to do a recursive call in order to change its state, it is possible to load entirely new versions of an actor by having an external recursive call.

Note: If you load a third version of a module while a process still runs with the first one, that process gets killed by the VM, which assumes it was an orphan process without a supervisor or a way to upgrade itself. If nobody runs the oldest version, it is simply dropped and the newest ones are kept instead.

There are ways to bind yourself to a system module that will send messages whenever a new version of a module is loaded. By doing this, you can trigger a module reload only when receiving such a message, and always do it with a code upgrade function, say MyModule:Upgrade(CurrentState), which will then be able to transform the state data structure according to the new version's specification. This 'subscription' handling is done automatically by the OTP framework, which we'll start studying soon enough. For the reminder application, we won't use the code server and will instead use a custom code_change message from the shell, doing very basic reloading. That's pretty much all you need to know to do hot code loading. Nevertheless, here's a more generic example:

-module(hotload).

-export([server/1, upgrade/1]).

server(State) ->

receive

update ->

NewState = ?MODULE:upgrade(State),

?MODULE:server(NewState);  %% loop in the new version of the module

SomeMessage ->

%% do something here

server(State)  %% stay in the same version no matter what.

end.

upgrade(OldState) ->

%% transform and return the state here.

As you can see, our ?MODULE:loop(S) fits this pattern.

I Said, Hide Your Messages

Hiding messages! If you expect people to build on your code and processes, you must hide the messages in interface functions. Here's what we used for the event module:

start() ->

register(?MODULE, Pid=spawn(?MODULE, init, [])),

Pid.

start_link() ->

register(?MODULE, Pid=spawn_link(?MODULE, init, [])),

Pid.

terminate() ->

?MODULE ! shutdown.

I decided to register the server module because, for now, we should only have one running at a time. If you were to expand the reminder use to support many users, it would be a good idea to instead register the names with the global module, or the gproc library. For the sake of this example app, this will be enough.

The first message we wrote is the next we should abstract away: how to subscribe. The little protocol or specification I wrote above called for a monitor, so this one is added there. At any point, if the reference returned by the subscribe message is in a DOWN message, the client will know the server has gone down.

subscribe(Pid) ->

Ref = erlang:monitor(process, whereis(?MODULE)),

?MODULE ! {self(), Ref, {subscribe, Pid}},

receive

{Ref, ok} ->

{ok, Ref};

{'DOWN', Ref, process, _Pid, Reason} ->

{error, Reason}

after 5000 ->

{error, timeout}

end.

The next one is the event adding:

add_event(Name, Description, TimeOut) ->

Ref = make_ref(),

?MODULE ! {self(), Ref, {add, Name, Description, TimeOut}},

receive

{Ref, Msg} -> Msg

after 5000 ->

{error, timeout}

end.

Note that I choose to forward the {error, bad_timeout} message that could be received to the client. I could have also decided to crash the client by raising erlang:error(bad_timout). Whether crashing the client or forwarding the error message is the thing to do is still debated in the community. Here's the alternative crashing function:

add_event2(Name, Description, TimeOut) ->

Ref = make_ref(),

?MODULE ! {self(), Ref, {add, Name, Description, TimeOut}},

receive

{Ref, {error, Reason}} -> erlang:error(Reason);

{Ref, Msg} -> Msg

after 5000 ->

{error, timeout}

end.

Then there's event cancellation, which just takes a name:

cancel(Name) ->

Ref = make_ref(),

?MODULE ! {self(), Ref, {cancel, Name}},

receive

{Ref, ok} -> ok

after 5000 ->

{error, timeout}

end.

Last of all is a small nicety provided for the client, a function used to accumulate all messages during a given period of time. If messages are found, they're all taken and the function returns as soon as possible:

listen(Delay) ->

receive

M = {done, _Name, _Description} ->

[M | listen(0)]

after Delay*1000 ->

[]

end.

A Test Drive

You should now be able to compile the application and give it a test run. To make things a bit simpler, we'll write a specific Erlang makefile to build the project. Open a file named Emakefile and put it in the project's base directory. The file contains Erlang terms and gives the Erlang compiler the recipe to cook wonderful and crispy .beam files:

{'src/*', [debug_info,

{i, "src"},

{i, "include"},

{outdir, "ebin"}]}.

This tells the compiler to add debug_info to the files (this is rarely an option you want to give up), to look for files in the src/ and include/ directory and to output them in ebin/.

Now, by going in your command line and running erl -make, the files should all be compiled and put inside the ebin/ directory for you. Start the Erlang shell by doing erl -pa ebin/. The -pa <directory> option tells the Erlang VM to add that path to the places it can look in for modules.

Another option is to start the shell as usual and call make:all([load]). This will look for a file named 'Emakefile' in the current directory, recompile it (if it changed) and load the new files.

You should now be able to track thousands of events (just replace the DateTime variables with whatever makes sense when you're writing the text):

1> evserv:start().

<0.34.0>

2> evserv:subscribe(self()).

{ok,#Ref<0.0.0.31>}

3> evserv:add_event("Hey there", "test", FutureDateTime).

ok

4> evserv:listen(5).

[]

5> evserv:cancel("Hey there").

ok

6> evserv:add_event("Hey there2", "test", NextMinuteDateTime).

ok

7> evserv:listen(2000).

[{done,"Hey there2","test"}]

Nice nice nice. Writing any client should now be simple enough given the few basic interface functions we have created.

Adding Supervision

In order to be a more stable application, we should write another 'restarter' as we did in the last chapter. Open up a file named sup.erl where our supervisor will be:

-module(sup).

-export([start/2, start_link/2, init/1, loop/1]).

start(Mod,Args) ->

spawn(?MODULE, init, [{Mod, Args}]).

start_link(Mod,Args) ->

spawn_link(?MODULE, init, [{Mod, Args}]).

init({Mod,Args}) ->

process_flag(trap_exit, true),

loop({Mod,start_link,Args}).

loop({M,F,A}) ->

Pid = apply(M,F,A),

receive

{'EXIT', Pid, Reason} ->

io:format("Process ~p exited for reason ~p~n",[Pid,Reason]),

loop({M,F,A});

{'EXIT', _From, shutdown} ->

exit(shutdown) % will kill the child too

end.

This is somewhat similar to the 'restarter', although this one is a tad more generic. It can take any module, as long as it has a start_link function. It will restart the process it watches indefinitely, unless the supervisor itself is terminated with a shutdown exit signal. Here it is in use:

1> c(evserv), c(sup).

{ok,sup}

2> SupPid = sup:start(evserv, []).

<0.43.0>

3> whereis(evserv).

<0.44.0>

4> exit(whereis(evserv), die).

true

Process <0.44.0> exited for reason die

5> exit(whereis(evserv), die).

Process <0.48.0> exited for reason die

true

6> exit(SupPid, shutdown).

true

7> whereis(evserv).

undefined

As you can see, killing the supervisor will also kill its child.

Note: We'll see much more advanced and flexible supervisors in the chapter about OTP supervisors. Those are the ones people are thinking of when they mention supervision trees. The supervisor demonstrated here is only the most basic form that exists and is not exactly fit for production environments compared to the real thing.

Namespaces (or lack thereof)

Because Erlang has a flat module structure (there is no hierarchy), It is frequent for some applications to enter in conflict. One example of this is the frequently used user module that almost every project attempts to define at least once. This clashes with the user module shipped with Erlang. You can test for any clashes with the function code:clash/0.

Because of this, the common pattern is to prefix every module name with the name of your project. In this case, our reminder application's modules should be renamed to reminder_evserv, reminder_sup and reminder_event.

Some programmers then decide to add a module, named after the application itself, which wraps common calls that programmers could use when using their own application. Example calls could be functions such as starting the application with a supervisor, subscribing to the server, adding and cancelling events, etc.

It's important to be aware of other namespaces, too, such as registered names that must not clash, database tables, etc.

That's pretty much it for a very basic concurrent Erlang application. This one showed we could have a bunch of concurrent processes without thinking too hard about it: supervisors, clients, servers, processes used as timers (and we could have thousands of them), etc. No need to synchronize them, no locks, no real main loop. Message passing has made it simple to compartmentalize our application into a few modules with separated concerns and tasks.

The basic calls inside evserv.erl could now be used to construct clients that would allow to interact with the event server from somewhere outside of the Erlang VM and make the program truly useful.

Before doing that, though, I suggest you read up on the OTP framework. The next few chapters will cover some of its building blocks, which will allow for much more robust and elegant applications. A huge part of Erlang's power comes from using it. It's a carefully crafted and well-engineered tool that any self-respecting Erlang programmer has to know.

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

What is OTP?

It's The Open Telecom Platform!

OTP stands for Open Telecom Platform, although it's not that much about telecom anymore (it's more about software that has the property of telecom applications, but yeah.) If half of Erlang's greatness comes from its concurrency and distribution and the other half comes from its error handling capabilities, then the OTP framework is the third half of it.

During the previous chapters, we've seen a few examples of common practices on how to write concurrent applications with the languages' built-in facilities: links, monitors, servers, timeouts, trapping exits, etc. There were a few 'gotchas' here and there on the order things need to be done, on how to avoid race conditions or to always remember that a process could die at any time. There was also hot code loading, naming processes and adding supervisors, to name a few.

Doing all of this manually is time consuming and sometimes prone to error. There are corner cases to be forgotten about and pits to fall into. The OTP framework takes care of this by grouping these essential practices into a set of libraries that have been carefully engineered and battle-hardened over years. Every Erlang programmer should use them.

The OTP framework is also a set of modules and standards designed to help you build applications. Given most Erlang programmers end up using OTP, most Erlang applications you'll encounter in the wild will tend to follow these standards.

The Common Process, Abstracted

One of the things we've done many times in the previous process examples is divide everything in accordance to very specific tasks. In most processes, we had a function in charge of spawning the new process, a function in charge of giving it its initial values, a main loop, etc.

These parts, as it turns out, are usually present in all concurrent programs you'll write, no matter what the process might be used for.

The engineers and computer scientists behind the OTP framework spotted these patterns and included them in a bunch of common libraries. These libraries are built with code that is equivalent to most of the abstractions we used (like using references to tag messages), with the advantage of being used for years in the field and also being built with far more caution than we were with our implementations. They contain functions to safely spawn and initialize processes, send messages to them in a fault-tolerant manner and many other things. Funnily enough, you should rarely need to use these libraries yourself. The abstractions they contain are so basic and universal that a lot more interesting things were built on top of them. Those libraries are the ones we'll use.

In the following chapters we'll see a few of the common uses of processes and then how they can be abstracted, then made generic. Then for each of these we'll also see the corresponding implementation with the OTP framework's behaviours and how to use each of them.

The Basic Server

The first common pattern I'll describe is one we've already used. When writing the event server, we had what could be called a client-server model. The event server would receive calls from the client, act on them and then reply to it if the protocol said to do so.

For this chapter, we'll use a very simple server, allowing us to focus on the essential properties of it. Here's the kitty_server:

%%%%% Naive version

-module(kitty_server).

-export([start_link/0, order_cat/4, return_cat/1, close_shop/1]).

-record(cat, {name, color=green, description}).

%%% Client API

start_link() -> spawn_link(fun init/0).

%% Synchronous call

order_cat(Pid, Name, Color, Description) ->

Ref = erlang:monitor(process, Pid),

Pid ! {self(), Ref, {order, Name, Color, Description}},

receive

{Ref, Cat} ->

erlang:demonitor(Ref, [flush]),

Cat;

{'DOWN', Ref, process, Pid, Reason} ->

erlang:error(Reason)

after 5000 ->

erlang:error(timeout)

end.

%% This call is asynchronous

return_cat(Pid, Cat = #cat{}) ->

Pid ! {return, Cat},

ok.

%% Synchronous call

close_shop(Pid) ->

Ref = erlang:monitor(process, Pid),

Pid ! {self(), Ref, terminate},

receive

{Ref, ok} ->

erlang:demonitor(Ref, [flush]),

ok;

{'DOWN', Ref, process, Pid, Reason} ->

erlang:error(Reason)

after 5000 ->

erlang:error(timeout)

end.

%%% Server functions

init() -> loop([]).

loop(Cats) ->

receive

{Pid, Ref, {order, Name, Color, Description}} ->

if Cats =:= [] ->

Pid ! {Ref, make_cat(Name, Color, Description)},

loop(Cats);

Cats =/= [] -> % got to empty the stock

Pid ! {Ref, hd(Cats)},

loop(tl(Cats))

end;

{return, Cat = #cat{}} ->

loop([Cat|Cats]);

{Pid, Ref, terminate} ->

Pid ! {Ref, ok},

terminate(Cats);

Unknown ->

%% do some logging here too

io:format("Unknown message: ~p~n", [Unknown]),

loop(Cats)

end.

%%% Private functions

make_cat(Name, Col, Desc) ->

#cat{name=Name, color=Col, description=Desc}.

terminate(Cats) ->

[io:format("~p was set free.~n",[C#cat.name]) || C <- Cats],

ok.

So this is a kitty server/store. The behavior is extremely simple: you describe a cat and you get that cat. If someone returns a cat, it's added to a list and is then automatically sent as the next order instead of what the client actually asked for (we're in this kitty store for the money, not smiles):

1> c(kitty_server).

{ok,kitty_server}

2> rr(kitty_server).

[cat]

3> Pid = kitty_server:start_link().

<0.57.0>

4> Cat1 = kitty_server:order_cat(Pid, carl, brown, "loves to burn bridges").

#cat{name = carl,color = brown,

description = "loves to burn bridges"}

5> kitty_server:return_cat(Pid, Cat1).

ok

6> kitty_server:order_cat(Pid, jimmy, orange, "cuddly").

#cat{name = carl,color = brown,

description = "loves to burn bridges"}

7> kitty_server:order_cat(Pid, jimmy, orange, "cuddly").

#cat{name = jimmy,color = orange,description = "cuddly"}

8> kitty_server:return_cat(Pid, Cat1).

ok

9> kitty_server:close_shop(Pid).

carl was set free.

ok

10> kitty_server:close_shop(Pid).

** exception error: no such process or port

in function  kitty_server:close_shop/1

Looking back at the source code for the module, we can see patterns we've previously applied. The sections where we set monitors up and down, apply timers, receive data, use a main loop, handle the init function, etc. should all be familiar. It should be possible to abstract away these things we end up repeating all the time.

Let's first take a look at the client API. The first thing we can notice is that both synchronous calls are extremely similar. These are the calls that would likely go in abstraction libraries as mentioned in the previous section. For now, we'll just abstract these away as a single function in a new module which will hold all the generic parts of the kitty server:

-module(my_server).

-compile(export_all).

call(Pid, Msg) ->

Ref = erlang:monitor(process, Pid),

Pid ! {self(), Ref, Msg},

receive

{Ref, Reply} ->

erlang:demonitor(Ref, [flush]),

Reply;

{'DOWN', Ref, process, Pid, Reason} ->

erlang:error(Reason)

after 5000 ->

erlang:error(timeout)

end.

This takes a message and a PID, sticks them into in the function, then forwards the message for you in a safe manner. From now on, we can just substitute the message sending we do with a call to this function. So if we were to rewrite a new kitty server to be paired with the abstracted my_server, it could begin like this:

-module(kitty_server2).

-export(start_link/0, order_cat/4, return_cat/1, close_shop/1]).

-record(cat, {name, color=green, description}).

%%% Client API

start_link() -> spawn_link(fun init/0).

%% Synchronous call

order_cat(Pid, Name, Color, Description) ->

my_server:call(Pid, {order, Name, Color, Description}).

%% This call is asynchronous

return_cat(Pid, Cat = #cat{}) ->

Pid ! {return, Cat},

ok.

%% Synchronous call

close_shop(Pid) ->

my_server:call(Pid, terminate).

The next big generic chunk of code we have is not as obvious as the call/2 function. Note that every process we've written so far has a loop where all the messages are pattern matched. This is a bit of a touchy part, but here we have to separate the pattern matching from the loop itself. One quick way to do it would be to add:

loop(Module, State) ->

receive

Message -> Module:handle(Message, State)

end.

And then the specific module can look like this:

handle(Message1, State) -> NewState1;

handle(Message2, State) -> NewState2;

...

handle(MessageN, State) -> NewStateN.

This is better. There are still ways to make it even cleaner. If you paid attention when reading the kitty_server module (and I hope you did!), you will have noticed we have a specific way to call synchronously and another one to call asynchronously. It would be pretty helpful if our generic server implementation could provide a clear way to know which kind of call is which.

In order to do this, we will need to match different kinds of messages in my_server:loop/2. This means we'll need to change the call/2 function a little bit so synchronous calls are made obvious by adding the atom sync to the message on the function's second line:

call(Pid, Msg) ->

Ref = erlang:monitor(process, Pid),

Pid ! {sync, self(), Ref, Msg},

receive

{Ref, Reply} ->

erlang:demonitor(Ref, [flush]),

Reply;

{'DOWN', Ref, process, Pid, Reason} ->

erlang:error(Reason)

after 5000 ->

erlang:error(timeout)

end.

We can now provide a new function for asynchronous calls. The function cast/2 will handle this:

cast(Pid, Msg) ->

Pid ! {async, Msg},

ok.

With this done, the loop can now look like this:

loop(Module, State) ->

receive

{async, Msg} ->

loop(Module, Module:handle_cast(Msg, State));

{sync, Pid, Ref, Msg} ->

loop(Module, Module:handle_call(Msg, Pid, Ref, State))

end.

And then you could also add specific slots to handle messages that don't fit the sync/async concept (maybe they were sent by accident) or to have your debug functions and other stuff like hot code reloading in there.

One disappointing thing with the loop above is that the abstraction is leaking. The programmers who will use my_server will still need to know about references when sending synchronous messages and replying to them. That makes the abstraction useless. To use it, you still need to understand all the boring details. Here's a quick fix for it:

loop(Module, State) ->

receive

{async, Msg} ->

loop(Module, Module:handle_cast(Msg, State));

{sync, Pid, Ref, Msg} ->

loop(Module, Module:handle_call(Msg, {Pid, Ref}, State))

end.

By putting both variables Pid and Ref in a tuple, they can be passed as a single argument to the other function as a variable with a name like From. Then the user doesn't have to know anything about the variable's innards. Instead, we'll provide a function to send replies that should understand what From contains:

reply({Pid, Ref}, Reply) ->

Pid ! {Ref, Reply}.

What is left to do is specify the starter functions (start, start_link and init) that pass around the module names and whatnot. Once they're added, the module should look like this:

-module(my_server).

-export([start/2, start_link/2, call/2, cast/2, reply/2]).

%%% Public API

start(Module, InitialState) ->

spawn(fun() -> init(Module, InitialState) end).

start_link(Module, InitialState) ->

spawn_link(fun() -> init(Module, InitialState) end).

call(Pid, Msg) ->

Ref = erlang:monitor(process, Pid),

Pid ! {sync, self(), Ref, Msg},

receive

{Ref, Reply} ->

erlang:demonitor(Ref, [flush]),

Reply;

{'DOWN', Ref, process, Pid, Reason} ->

erlang:error(Reason)

after 5000 ->

erlang:error(timeout)

end.

cast(Pid, Msg) ->

Pid ! {async, Msg},

ok.

reply({Pid, Ref}, Reply) ->

Pid ! {Ref, Reply}.

%%% Private stuff

init(Module, InitialState) ->

loop(Module, Module:init(InitialState)).

loop(Module, State) ->

receive

{async, Msg} ->

loop(Module, Module:handle_cast(Msg, State));

{sync, Pid, Ref, Msg} ->

loop(Module, Module:handle_call(Msg, {Pid, Ref}, State))

end.

The next thing to do is reimplement the kitty server, now kitty_server2 as a callback module that will respect the interface we defined for my_server. We'll keep the same interface as the previous implementation, except all the calls are now redirected to go through my_server:

-module(kitty_server2).

-export([start_link/0, order_cat/4, return_cat/2, close_shop/1]).

-export([init/1, handle_call/3, handle_cast/2]).

-record(cat, {name, color=green, description}).

%%% Client API

start_link() -> my_server:start_link(?MODULE, []).

%% Synchronous call

order_cat(Pid, Name, Color, Description) ->

my_server:call(Pid, {order, Name, Color, Description}).

%% This call is asynchronous

return_cat(Pid, Cat = #cat{}) ->

my_server:cast(Pid, {return, Cat}).

%% Synchronous call

close_shop(Pid) ->

my_server:call(Pid, terminate).

Note that I added a second -export() at the top of the module. Those are the functions my_server will need to call to make everything work:

%%% Server functions

init([]) -> []. %% no treatment of info here!

handle_call({order, Name, Color, Description}, From, Cats) ->

if Cats =:= [] ->

my_server:reply(From, make_cat(Name, Color, Description)),

Cats;

Cats =/= [] ->

my_server:reply(From, hd(Cats)),

tl(Cats)

end;

handle_call(terminate, From, Cats) ->

my_server:reply(From, ok),

terminate(Cats).

handle_cast({return, Cat = #cat{}}, Cats) ->

[Cat|Cats].

And then what needs to be done is to re-add the private functions:

%%% Private functions

make_cat(Name, Col, Desc) ->

#cat{name=Name, color=Col, description=Desc}.

terminate(Cats) ->

[io:format("~p was set free.~n",[C#cat.name]) || C <- Cats],

exit(normal).

Just make sure to replace the ok we had before by exit(normal) in terminate/1, otherwise the server will keep going on.

The code should be compilable and testable, and run in exactly the same manner as it was before. The code is quite similar, but let's see what changed.

Specific Vs. Generic

What we've just done is get an understanding the core of OTP (conceptually speaking). This is what OTP really is all about: taking all the generic components, extracting them in libraries, making sure they work well and then reusing that code when possible. Then all that's left to do is focus on the specific stuff, things that will always change from application to application.

Obviously, there isn't much to save by doing things that way with only the kitty server. It looks a bit like abstraction for abstraction's sake. If the app we had to ship to a customer were nothing but the kitty server, then the first version might be fine. If you're going to have larger applications then it might be worth it to separate generic parts of your code from the specific sections.

Let's imagine for a moment that we have some Erlang software running on a server. Our software has a few kitty servers running, a veterinary process (you send your broken kitties and it returns them fixed), a kitty beauty salon, a server for pet food, supplies, etc. Most of these can be implemented with a client-server pattern. As time goes, your complex system becomes full of different servers running around.

Adding servers adds complexity in terms of code, but also in terms of testing, maintenance and understanding. Each implementation might be different, programmed in different styles by different people, and so on. However, if all these servers share the same common my_server abstraction, you substantially reduce that complexity. You understand the basic concept of the module instantly ("oh, it's a server!"), there's a single generic implementation of it to test, document, etc. The rest of the effort can be put on each specific implementation of it.

This means you reduce a lot of time tracking and solving bugs (just do it at one place for all servers). It also means that you reduce the number of bugs you introduce. If you were to re-write the my_server:call/3 or the process' main loop all the time, not only would it be more time consuming, but chances of forgetting one step or the other would skyrocket, and so would bugs. Fewer bugs mean fewer calls during the night to go fix something, which is definitely good for all of us. Your mileage may vary, but I'll bet you don't appreciate going to the office on days off to fix bugs either.

Another interesting thing about what we did when separating the generic from the specific is that we instantly made it much easier to test our individual modules. If you wanted to unit test the old kitty server implementation, you'd need to spawn one process per test, give it the right state, send your messages and hope for the reply you expected. On the other hand, our second kitty server only requires us to run the function calls over the 'handle_call/3' and 'handle_cast/2' functions and see what they output as a new state. No need to set up servers, manipulate the state. Just pass it in as a function parameter. Note that this also means the generic aspect of the server is much easier to test given you can just implement very simple functions that do nothing else than let you focus on the behaviour you want to observe, without the rest.

A much more 'hidden' advantage of using common abstractions in that way is that if everyone uses the exact same backend for their processes, when someone optimizes that single backend to make it a little bit faster, every process using it out there will run a little bit faster too. For this principle to work in practice, it's usually necessary to have a whole lot of people using the same abstractions and putting effort on them. Luckily for the Erlang community, that's what happens with the OTP framework.

Back to our modules. There are a bunch of things we haven't yet addressed: named processes, configuring the timeouts, adding debug information, what to do with unexpected messages, how to tie in hot code loading, handling specific errors, abstracting away the need to write most replies, handling most ways to shut a server down, making sure the server plays nice with supervisors, etc. Going over all of this is superfluous for this text, but would be necessary in real products that need to be shipped. Again, you might see why doing all of this by yourself is a bit of a risky task. Luckily for you (and the people who'll support your applications), the Erlang/OTP team managed to handle all of that for you with the gen_server behaviour. gen_server is a bit like my_server on steroids, except it has years and years of testing and production use behind it.

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Clients and Servers

Callback to the Future

The first OTP behaviour we'll see is one of the most used ones. Its name is gen_server and it has an interface a bit similar to the one we've written with my_server in last chapter; it gives you a few functions to use it and in exchange, your module has to already have a few functions gen_server will use.

init

The first one is an init/1 function. It is similar to the one we've used with my_server in that it is used to initialize the server's state and do all of these one-time tasks that it will depend on. The function can return {ok, State}, {ok, State, TimeOut}, {ok, State, hibernate}, {stop, Reason} or ignore.

The normal {ok, State} return value doesn't really need explaining past saying that State will be passed directly to the main loop of the process as the state to keep later on. The TimeOut variable is meant to be added to the tuple whenever you need a deadline before which you expect the server to receive a message. If no message is received before the deadline, a special one (the atom timeout) is sent to the server, which should be handled with handle_info/2 (more on this later.)

On the other hand, if you do expect the process to take a long time before getting a reply and are worried about memory, you can add the hibernate atom to the tuple. Hibernation basically reduces the size of the process' state until it gets a message, at the cost of some processing power. If you are in doubt about using hibernation, you probably don't need it.

Returning {stop, Reason} should be done when something went wrong during the initialization.

Note: here's a more technical definition of process hibernation. It's no big deal if some readers do not understand or care about it. When the BIF erlang:hibernate(M,F,A) is called, the call stack for the currently running process is discarded (the function never returns). The garbage collection then kicks in, and what's left is one continuous heap that is shrunken to the size of the data in the process. This basically compacts all the data so the process takes less place.

Once the process receives a message, the function M:F with A as arguments is called and the execution resumes.

Note: while init/1 is running, execution is blocked in the process that spawned the server. This is because it is waiting for a 'ready' message sent automatically by the gen_server module to make sure everything went fine.

handle_call

The function handle_call/3 is used to work with synchronous messages (we'll see how to send them soon). It takes 3 arguments: Request, From, and State. It's pretty similar to how we programmed our own handle_call/3 in my_server. The biggest difference is how you reply to messages. In our own abstraction of a server, it was necessary to use my_server:reply/2 to talk back to the process. In the case of gen_servers, there are 8 different return values possible, taking the form of tuples.

Because there are many of them, here's a simple list instead:

{reply,Reply,NewState}

{reply,Reply,NewState,Timeout}

{reply,Reply,NewState,hibernate}

{noreply,NewState}

{noreply,NewState,Timeout}

{noreply,NewState,hibernate}

{stop,Reason,Reply,NewState}

{stop,Reason,NewState}

For all of these, Timeout and hibernate work the same way as for init/1. Whatever is in Reply will be sent back to whoever called the server in the first place. Notice that there are three possible noreply options. When you use noreply, the generic part of the server will assume you're taking care of sending the reply back yourself. This can be done with gen_server:reply/2, which can be used in the same way as my_server:reply/2.

Most of the time, you'll only need the reply tuples. There are still a few valid reasons to use noreply: whenever you want another process to send the reply for you or when you want to send an acknowledgement ('hey! I received the message!') but still process it afterwards (without replying this time), etc. If this is what you choose to do, it is absolutely necessary to use gen_server:reply/2 because otherwise the call will time out and cause a crash.

handle_cast

The handle_cast/2 callback works a lot like the one we had in my_server: it takes the parameters Message and State and is used to handle asynchronous calls. You do whatever you want in there, in a manner quite similar to what's doable with handle_call/3. On the other hand, only tuples without replies are valid return values:

{noreply,NewState}

{noreply,NewState,Timeout}

{noreply,NewState,hibernate}

{stop,Reason,NewState}

handle_info

You know how I mentioned our own server didn't really deal with messages that do not fit our interface, right? Well handle_info/2 is the solution. It's very similar to handle_cast/2 and in fact returns the same tuples. The difference is that this callback is only there for messages that were sent directly with the ! operator and special ones like init/1's timeout, monitors' notifications and 'EXIT' signals.

terminate

The callback terminate/2 is called whenever one of the three handle_Something functions returns a tuple of the form {stop, Reason, NewState} or {stop, Reason, Reply, NewState}. It takes two parameters, Reason and State, corresponding to the same values from the stop tuples.

terminate/2 will also be called when its parent (the process that spawned it) dies, if and only if the gen_server is trapping exits.

Note: if any reason other than normal, shutdown or {shutdown, Term} is used when terminate/2 is called, the OTP framework will see this as a failure and start logging a bunch of stuff here and there for you.

This function is pretty much the direct opposite of init/1 so whatever was done in there should have its opposite in terminate/2. It's your server's janitor, the function in charge of locking the door after making sure everyone's gone. Of course, the function is helped by the VM itself, which should usually delete all ETS tables, close all ports, etc. for you. Note that the return value of this function doesn't really matter, because the code stops executing after it's been called.

code_change

The function code_change/3 is there to let you upgrade code. It takes the form code_change(PreviousVersion, State, Extra). Here, the variable PreviousVersion is either the version term istelf in the case of an upgrade (read More About Modules again if you forget what this is), or {down, Version} in the case of a downgrade (just reloading older code). The State variable holds all of the current's server state so you can convert it.

Imagine for a moment that we used an orddict to store all of our data. However, as time goes on, the orddict becomes too slow and we decide to change it for a regular dict. In order to avoid the process crashing on the next function call, the conversion from one data structure to the other can be done in there, safely. All you have to do is return the new state with {ok, NewState}.

The Extra variable isn't something we'll worry about for now. It's mostly used in larger OTP deployment, where specific tools exist to upgrade entire releases on a VM. We're not there yet.

So now we've got all the callbacks defined. Don't worry if you're a bit lost: the OTP framework is a bit circular sometimes, where to understand part A of the framework you have to understand part B, but then part B requires to see part A to be useful. The best way to get over that confusion is to actually implement a gen_server.

.BEAM me up, Scotty!

This is going to be the kitty_gen_server. It's going to be mostly similar to kitty_server2, with only minimal API changes. First start a new module with the following lines in it:

-module(kitty_gen_server).

-behaviour(gen_server).

And try to compile it. You should get something like this:

1> c(kitty_gen_server).

./kitty_gen_server.erl:2: Warning: undefined callback function code_change/3 (behaviour 'gen_server')

./kitty_gen_server.erl:2: Warning: undefined callback function handle_call/3 (behaviour 'gen_server')

./kitty_gen_server.erl:2: Warning: undefined callback function handle_cast/2 (behaviour 'gen_server')

./kitty_gen_server.erl:2: Warning: undefined callback function handle_info/2 (behaviour 'gen_server')

./kitty_gen_server.erl:2: Warning: undefined callback function init/1 (behaviour 'gen_server')

./kitty_gen_server.erl:2: Warning: undefined callback function terminate/2 (behaviour 'gen_server')

{ok,kitty_gen_server}

The compilation worked, but there are warnings about missing callbacks. This is because of the gen_server behaviour. A behaviour is basically a way for a module to specify functions it expects another module to have. The behaviour is the contract sealing the deal between the well-behaved generic part of the code and the specific, error-prone part of the code (yours).

Note: both 'behavior' and 'behaviour' are accepted by the Erlang compiler.

Defining your own behaviours is really simple. You just need to export a function called behaviour_info/1 implemented as follows:

-module(my_behaviour).

-export([behaviour_info/1]).

%% init/1, some_fun/0 and other/3 are now expected callbacks

behaviour_info(callbacks) -> [{init,1}, {some_fun, 0}, {other, 3}];

behaviour_info(_) -> undefined.

And that's about it for behaviours. You can just use -behaviour(my_behaviour). in a module implementing them to get compiler warnings if you forgot a function. Anyway, back to our third kitty server.

The first function we had was start_link/0. This one can be changed to the following:

start_link() -> gen_server:start_link(?MODULE, [], []).

The first parameter is the callback module, the second one is the list of parameters to pass to init/1 and the third one is about debugging options that won't be covered right now. You could add a fourth parameter in the first position, which would be the name to register the server with. Note that while the previous version of the function simply returned a pid, this one instead returns {ok, Pid}.

Next functions now:

%% Synchronous call

order_cat(Pid, Name, Color, Description) ->

gen_server:call(Pid, {order, Name, Color, Description}).

%% This call is asynchronous

return_cat(Pid, Cat = #cat{}) ->

gen_server:cast(Pid, {return, Cat}).

%% Synchronous call

close_shop(Pid) ->

gen_server:call(Pid, terminate).

All of these calls are a one-to-one change. Note that a third parameter can be passed to gen_server:call/2-3 to give a timeout. If you don't give a timeout to the function (or the atom infinity), the default is set to 5 seconds. If no reply is received before time is up, the call crashes.

Now we'll be able to add the gen_server callbacks. The following table shows the relationship we have between calls and callbacks:

gen_server

YourModule

start/3-4

init/1

start_link/3-4

init/1

call/2-3

handle_call/3

cast/2

handle_cast/2

And then you have the other callbacks, those that are more about special cases:

- handle_info/2

- terminate/2

- code_change/3

Let's begin by changing those we already have to fit the model: init/1, handle_call/3 and handle_cast/2.

%%% Server functions

init([]) -> {ok, []}. %% no treatment of info here!

handle_call({order, Name, Color, Description}, _From, Cats) ->

if Cats =:= [] ->

{reply, make_cat(Name, Color, Description), Cats};

Cats =/= [] ->

{reply, hd(Cats), tl(Cats)}

end;

handle_call(terminate, _From, Cats) ->

{stop, normal, ok, Cats}.

handle_cast({return, Cat = #cat{}}, Cats) ->

{noreply, [Cat|Cats]}.

Again, very little has changed there. In fact, the code is now shorter, thanks to smarter abstractions. Now we get to the new callbacks. The first one is handle_info/2. Given this is a toy module and we have no logging system pre-defined, just outputting the unexpected messages will be enough:

handle_info(Msg, Cats) ->

io:format("Unexpected message: ~p~n",[Msg]),

{noreply, Cats}.

The next one is the terminate/2 callback. It will be very similar to the terminate/1 private function we had:

terminate(normal, Cats) ->

[io:format("~p was set free.~n",[C#cat.name]) || C <- Cats],

ok.

And then the last callback, code_change/3:

code_change(_OldVsn, State, _Extra) ->

%% No change planned. The function is there for the behaviour,

%% but will not be used. Only a version on the next

{ok, State}.

Just remember to keep in the make_cat/3 private function:

%%% Private functions

make_cat(Name, Col, Desc) ->

#cat{name=Name, color=Col, description=Desc}.

And we can now try the brand new code:

1> c(kitty_gen_server).

{ok,kitty_gen_server}

2> rr(kitty_gen_server).

[cat]

3> {ok, Pid} = kitty_gen_server:start_link().

{ok,<0.253.0>}

4> Pid ! <<"Test handle_info">>.

Unexpected message: <<"Test handle_info">>

<<"Test handle_info">>

5> Cat = kitty_gen_server:order_cat(Pid, "Cat Stevens", white, "not actually a cat").

#cat{name = "Cat Stevens",color = white,

description = "not actually a cat"}

6> kitty_gen_server:return_cat(Pid, Cat).

ok

7> kitty_gen_server:order_cat(Pid, "Kitten Mittens", black, "look at them little paws!").

#cat{name = "Cat Stevens",color = white,

description = "not actually a cat"}

8> kitty_gen_server:order_cat(Pid, "Kitten Mittens", black, "look at them little paws!").

#cat{name = "Kitten Mittens",color = black,

description = "look at them little paws!"}

9> kitty_gen_server:return_cat(Pid, Cat).

ok

10> kitty_gen_server:close_shop(Pid).

"Cat Stevens" was set free.

ok

Oh and hot damn, it works!

So what can we say about this generic adventure? Probably the same generic stuff as before: separating the generic from the specific is a great idea on every point. Maintenance is simpler, complexity is reduced, the code is safer, easier to test and less prone to bugs. If there are bugs, they are easier to fix. Generic servers are only one of the many available abstractions, but they're certainly one of the most used ones. We'll see more of these abstractions and behaviours in the next chapters.

Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!

Rage Against The Finite-State Machines

What Are They?

A finite-state machine (FSM) is not really a machine, but it does have a finite number of states. I've always found finite-state machines easier to understand with graphs and diagrams. For example, the following would be a simplistic diagram for a (very dumb) dog as a state machine:

Here the dog has 3 states: sitting, barking or wagging its tail. Different events or inputs may force it to change its state. If a dog is calmly sitting and sees a squirrel, it will start barking and won't stop until you pet it again. However, if the dog is sitting and you pet it, we have no idea what might happen. In the Erlang world, the dog could crash (and eventually be restarted by its supervisor). In the real world that would be a freaky event, but your dog would come back after being ran over by a car, so it's not all bad.

Here's a cat's state diagram for a comparison:

This cat has a single state, and no event can ever change it.

Implementing the cat state machine in Erlang is a fun and simple task:

-module(cat_fsm).

-export([start/0, event/2]).

start() ->

spawn(fun() -> dont_give_crap() end).

event(Pid, Event) ->

Ref = make_ref(), % won't care for monitors here

Pid ! {self(), Ref, Event},

receive

{Ref, Msg} -> {ok, Msg}

after 5000 ->

{error, timeout}

end.

dont_give_crap() ->

receive

{Pid, Ref, _Msg} -> Pid ! {Ref, meh};

_ -> ok

end,

io:format("Switching to 'dont_give_crap' state~n"),

dont_give_crap().

We can try the module to see that the cat really never gives a crap:

1> c(cat_fsm).

{ok,cat_fsm}

2> Cat = cat_fsm:start().

<0.67.0>

3> cat_fsm:event(Cat, pet).

Switching to 'dont_give_crap' state

{ok,meh}

4> cat_fsm:event(Cat, love).

Switching to 'dont_give_crap' state

{ok,meh}

5> cat_fsm:event(Cat, cherish).

Switching to 'dont_give_crap' state

{ok,meh}

The same can be done for the dog FSM, except more states are available:

-module(dog_fsm).

-export([start/0, squirrel/1, pet/1]).

start() ->

spawn(fun() -> bark() end).

squirrel(Pid) -> Pid ! squirrel.

pet(Pid) -> Pid ! pet.

bark() ->

io:format("Dog says: BARK! BARK!~n"),

receive

pet ->

wag_tail();

_ ->

io:format("Dog is confused~n"),

bark()

after 2000 ->

bark()

end.

wag_tail() ->

io:format("Dog wags its tail~n"),

receive

pet ->

sit();

_ ->

io:format("Dog is confused~n"),

wag_tail()

after 30000 ->

bark()

end.

sit() ->

io:format("Dog is sitting. Gooooood boy!~n"),

receive

squirrel ->

bark();

_ ->

io:format("Dog is confused~n"),

sit()

end.

It should be relatively simple to match each of the states and transitions to what was on the diagram above. Here's the FSM in use:

6> c(dog_fsm).

{ok,dog_fsm}

7> Pid = dog_fsm:start().

Dog says: BARK! BARK!

<0.46.0>

Dog says: BARK! BARK!

Dog says: BARK! BARK!

Dog says: BARK! BARK!

8> dog_fsm:pet(Pid).

pet

Dog wags its tail

9> dog_fsm:pet(Pid).

Dog is sitting. Gooooood boy!

pet

10> dog_fsm:pet(Pid).

Dog is confused

pet

Dog is sitting. Gooooood boy!

11> dog_fsm:squirrel(Pid).

Dog says: BARK! BARK!

squirrel

Dog says: BARK! BARK!

12> dog_fsm:pet(Pid).

Dog wags its tail

pet

13> %% wait 30 seconds

Dog says: BARK! BARK!

Dog says: BARK! BARK!

Dog says: BARK! BARK!

13> dog_fsm:pet(Pid).

Dog wags its tail

pet

14> dog_fsm:pet(Pid).

Dog is sitting. Gooooood boy!

pet

You can follow along with the schema if you want (I usually do, it helps being sure that nothing's wrong).

That's really the core of FSMs implemented as Erlang processes. There are things that could have been done differently: we could have passed state in the arguments of the state functions in a way similar to what we do with servers' main loop. We could also have added an init and terminate functions, handled code updates, etc.

Another difference between the dog and cat FSMs is that the cat's events are synchronous and the dog's events are asynchronous. In a real FSM, both could be used in a mixed manner, but I went for the simplest representation out of pure untapped laziness. There are other forms of event the examples do not show: global events that can happen in any state.

One example of such an event could be when the dog gets a sniff of food. Once the smell food event is triggered, no matter what state the dog is in, he'd go looking for the source of food.

Now we won't spend too much time implementing all of this in our 'written-on-a-napkin' FSM. Instead we'll move directly to the gen_fsm behaviour.

Generic Finite-State Machines

The gen_fsm behaviour is somewhat similar to gen_server in that it is a specialised version of it. The biggest difference is that rather than handling calls and casts, we're handling synchronous and asynchronous events. Much like our dog and cat examples, each state is represented by a function. Again, we'll go through the callbacks our modules need to implement in order to work.

init

This is the same init/1 as used for generic servers, except the return values accepted are {ok, StateName, Data}, {ok, StateName, Data, Timeout}, {ok, StateName, Data, hibernate} and {stop, Reason}. The stop tuple works in the same manner as for gen_servers, and hibernate and Timeout keep the same semantics.

What's new here is that StateName variable. StateName is an atom and represents the next callback function to be called.

StateName

The functions StateName/2 and StateName/3 are placeholder names and you are to decide what they will be. Let's suppose the init/1 function returns the tuple {ok, sitting, dog}. This means the finite state machine will be in a sitting state. This is not the same kind of state as we had seen with gen_server; it is rather equivalent to the sit, bark and wag_tail states of the previous dog FSM. These states dictate a context in which you handle a given event.

An example of this would be someone calling you on your phone. If you're in the state 'sleeping on a Saturday morning', your reaction might be to yell in the phone. If your state is 'waiting for a job interview', chances are you'll pick the phone and answer politely. On the other hand, if you're in the state 'dead', then I am surprised you can even read this text at all.

Back to our FSM. The init/1 function said we should be in the sitting state. Whenever the gen_fsm process receives an event, either the function sitting/2 or sitting/3 will be called. The sitting/2 function is called for asynchronous events and sitting/3 for synchronous ones.

The arguments for sitting/2 (or generally StateName/2) are Event, the actual message sent as an event, and StateData, the data that was carried over the calls. sitting/2 can then return the tuples {next_state, NextStateName, NewData}, {next_state, NextStateName, NewData, Timeout}, {next_state, NextStateName, hibernate} and {stop, Reason, Data}.

The arguments for sitting/3 are similar, except there is a From variable in between Event and StateData. The From variable is used in exactly the same way as it was for gen_servers, including gen_fsm:reply/2. The StateData/3 functions can return the following tuples:

{reply, Reply, NextStateName, NewStateData}

{reply, Reply, NextStateName, NewStateData, Timeout}

{reply, Reply, NextStateName, NewStateData, hibernate}

{next_state, NextStateName, NewStateData}

{next_state, NextStateName, NewStateData, Timeout}

{next_state, NextStateName, NewStateData, hibernate}

{stop, Reason, Reply, NewStateData}

{stop, Reason, NewStateData}

Note that there's no limit on how many of these functions you can have, as long as they are exported. The atoms returned as NextStateName in the tuples will determine whether the function will be called or not.

handle_event

In the last section, I mentioned global events that would trigger a specific reaction no matter what state we're in (the dog smelling food will drop whatever it is doing and will instead look for food). For these events that should be treated the same way in every state, the handle_event/3 callback is what you want. The function takes the same arguments and returns the same values as StateName/2.

handle_sync_event

The handle_sync_event/4 callback is to StateName/3 what handle_event/2 is to StateName/2. It handles synchronous global events, takes the same parameters and returns the same kind of tuples as StateName/3.

Now might be a good time to explain how we know whether an event is global or if it's meant to be sent to a specific state. To determine this we can look at the function used to send an event to the FSM. Asynchronous events aimed at any StateName/2 function are sent with send_event/2, synchronous events to be picked up by StateName/3 are to be sent with sync_send_event/2-3.

The two equivalent functions for global events are send_all_state_event/2 and sync_send_all_state_event/2-3 (quite a long name).

code_change

This works exactly the same as it did for gen_servers except that it takes an extra state parameter when called like code_change(OldVersion, StateName, Data, Extra), and returns a tuple of the form {ok, NextStateName, NewStateData}.

terminate

This should, again, act a bit like what we have for generic servers. terminate/3 should do the opposite of init/1.

A Trading System Specification

It's time to put all of this in practice. Many Erlang tutorials about finite-state machines use examples containing telephone switches and similar things. It's my guess that most programmers will rarely have to deal with telephone switches for state machines. Because of that, we're going to look at an example which is more fitting for many developers: we'll design and implement an item trading system for some fictional and non-existing video game.

The design I have picked is somewhat challenging. Rather than using a broker through which players route items and confirmations (which, frankly, would be easier), we're going to implement a server where both players speak to each other directly (which would have the advantage of being distributable).

Because the implementation is tricky, I'll spend a good while describing it, the kind of problems to be faced and the ways to fix them.

First of all, we should define the actions that can be done by our players when trading. The first is asking for a trade to be set up. The other user should also be able to accept that trade. We won't give them the right to deny a trade, though, because we want to keep things simple. It will be easy to add this feature once the whole thing is done.

Once the trade is set up, our users should be able to negotiate with each other. This means they should be able to make offers and then retract them if they want. When both players are satisfied with the offer, they can each declare themselves as ready to finalise the trade. The data should then be saved somewhere on both sides. At any point in time, it should also make sense for any of the players to cancel the whole trade. Some pleb could be offering only items deemed unworthy to the other party (who might be very busy) and so it should be possible to backhand them with a well-deserved cancellation.

In short, the following actions should be possible:

- ask for a trade

- accept a trade

- offer items

- retract an offer

- declare self as ready

- brutally cancel the trade

Now, when each of these actions is taken, the other player's FSM should be made aware of it. This makes sense, because when Jim tells his FSM to send an item to Carl, Carl's FSM has to be made aware of it. This means both players can talk to their own FSM, which will talk to the other's FSM. This gives us something a bit like this:

The first thing to notice when we have two identical processes communicating with each other is that we have to avoid synchronous calls as much as possible. The reason for this is that if Jim's FSM sends a message to Carl's FSM and then waits for its reply while at the same time Carl's FSM sends a message over to Jim's FSM and waits for its own specific reply, both end up waiting for the other without ever replying. This effectively freezes both FSMs. We have a deadlock.

One solution to this is to wait for a timeout and then move on, but then there will be leftover messages in both processes' mailboxes and the protocol will be messed up. This certainly is a can of worms, and so we want to avoid it.

The simplest way to do it is to avoid all synchronous messages and go fully asynchronous. Note that Jim might still make a synchronous call to his own FSM; there's no risk here because the FSM won't need to call Jim and so no deadlock can occur between them.

When two of these FSMs communicate together, the whole exchange might look a bit like this:

Both FSMs are in an idle state. When you ask Jim to trade, Jim has to accept before things move on. Then both of you can offer items or withdraw them. When you are both declaring yourself ready, the trade can take place. This is a simplified version of all that can happen and we'll see all possible cases with more detail in the next paragraphs.

Here comes the tough part: defining the state diagram and how state transitions happen. Usually a good bit of thinking goes into this, because you have to think of all the small things that could go wrong. Some things might go wrong even after having reviewed it many times. Because of this, I'll simply put the one I decided to implement here and then explain it.

At first, both finite-state machines start in the idle state. At this point, one thing we can do is ask some other player to negotiate with us:

We go into idle_wait mode in order to wait for an eventual reply after our FSM forwarded the demand. Once the other FSM sends the reply, ours can switch to negotiate:

The other player should also be in negotiate state after this. Obviously, if we can invite the other, the other can invite us. If all goes well, this should end up looking like this:

So this is pretty much the opposite as the two previous state diagrams bundled into one. Note that we expect the player to accept the offer in this case. What happens if by pure luck, we ask the other player to trade with us at the same time he asks us to trade?

What happens here is that both clients ask their own FSM to negotiate with the other one. As soon as the ask negotiate messages are sent, both FSMs switch to idle_wait state. Then they will be able to process the negotiation question. If we review the previous state diagrams, we see that this combination of events is the only time we'll receive ask negotiate messages while in the idle_wait state. Consequently, we know that getting these messages in idle_wait means that we hit the race condition and can assume both users want to talk to each other. We can move both of them to negotiate state. Hooray.

So now we're negotiating. According to the list of actions I listed earlier, we must support users offering items and then retracting the offer:

All this does is forward our client's message to the other FSM. Both finite-state machines will need to hold a list of items offered by either player, so they can update that list when receiving such messages. We stay in the negotiate state after this; maybe the other player wants to offer items too:

Here, our FSM basically acts in a similar manner. This is normal. Once we get tired of offering things and think we're generous enough, we have to say we're ready to officialise the trade. Because we have to synchronise both players, we'll have to use an intermediary state, as we did for idle and idle_wait:

What we do here is that as soon as our player is ready, our FSM asks Jim's FSM if he's ready. Pending its reply, our own FSM falls into its wait state. The reply we'll get will depend on Jim's FSM state: if it's in wait state, it'll tell us that it's ready. Otherwise, it'll tell us that it's not ready yet. That's precisely what our FSM automatically replies to Jim if he asks us if we are ready when in negotiate state:

Our finite state machine will remain in negotiate mode until our player says he's ready. Let's assume he did and we're now in the wait state. However, Jim's not there yet. This means that when we declared ourselves as ready, we'll have asked Jim if he was also ready and his FSM will have replied 'not yet':

He's not ready, but we are. We can't do much but keep waiting. While waiting after Jim, who's still negotiating by the way, it is possible that he will try to send us more items or maybe cancel his previous offers:

Of course, we want to avoid Jim removing all of his items and then clicking "I'm ready!", screwing us over in the process. As soon as he changes the items offered, we go back into the negotiate state so we can either modify our own offer, or examine the current one and decide we're ready. Rinse and repeat.

At some point, Jim will be ready to finalise the trade too. When this happens, his final state machine will ask ours if we are ready:

What our FSM does is reply that we indeed are ready. We stay in the waiting state and refuse to move to the ready state though. Why is this? Because there's a potential race condition! Imagine that the following sequence of events takes place, without doing this necessary step:

This is a bit complex, so I'll explain. Because of the way messages are received, we could possibly only process the item offer after we declared ourselves ready and also after Jim declared himself as ready. This means that as soon as we read the offer message, we switch back to negotiate state. During that time, Jim will have told us he is ready. If he were to change states right there and move on to ready (as illustrated above), he'd be caught waiting indefinitely while we wouldn't know what the hell to do. This could not only happen the other way around, it could happen at the same time, giving us a double race condition! Ugh.

One way to solve this is by adding one layer of indirection (Thanks to David Wheeler). This is why we stay in wait mode and send 'ready!' (as shown in our previous state diagram). Here's how we deal with that 'ready!' message:

When we receive 'ready!', we send 'ready!' back again. This is to make sure that we won't have the 'double race condition' mentioned above. We then send an 'ack' message (and the Jim's FSM will do the same) before moving to ready state. The reason why this 'ack' message exists is due to some implementation details about synchronising clients. I've put it in the diagram for the sake of being correct, but I won't explain it until later. Forget about it for now. We finally managed to synchronise both players. Whew.

So now there's the ready state. This one is a bit special. Both players are ready and have basically given the finite-state machines all the control they need. This lets us implement a bastardized version of a two-phase commit to make sure things go right when making the trade official:

Our version (as described above) will be rather simplistic. Writing a truly correct two-phase commit would require a lot more code than what is necessary for us to understand finite-state machines.

Finally, we only have to allow the trade to be cancelled at any time. This means that somehow, no matter what state we're in, we're going to listen to the 'cancel' message from both sides and quit the transaction. It should also be common courtesy to let the other side know we're gone before leaving.

Alright! It's a whole lot of information to absorb at once. Don't worry if it takes a while to fully grasp it. It took a bunch of people to look over my protocol to see if it was right, and even then we all missed a few race conditions that I then caught a few days later when reviewing the code while writing this text. It's normal to need to read it more than once, especially if you are not used to asynchronous protocols. If this is the case, I fully encourage you to try and design your own protocol. Then ask yourself "what happens if two people do the same actions very fast? What if they chain two other events quickly? What do I do with messages I don't handle when changing states?" You'll see that the complexity grows real fast. You might find a solution similar to mine, possibly a better one (let me know if this is the case!) No matter the outcome, it's a very interesting thing to work on and our FSMs are still relatively simple.

Once you've digested all of this (or before, if you're a rebel reader), you can go to the next section, where we implement the gaming system. For now you can take a nice coffee break if you feel like doing so.

Game trading between two players

The first thing that needs to be done to implement our protocol with OTP's gen_fsm is to create the interface. There will be 3 callers for our module: the player, the gen_fsm behaviour and the other player's FSM. We will only need to export the player function and gen_fsm functions, though. This is because the other FSM will also run within the trade_fsm module and can access them from the inside:

-module(trade_fsm).

-behaviour(gen_fsm).

%% public API

-export([start/1, start_link/1, trade/2, accept_trade/1,

make_offer/2, retract_offer/2, ready/1, cancel/1]).

%% gen_fsm callbacks

-export([init/1, handle_event/3, handle_sync_event/4, handle_info/3,

terminate/3, code_change/4,

% custom state names

idle/2, idle/3, idle_wait/2, idle_wait/3, negotiate/2,

negotiate/3, wait/2, ready/2, ready/3]).

So that's our API. You can see I'm planning on having some functions being both synchronous and asynchronous. This is mostly because we want our client to call us synchronously in some cases, but the other FSM can do it asynchronously. Having the client synchronous simplifies our logic a whole lot by limiting the number of contradicting messages that can be sent one after the other. We'll get there. Let's first implement the actual public API according to the protocol defined above:

%%% PUBLIC API

start(Name) ->

gen_fsm:start(?MODULE, [Name], []).

start_link(Name) ->

gen_fsm:start_link(?MODULE, [Name], []).

%% ask for a begin session. Returns when/if the other accepts

trade(OwnPid, OtherPid) ->

gen_fsm:sync_send_event(OwnPid, {negotiate, OtherPid}, 30000).

%% Accept someone's trade offer.

accept_trade(OwnPid) ->

gen_fsm:sync_send_event(OwnPid, accept_negotiate).

%% Send an item on the table to be traded

make_offer(OwnPid, Item) ->

gen_fsm:send_event(OwnPid, {make_offer, Item}).

%% Cancel trade offer

retract_offer(OwnPid, Item) ->

gen_fsm:send_event(OwnPid, {retract_offer, Item}).

%% Mention that you're ready for a trade. When the other

%% player also declares being ready, the trade is done

ready(OwnPid) ->

gen_fsm:sync_send_event(OwnPid, ready, infinity).

%% Cancel the transaction.

cancel(OwnPid) ->

gen_fsm:sync_send_all_state_event(OwnPid, cancel).

This is rather standard; all these 'gen_fsm' functions have been covered before (except start/3-4 and start_link/3-4 which I believe you can figure out) in this chapter.

Next we'll implement the FSM to FSM functions. The first ones have to do with trade setups, when we first want to ask the other user to join us in a trade:

%% Ask the other FSM's Pid for a trade session

ask_negotiate(OtherPid, OwnPid) ->

gen_fsm:send_event(OtherPid, {ask_negotiate, OwnPid}).

%% Forward the client message accepting the transaction

accept_negotiate(OtherPid, OwnPid) ->

gen_fsm:send_event(OtherPid, {accept_negotiate, OwnPid}).

The first function asks the other pid if they want to trade, and the second one is used to reply to it (asynchronously, of course).

We can then write the functions to offer and cancel offers. According to our protocol above, this is what they should be like:

%% forward a client's offer

do_offer(OtherPid, Item) ->

gen_fsm:send_event(OtherPid, {do_offer, Item}).

%% forward a client's offer cancellation

undo_offer(OtherPid, Item) ->

gen_fsm:send_event(OtherPid, {undo_offer, Item}).

So, now that we've got these calls done, we need to focus on the rest. The remaining calls relate to being ready or not and handling the final commit. Again, given our protocol above, we have three calls: are_you_ready, which can have the replies not_yet or ready!:

%% Ask the other side if he's ready to trade.

are_you_ready(OtherPid) ->

gen_fsm:send_event(OtherPid, are_you_ready).

%% Reply that the side is not ready to trade

%% i.e. is not in 'wait' state.

not_yet(OtherPid) ->

gen_fsm:send_event(OtherPid, not_yet).

%% Tells the other fsm that the user is currently waiting

%% for the ready state. State should transition to 'ready'

am_ready(OtherPid) ->

gen_fsm:send_event(OtherPid, 'ready!').

The only functions left are those which are to be used by both FSMs when doing the commit in the ready state. Their precise usage will be described more in detail later, but for now, the names and the sequence/state diagram from earlier should be enough. Nonetheless, you can still transcribe them to your own version of trade_fsm:

%% Acknowledge that the fsm is in a ready state.

ack_trans(OtherPid) ->

gen_fsm:send_event(OtherPid, ack).

%% ask if ready to commit

ask_commit(OtherPid) ->

gen_fsm:sync_send_event(OtherPid, ask_commit).

%% begin the synchronous commit

do_commit(OtherPid) ->

gen_fsm:sync_send_event(OtherPid, do_commit).

Oh and there's also the courtesy function allowing us to warn the other FSM we cancelled the trade:

notify_cancel(OtherPid) ->

gen_fsm:send_all_state_event(OtherPid, cancel).

We can now move to the really interesting part: the gen_fsm callbacks. The first callback is init/1. In our case, we'll want each FSM to hold a name for the user it represents (that way, our output will be nicer) in the data it keeps passing on to itself. What else do we want to hold in memory? In our case, we want the other's pid, the items we offer and the items the other offers. We're also going to add the reference of a monitor (so we know to abort if the other dies) and a from field, used to do delayed replies:

-record(state, {name="",

other,

ownitems=[],

otheritems=[],

monitor,

from}).

In the case of init/1, we'll only care about our name for now. Note that we'll begin in the idle state:

init(Name) ->

{ok, idle, #state{name=Name}}.

The next callbacks to consider would be the states themselves. So far I've described the state transitions and calls that can be made, but We'll need a way to make sure everything goes alright. We'll write a few utility functions first:

%% Send players a notice. This could be messages to their clients

%% but for our purposes, outputting to the shell is enough.

notice(#state{name=N}, Str, Args) ->

io:format("~s: "++Str++"~n", [N|Args]).

%% Unexpected allows to log unexpected messages

unexpected(Msg, State) ->

io:format("~p received unknown event ~p while in state ~p~n",

[self(), Msg, State]).

And we can start with the idle state. For the sake of convention, I'll cover the asynchronous version first. This one shouldn't need to care for anything but the other player asking for a trade given our own player, if you look at the API functions, will use a synchronous call:

idle({ask_negotiate, OtherPid}, S=#state{}) ->

Ref = monitor(process, OtherPid),

notice(S, "~p asked for a trade negotiation", [OtherPid]),

{next_state, idle_wait, S#state{other=OtherPid, monitor=Ref}};

idle(Event, Data) ->

unexpected(Event, idle),

{next_state, idle, Data}.

A monitor is set up to allow us to handle the other dying, and its ref is stored in the FSM's data along with the other's pid, before moving to the idle_wait state. Note that we will report all unexpected events and ignore them by staying in the state we were already in. We can have a few out of band messages here and there that could be the result of race conditions. It's usually safe to ignore them, but we can't easily get rid of them. It's just better not to crash the whole FSM on these unknown, but somewhat expected messages.

When our own client asks the FSM to contact another player for a trade, it will send a synchronous event. The idle/3 callback will be needed:

idle({negotiate, OtherPid}, From, S=#state{}) ->

ask_negotiate(OtherPid, self()),

notice(S, "asking user ~p for a trade", [OtherPid]),

Ref = monitor(process, OtherPid),

{next_state, idle_wait, S#state{other=OtherPid, monitor=Ref, from=From}};

idle(Event, _From, Data) ->

unexpected(Event, idle),

{next_state, idle, Data}.

We proceed in a way similar to the asynchronous version, except we need to actually ask the other side whether they want to negotiate with us or not. You'll notice that we do not reply to the client yet. This is because we have nothing interesting to say, and we want the client locked and waiting for the trade to be accepted before doing anything. The reply will only be sent if the other side accepts once we're in idle_wait.

When we're there, we have to deal with the other accepting to negotiate and the other asking to negotiate (the result of a race condition, as described in the protocol):

idle_wait({ask_negotiate, OtherPid}, S=#state{other=OtherPid}) ->

gen_fsm:reply(S#state.from, ok),

notice(S, "starting negotiation", []),

{next_state, negotiate, S};

%% The other side has accepted our offer. Move to negotiate state

idle_wait({accept_negotiate, OtherPid}, S=#state{other=OtherPid}) ->

gen_fsm:reply(S#state.from, ok),

notice(S, "starting negotiation", []),

{next_state, negotiate, S};

idle_wait(Event, Data) ->

unexpected(Event, idle_wait),

{next_state, idle_wait, Data}.

This gives us two transitions to the negotiate state, but remember that we must use gen_fsm:reply/2 reply to our client to tell it it's okay to start offering items. There's also the case of our FSM's client accepting the trade suggested by the other party:

idle_wait(accept_negotiate, _From, S=#state{other=OtherPid}) ->

accept_negotiate(OtherPid, self()),

notice(S, "accepting negotiation", []),

{reply, ok, negotiate, S};

idle_wait(Event, _From, Data) ->

unexpected(Event, idle_wait),

{next_state, idle_wait, Data}.

Again, this one moves on to the negotiate state. Here, we must handle asynchronous queries to add and remove items coming both from the client and the other FSM. However, we have not yet decided how to store items. Because I'm somewhat lazy and I assume users won't trade that many items, simple lists will do it for now. However, we might change our mind at a later point, so it would be a good idea to wrap item operations in their own functions. Add the following functions at the bottom of the file with notice/3 and unexpected/2:

%% adds an item to an item list

add(Item, Items) ->

[Item | Items].

%% remove an item from an item list

remove(Item, Items) ->

Items -- Item.

Simple, but they have the role of isolating the actions (adding and removing items) from their implementation (using lists). We could easily move to proplists, arrays or whatever data structure without disrupting the rest of the code.

Using both of these functions, we can implement offering and removing items:

negotiate({make_offer, Item}, S=#state{ownitems=OwnItems}) ->

do_offer(S#state.other, Item),

notice(S, "offering ~p", [Item]),

{next_state, negotiate, S#state{ownitems=add(Item, OwnItems)}};

%% Own side retracting an item offer

negotiate({retract_offer, Item}, S=#state{ownitems=OwnItems}) ->

undo_offer(S#state.other, Item),

notice(S, "cancelling offer on ~p", [Item]),

{next_state, negotiate, S#state{ownitems=remove(Item, OwnItems)}};

%% other side offering an item

negotiate({do_offer, Item}, S=#state{otheritems=OtherItems}) ->

notice(S, "other player offering ~p", [Item]),

{next_state, negotiate, S#state{otheritems=add(Item, OtherItems)}};

%% other side retracting an item offer

negotiate({undo_offer, Item}, S=#state{otheritems=OtherItems}) ->

notice(S, "Other player cancelling offer on ~p", [Item]),

{next_state, negotiate, S#state{otheritems=remove(Item, OtherItems)}};

This is an ugly aspect of using asynchronous messages on both sides. One set of message has the form 'make' and 'retract', while the other has 'do' and 'undo'. This is entirely arbitrary and only used to differentiate between player-to-FSM communications and FSM-to-FSM communications. Note that on those coming from our own player, we have to tell the other side about the changes we're making.

Another responsibility is to handle the are_you_ready message we mentioned in the protocol. This one is the last asynchronous event to handle in the negotiate state:

negotiate(are_you_ready, S=#state{other=OtherPid}) ->

io:format("Other user ready to trade.~n"),

notice(S,

"Other user ready to transfer goods:~n"

"You get ~p, The other side gets ~p",

[S#state.otheritems, S#state.ownitems]),

not_yet(OtherPid),

{next_state, negotiate, S};

negotiate(Event, Data) ->

unexpected(Event, negotiate),

{next_state, negotiate, Data}.

As described in the protocol, whenever we're not in the wait state and receive this message, we must reply with not_yet. Were also outputting trade details to the user so a decision can be made.

When such a decision is made and the user is ready, the ready event will be sent. This one should be synchronous because we don't want the user to keep modifying his offer by adding items while claiming he's ready:

negotiate(ready, From, S = #state{other=OtherPid}) ->

are_you_ready(OtherPid),

notice(S, "asking if ready, waiting", []),

{next_state, wait, S#state{from=From}};

negotiate(Event, _From, S) ->

unexpected(Event, negotiate),

{next_state, negotiate, S}.

At this point a transition to the wait state should be made. Note that just waiting for the other is not interesting. We save the From variable so we can use it with gen_fsm:reply/2 when we have something to tell to the client.

The wait state is a funny beast. New items might be offered and retracted because the other user might not be ready. It makes sense, then, to automatically rollback to the negotiating state. It would suck to have great items offered to us, only for the other to remove them and declare himself ready, stealing our loot. Going back to negotiation is a good decision:

wait({do_offer, Item}, S=#state{otheritems=OtherItems}) ->

gen_fsm:reply(S#state.from, offer_changed),

notice(S, "other side offering ~p", [Item]),

{next_state, negotiate, S#state{otheritems=add(Item, OtherItems)}};

wait({undo_offer, Item}, S=#state{otheritems=OtherItems}) ->

gen_fsm:reply(S#state.from, offer_changed),

notice(S, "Other side cancelling offer of ~p", [Item]),

{next_state, negotiate, S#state{otheritems=remove(Item, OtherItems)}};

Now that's something meaningful and we reply to the player with the coordinates we stored in S#state.from. The next set of messages we need to worry about are those related to with synchronising both FSMs so they can move to the ready state and confirm the trade. For this one we should really focus on the protocol defined earlier.

The three messages we could have are are_you_ready (because the other user just declared himself ready), not_yet (because we asked the other if he was ready and he was not) and ready! (because we asked the other if he was ready and he was).

We'll start with are_you_ready. Remember that in the protocol we said that there could be a race condition hidden there. The only thing we can do is send the ready! message with am_ready/1 and deal with the rest later:

wait(are_you_ready, S=#state{}) ->

am_ready(S#state.other),

notice(S, "asked if ready, and I am. Waiting for same reply", []),

{next_state, wait, S};

We'll be stuck waiting again, so it's not worth replying to our client yet. Similarly, we won't reply to the client when the other side sends a not_yet to our invitation:

wait(not_yet, S = #state{}) ->

notice(S, "Other not ready yet", []),

{next_state, wait, S};

On the other hand, if the other is ready, we send an extra ready! message to the other FSM, reply to our own user and then move to the ready state:

wait('ready!', S=#state{}) ->

am_ready(S#state.other),

ack_trans(S#state.other),

gen_fsm:reply(S#state.from, ok),

notice(S, "other side is ready. Moving to ready state", []),

{next_state, ready, S};

%% DOn't care about these!

wait(Event, Data) ->

unexpected(Event, wait),

{next_state, wait, Data}.

You might have noticed that I've used ack_trans/1. In fact, both FSMs should use it. Why is this? To understand this we have to start looking at what goes on in the ready! state.

When in the ready state, both players' actions become useless (except cancelling). We won't care about new item offers. This gives us some liberty. Basically, both FSMs can freely talk to each other without worrying about the rest of the world. This lets us implement our bastardization of a two-phase commit. To begin this commit without either player acting, we'll need an event to trigger an action from the FSMs. The ack event from ack_trans/1 is used for that. As soon as we're in the ready state, the message is treated and acted upon; the transaction can begin.

Two-phase commits require synchronous communications, though. This means we can't have both FSMs starting the transaction at once, because they'll end up deadlocked. The secret is to find a way to decide that one finite state machine should initiate the commit, while the other will sit and wait for orders from the first one.

It turns out that the engineers and computer scientists who designed Erlang were pretty smart (well, we knew that already). The pids of any process can be compared to each other and sorted. This can be done no matter when the process was spawned, whether it's still alive or not, or if it comes from another VM (we'll see more about this when we get into distributed Erlang).

Knowing that two pids can be compared and one will be greater than the other, we can write a function priority/2 that will take two pids and tell a process whether it's been elected or not:

priority(OwnPid, OtherPid) when OwnPid > OtherPid -> true;

priority(OwnPid, OtherPid) when OwnPid < OtherPid -> false.

And by calling that function, we can have one process starting the commit and the other following the orders.

Here's what this gives us when included in the ready state, after receiving the ack message:

ready(ack, S=#state{}) ->

case priority(self(), S#state.other) of

true ->

try

notice(S, "asking for commit", []),

ready_commit = ask_commit(S#state.other),

notice(S, "ordering commit", []),

ok = do_commit(S#state.other),

notice(S, "committing...", []),

commit(S),

{stop, normal, S}

catch Class:Reason ->

%% abort! Either ready_commit or do_commit failed

notice(S, "commit failed", []),

{stop, {Class, Reason}, S}

end;

false ->

{next_state, ready, S}

end;

ready(Event, Data) ->

unexpected(Event, ready),

{next_state, ready, Data}.

This big try ... catch expression is the leading FSM deciding how the commit works. Both ask_commit/1 and do_commit/1 are synchronous. This lets the leading FSM call them freely. You can see that the other FSM just goes and wait. It will then receive the orders from the leading process. The first message should be ask_commit. This is just to make sure both FSMs are still there; nothing wrong happened, they're both dedicated to completing the task:

ready(ask_commit, _From, S) ->

notice(S, "replying to ask_commit", []),

{reply, ready_commit, ready, S};

Once this is received, the leading process will ask to confirm the transaction with do_commit. That's when we must commit our data:

ready(do_commit, _From, S) ->

notice(S, "committing...", []),

commit(S),

{stop, normal, ok, S};

ready(Event, _From, Data) ->

unexpected(Event, ready),

{next_state, ready, Data}.

And once it's done, we leave. The leading FSM will receive ok as a reply and will know to commit on its own end afterwards. This explains why we need the big try ... catch: if the replying FSM dies or its player cancels the transaction, the synchronous calls will crash after a timeout. The commit should be aborted in this case.

Just so you know, I defined the commit function as follows:

commit(S = #state{}) ->

io:format("Transaction completed for ~s. "

"Items sent are:~n~p,~n received are:~n~p.~n"

"This operation should have some atomic save "

"in a database.~n",

[S#state.name, S#state.ownitems, S#state.otheritems]).

Pretty underwhelming, eh? It's generally not possible to do a true safe commit with only two participants—a third party is usually required to judge if both players did everything right. If you were to write a true commit function, it should contact that third party on behalf of both players, and then do the safe write to a database for them or rollback the whole exchange. We won't go into such details and the current commit/1 function will be enough for the needs of this book.

We're not done yet. We have not yet covered two types of events: a player cancelling the trade and the other player's finite state machine crashing. The former can be dealt with by using the callbacks handle_event/3 and handle_sync_event/4. Whenever the other user cancels, we'll receive an asynchronous notification:

%% The other player has sent this cancel event

%% stop whatever we're doing and shut down!

handle_event(cancel, _StateName, S=#state{}) ->

notice(S, "received cancel event", []),

{stop, other_cancelled, S};

handle_event(Event, StateName, Data) ->

unexpected(Event, StateName),

{next_state, StateName, Data}.

When we do it we must not forget to tell the other before quitting ourselves:

%% This cancel event comes from the client. We must warn the other

%% player that we have a quitter!

handle_sync_event(cancel, _From, _StateName, S = #state{}) ->

notify_cancel(S#state.other),

notice(S, "cancelling trade, sending cancel event", []),

{stop, cancelled, ok, S};

%% Note: DO NOT reply to unexpected calls. Let the call-maker crash!

handle_sync_event(Event, _From, StateName, Data) ->

unexpected(Event, StateName),

{next_state, StateName, Data}.

And voilà! The last event to take care of is when the other FSM goes down. Fortunately, we had set a monitor back in the idle state. We can match on this and react accordingly:

handle_info({'DOWN', Ref, process, Pid, Reason}, _, S=#state{other=Pid, monitor=Ref}) ->

notice(S, "Other side dead", []),

{stop, {other_down, Reason}, S};

handle_info(Info, StateName, Data) ->

unexpected(Info, StateName),

{next_state, StateName, Data}.

Note that even if the cancel or DOWN events happen while we're in the commit, everything should be safe and nobody should get its items stolen.

Note: we used io:format/2 for most of our messages to let the FSMs communicate with their own clients. In a real world application, we might want something more flexible than that. One way to do it is to let the client send in a Pid, which will receive the notices sent to it. That process could be linked to a GUI or any other system to make the player aware of the events. The io:format/2 solution was chosen for its simplicity: we want to focus on the FSM and the asynchronous protocols, not the rest.

Only two callbacks left to cover! They're code_change/4 and terminate/3. For now, we don't have anything to do with code_change/4 and only export it so the next version of the FSM can call it when it'll be reloaded. Our terminate function is also really short because we didn't handle real resources in this example:

code_change(_OldVsn, StateName, Data, _Extra) ->

{ok, StateName, Data}.

%% Transaction completed.

terminate(normal, ready, S=#state{}) ->

notice(S, "FSM leaving.", []);

terminate(_Reason, _StateName, _StateData) ->

ok.

Whew.

We can now try it. Well, trying it is a bit annoying because we need two processes to communicate to each other. To solve this, I've written the tests in the file trade_calls.erl, which can run 3 different scenarios. The first one is main_ab/0. It will run a standard trade and output everything. The second one is main_cd/0 and will cancel the transaction halfway through. The last one is main_ef/0 and is very similar to main_ab/0, except it contains a different race condition. The first and third tests should succeed, while the second one should fail (with a crapload of error messages, but that's how it goes). You can try it if you feel like it.

That Was Quite Something

If you've found this chapter a bit harder than the others, I must remind you that it's entirely normal. I've just gone crazy and decided to make something hard out of the generic finite-state machine behaviour. If you feel confused, ask yourself these questions: Can you understand how different events are handled depending on the state your process is in? Do you understand how you can transition from one state to the other? Do you know when to use send_event/2 and sync_send_event/2-3 as opposed to send_all_state_event/2 and sync_send_all_state_event/3? If you answered yes to these questions, you understand what gen_fsm is about.

The rest of it with the asynchronous protocols, delaying replies and carrying the From variable, giving a priority to processes for synchronous calls, bastardized two-phase commits and whatnot are not essential to understand. They're mostly there to show what can be done and to highlight the difficulty of writing truly concurrent software, even in a language like Erlang. Erlang doesn't excuse you from planning or thinking, and Erlang won't solve your problems for you. It'll only give you tools.

That being said, if you understood everything about these points, you can be proud of yourself (especially if you had never written concurrent software before). You are now starting to really think concurrently.

Fit for the Real World?

In a real game, there is a lot more stuff going on that could make trading even more complex. Items could be worn by the characters and damaged by enemies while they're being traded. Maybe items could be moved in and out of the inventory while being exchanged. Are the players on the same server? If not, how do you synchronise commits to different databases?

Our trade system is sane when detached from the reality of any game. Before trying to fit it in a game (if you dare), make sure everything goes right. Test it, test it, and test it again. You'll likely find that testing concurrent and parallel code is a complete pain. You'll lose hair, friends and a piece of your sanity. Even after this, you'll have to know your system is always as strong as its weakest link and thus potentially very fragile nonetheless.

Don't Drink Too Much Kool-Aid:

While the model for this trade system seems sound, subtle concurrency bugs and race conditions can often rear their ugly heads a long time after they were written, and even if they've been running for years. While my code is generally bullet proof (yeah, right), you sometimes have to face swords and knives. Beware the dormant bugs.

Fortunately, we can put all of this madness behind us. We'll next see how OTP allows you to handle various events, such as alarms and logs, with the help of the gen_event behaviour.

50 10 30 5 90 20 40 2 25 10 8 0

Table of Contents

Table of Contents

Introduction

Starting Out

Starting Out (for real)

Modules

Syntax in Functions

Types (or lack thereof)

Recursion

Higher Order Functions

Errors and Exceptions

Functionally Solving Problems

A Short Visit to Common Data Structures

The Hitchhiker's Guide to Concurrency

More On Multiprocessing

Errors and Processes

Designing a Concurrent Application

What is OTP?

Clients and Servers

Rage Against The Finite-State Machines

Coming Soon

Introduction

About this tutorial

So what's Erlang?

Don't drink too much Kool-Aid

What you need to dive in

Where to get Help

Starting Out

The Shell

Shell Commands

Starting Out (for real)

Numbers

Invariable Variables

Atoms

Boolean Algebra & Comparison operators

Tuples

Lists!

List Comprehensions

Bit Syntax!

Binary Comprehensions

Modules

What are modules

Module Declaration

Compiling the code

More About Modules

Syntax in functions

Pattern Matching

Guards, Guards!

What the If!?

In Case ... of

Which to use?

Types (or lack thereof)

Dynamite-strong Typing

Type conversions

To Guard a Data Type

For Type Junkies

Recursion

Hello recursion!

Length

Length of a Tail Recursion

More recursive functions

Quick, Sort!

More than lists

Thinking recursively

Higher Order Functions

Let's get functional

Anonymous functions

Maps, filters, folds and more

Errors and Exceptions

Not so fast!

A Compilation of Errors

No, YOUR logic is wrong!

Run-time Errors

Raising Exceptions

Dealing with Exceptions

Wait, there's more!

Try a try in a tree

Functionally Solving Problems

Reverse Polish Notation Calculator

Heathrow to London

A Short Visit to Common Data Structures

Won't be too long, promised!

Records

Key-Value Stores

Arrays

A Set of Sets

Directed Graphs

Queues

End of the short visit

The Hitchhiker's Guide to Concurrency

Don't Panic

Concepts of Concurrency

Not Entirely Unlike Linear Scaling

So long and thanks for all the fish!

Errors and Processes

Links

It's a Trap!

Monitors

Naming Processes

More On Multiprocessing

State Your State

We love messages, but we keep them secret

Time Out

Selective Receives

Designing a Concurrent Application

Understanding the Problem

Defining the Protocol

Lay Them Foundations

An Event Module

The Event Server

Hot Code Loving

I Said, Hide Your Messages

A Test Drive

Adding Supervision

Namespaces (or lack thereof)

What is OTP?

It's The Open Telecom Platform!

The Common Process, Abstracted

The Basic Server

Specific Vs. Generic

Clients and Servers

Callback to the Future

.BEAM me up, Scotty!

Rage Against The Finite-State Machines

What Are They?

Generic Finite-State Machines

A Trading System Specification

Game trading between two players

That Was Quite Something

Fit for the Real World?