The way you organize source code in your projects has significant impact on the ergonomics of working with it. This is especially true as the project grows bigger in size and functionality. Properly structured project will make refactoring and changing functionality easier. A big ball of mud - will make your job more difficult.
In this subjective guide, I will walk you through what I found working well for me, when it comes to structuring Elixir and Phoenix projects.
A big ball of mud. One iteration at a time.
The worst thing you can do is to corner yourself into working with a big ball of mud: a system that lacks any perceivable architecture. I have done that several times, usually for the same reasons anyone else would: pressure from the business owners, lack of time, budget and resources. I also think certain tools I used in the past do make it really easy to structure projects this way.
The runtime of - say - Ruby is very much different from the runtime of Erlang. Things usually happen sequentially, and concurrency is rarely employed. As a result, there are no clear runtime boundaries, that would make it easier for us - developers - to draw the borders for our architectural components. Yes - we can force ourselves to make things better organized, create new directory structure, make sure our components interact only within designed borders - but there’s nothing in the language nor the runtime that would help us enforce it.
As a result, with increasing business pressure, we tend to bend the rules, put more and more of the code in the same place, slowly building unmanageable ball of mud - one step at a time.
Luckily for us, in the brave new world of Elixir we are given the tools, and it would be a shame not to use them.
Umbrella projects & OTP applications
Erlang comes with the concept of OTP applications. It is a simple - yet neat way to deploy, run, and manage multiple logical pieces of functionality within single BEAM instance, or a cluster of instances.
OTP applications can be either in a form of library applications - groups of modules exposing functionality to the rest of the cluster, or in a form of runtime entities - by starting own supervision trees.
While both are fine ways to group pieces of related functionality together, the second form is arguably more useful for structuring your application. It is future-proof, if you need to scale from single instance to a cluster of nodes, it also forces you to think in terms of business processes rather than entities defined by the data. This is something that struck me as odd as a newcomer from object-oriented programming, where we would - almost without thinking - structure our code in classes, wrapping some data around.
Elixir comes with a handy feature allowing you to easily use OTP applications in your project - an umbrella project.
To create an umbrella project, simply run:
mix new my_app --umbrella
And it will create an overall directory structure for you. Within
apps/ folder, you can generate a number of OTP applications (with
mix new appname or
mix phx.new appname). You can also specify
dependencies between those applications, to ensure they will be started
in the proper order (with
in_umbrella: true in respectful
Whenever you start the project from the top-level directory of the umbrella project, Elixir would build the dependency tree of the OTP applications, and start them up in an order that ensures each app has its dependencies started before it is brought up.
OTP applications are my favorite way of breaking down Elixir projects
into smaller chunks. I usually deal with web apps, and Phoenix is the
tool I use a lot, but the process starts by creating an umbrella
project. Then I would generate an
ui app for the front-end,
app for the back-end and
core app for the business logic, breaking it
down further as I go.
Having done that, you can still insist on building piles of mud within your individual OTP applications, or make one large pile of mud by tightly coupling them all together. The question remains if it is worth doing so in the long run.
OTP applications as microservices
Yay, a buzzword. Microservices. I could also squeeze in SOA I guess, but that might be the thing of the past already.
In previous section I mentioned that you can start the whole umbrella
project from its root directory, and that’s very much useful. What you
can also do, however, is to start the individual OTP applications by
going to their respectful directories in
apps/ and starting them
individually. The same applies to running tests or mix tasks - issuing
command to start those would run them within the scope of individual OTP
applications - not recursively within umbrella project.
Let’s consider a custom e-commerce platform, consisting of three user-facing components: user interface for customers to purchase goods, admin interface for store owners to manage the inventory, and API for external services to read and query product database. One could structure such system using Elixir and OTP applications to consist of the following components:
ui- our store front
admin- administrative dashboard
api- API layer for other machines to interact with
db- bottom-level database access layer
inventory- business logic related to querying and updating inventory
uploads- handling file uploads, such as product images
transactions- charging customers, interacting with payment gateway
Elixir comes with a handy mix task to print the dependencies between
apps in umbrella project. Simply run
mix app.tree --exclude logger --exclude elixir from top
directory of the project and you’d see:
==> uploads uploads ==> db db ==> checkout checkout └── db ==> inventory inventory └── db ==> transactions transactions └── db ==> api api └── inventory └── db ==> ui ui ├── transactions │ └── db ├── inventory │ └── db └── checkout └── db ==> admin admin ├── uploads └── inventory └── db
The tree above, being read from top to bottom, stopping at lines marked
==> also shows us in which order the individual OTP applications
will be started. As we can see, first the applications having no
dependencies - leafs in the tree - will be brought to life (
db), then applications directly depending on those, after that
another layer and so on.
Do not cross more than one layer boundary
The rule of thumb I use when structuring applications in this way, is to
allow the top-level applications to directly communicate with layer
below, but any communication crossing more than one line is forbidden.
As an example,
ui can retrieve products using
inventory, that would
query them directly in the
db. Reaching out from
db is -
however - strictly forbidden.
OTP applications can further help us with adding extra layers of
uploads application could be only a front-end to
aws_uploads application. If we keep the strict rule of not
crossing more than one border when communicating between applications,
we could then replace
google_cloud_uploads - without having to make a single change in
The other interesting thing we can do in such set up, is to test the
applications in isolation. If we want to test
starting a SQL database and populating the data, we could do that. We
can easily mock the interface of
db, using the new
mox for example. This gives us
freedom to develop the individual applications separately, which comes
in exceptionally handy during the times of refactoring. If you keep all
your code in one Elixir app, big refactoring tasks are difficult because
the application will not even compile until you make a significant
amount of changes, and it is a big conceptual overhead to keep track of all
the requirements at once. OTP applications make the code bases you work
Directory structure and naming things
I like to keep the directory structure aligned with naming of things. For our e-commerce application from the previous section, I would go with a top-level module namespacing schema that respects names of individual applications. This maps nicely to Erlang programmers practice of prefixing modules with application name too.
ui application, all of the modules would start with
prefix. For example
Since I do not keep any complex business logic in the Phoenix
applications, I would not create a separate
web subdirectory and a
UiWeb namespace. Instead, I would put all of my controllers in
apps/ui/lib/controllers. Everything in Phoenix apps is concerned with
“web”, there is no need to separate it any further from “non-web” stuff.
The rule of thumb that’s worth following is trying to keep the directory
structure as flat as it is possible and manageable. You
don’t really need to type more than necessary when opening/saving these
files - as long as there is sane number of files in a directory.
Ui.RegistrationController is in my humble opinion way better than
Ui.WebUi.Accounts.RegistrationController. There is no need to multiply
namespaces for the sake of multiplying namespaces.
If you keep the module names reasonably short, you can get away without
aliases in your code. If you don’t use aliases much, and especially
alias XXX, as: YYY construct, your code is more understandable.
I learned that there is also no need to wrap all the modules in the
whole project in a top-level namespace. There’s no need to have
Db.Account are good enough.
Server and client don’t belong together
One of the things I never liked in Elixir, and it seems I was in good
was the fact that in most of the tutorials and documentation, client and
server implementation for processes were coupled in the same module. In
GenServer would first define public API, then all
of the callbacks.
It does not have to be like that, and I think it is hurtful in most cases if you write such code.
If you have layers of OTP applications in place, you may want to expose
certain features in form of servers, usually using
similar OTP behaviour. Going back to our e-commerce example, one could
expose file uploads functionality as such service. I mentioned before,
uploads can be further split into
The good practice in such case would be to put the client API in the
uploads, and create processes for services in
implementing only life cycle functions and callbacks.
This breakdown will come in handy when you think of more complex
deployment scenarios. For example, you could deploy your
aws_uploads applications to two different nodes in the cluster. The
admin would also need to load the code from the
uploads, but it may be completely unaware of the implementation
details of concrete uploads mechanism defined in
on remote node.
Elixir comes with super cool concept of OTP applications, inherited from Erlang. Do use it. You can solve any problem with extra layers of abstractions (except the problem of too many layers of abstractions)!
Post by Hubert Łępicki
Hubert is partner at AmberBit. Rails, Elixir and functional programming are his areas of expertise.