Concurrent Data Processing in Elixir

by Svilen Gospodinov

Learn different ways of writing concurrent code in Elixir and increase your application’s performance, without sacrificing scalability or fault-tolerance. Most projects benefit from running background tasks and processing data concurrently, but the world of OTP and various libraries can be challenging. Which Supervisor and what strategy to use? What about GenServer? Maybe you need back-pressure, but is GenStage, Flow, or Broadway a better choice? You will learn everything you need to know to answer these questions, start building highly concurrent applications in no time, and write code that’s not only fast, but also resilient to errors and easy to scale.

Whether you are building a high-frequency stock trading application or a consumer web app, you need to know how to leverage concurrency to build applications that are fast and efficient. Elixir and the OTP offer a range of powerful tools, and this guide will show you how to choose the best tool for each job, and use it effectively to quickly start building highly concurrent applications.

Learn about Tasks, supervision trees, and the different types of Supervisors available to you. Understand why processes and process linking are the building blocks of concurrency in Elixir. Get comfortable with the OTP and use the GenServer behaviour to maintain process state for long-running jobs. Easily scale the number of running processes using the Registry. Handle large volumes of data and traffic spikes with GenStage, using back-pressure to your advantage. Create your first multi-stage data processing pipeline using producer, consumer, and producer-consumer stages. Process large collections with Flow, using MapReduce and more in parallel. Thanks to Broadway, you will see how easy it is to integrate with popular message broker systems, or even existing GenStage producers.

Start building the high-performance and fault-tolerant applications Elixir is famous for today.

Genres: Technology, Programming

170 pages, ebook

First published August 1, 2021

Reviews

[Mike Zornek · Rating 4 out of 5]

I liked the book a lot. Was a good overview book of concurrency in Elixir via the core language and then popular frameworks. Some might be disappointed it does not go deeper into Broadway, and that might be a worth book unto itself but I, who really enjoy short/focused books thought this book was great and serves a purpose for many.

[Héctor Iván Patricio Moreno · Rating 4 out of 5]

Trata sobre cómo hacer programas que llamaríamos "data-intensive", es decir, en los que su principal componente es la transformación de datos usando las características funcionales y de concurrencia de Elixir para hacer los procesos más eficientes y aprovechar todos los recursos de la computadora en donde están corriendo tus programas.

El libro abarca:
1. Tasks. Es la forma sencilla de crear procesos asíncronos con Elixir, una construcción encima de los procesos puros que te ayudará a levantar tareas concurrentes de manera fácil de entender y programar.
2. GenServers. Son procesos que pueden mantener un estado propio y que viven un largo tiempo, listos para atender peticiones de tareas para las que los programaste específicamente.
3. GenStage. Es una biblioteca de Elixir programada para crear de manera sencilla pipelines de procesamiento de datos que soporten "back pressure", un concepto que viene de la industria mecánica y automotriz que tiene que ver con limitar el flujo de datos desde las etapas de la cadena que hacen el trabajo, en vez de planear de antemano o tratar de adivinar la demanda.
4. Flow. Es una biblioteca de Elixir que constituye una abstracción sobre GenStage para crear de manera más sencilla pipelines de datos y al mismo una tiempo una manera de aprovechar los poderes de concurrencia de Elixir para operaciones como map, reduce, filter etc.
5. Broadway. Es una biblioteca de Elixir que permite crear pipelines de datos comenzando desde productores de eventos externos como RabbitMQ o Google PubSub. Es la versión más completa y con esteroídes de Flow, así como Flow es una abstracción sobre GenStage.

El libro explica todo con ejemplos bien pensados y para cada una de las herramientas pone un proyecto. Lo mejor es que después revisita algunos de estos proyectos introduciendo las nuevas herramientas que después te va enseñando.

Lo que no me gustó tanto del libro es que se siente como un tutorial gigantezco enfocado completamente en las herramientas y toca muy poco conceptos más profundos, pero es un muy buena lectura si tienes cosas que te estén presionando para sacar un proyecto enfocado en procesamiento de datos y quieres usar Elixir.

[Mohammad · Rating 5 out of 5]

Svilen took it from bare-bones concurrency primitives to the various important library with clear and crisp code samples to put the learning into practice and it will be a great guide to anyone who wants to take advantage of all their machines cores to the max.

[John · Rating 4 out of 5]

The parts about GenServer are explained in much more detail in the "Elixir in Action" book, here it is just a quick overview. But the chapters 3, 4 and 5 are a great intro to GenStage, Flow and Broadway, respectively.

[Euen · Rating 4 out of 5]

I enjoyed this book. It gave me a good overview of some of the tools to achieve concurrency in Elixi which I was unaware coming from Erlang like Flow and Broadway. And is very easy to follow.

[Hoon Wee · Rating 5 out of 5]

Explains very well about the topic - understanding concurrent Elixir ecosystem has never been this easy.