Engineering Elixir Applications

Navigate Each Stage of Software Delivery with Confidence

by Ellie Fairholm and Josep Giralt D'Lacoste

The days of separate dev and ops teams are over—knowledge silos and the “throw it over the fence” culture they create are the enemy of progress. As an engineer or developer, you need to confidently own each stage of the software delivery process. This book introduces a new paradigm, BEAMOps, that helps you build, test, deploy, and debug BEAM applications. Create effective development and deployment strategies; leverage continuous improvement pipelines; and ensure environment integrity. Combine operational orchestrators such as Docker Swarm with the distribution, fault tolerance, and scalability of the BEAM, to create robust and reliable applications.

BEAMOps starts by building a solid foundation for your project. On the infrastructure side, see how to use Terraform before you even start coding to automate your deployment and operations. On the project management side, we show you how to use issues and milestones to simplify tracking; you’ll use this foundation as you go through the book, implementing each of the steps required to deploy a scalable Elixir application.

Now that you have a foundation, you can start building. Create a Phoenix LiveView application and explore mix releases. Make your deployments reliable with Docker. Continuously improve your codebase by implementing an efficient continuous deployment/integration pipeline with GitHub Actions. Scale the Phoenix LiveView application and operate a distributed BEAM system in production using AWS EC2 nodes, AWS load balancers, and a remote Docker swarm. Recognize which application metrics should be collected and monitored, and set alerts when certain thresholds are met to ensure that your application auto-scales. Ship less code more often, and ensure it works!

The pragmatic BEAMOps approach that we teach in this book—an extension of the well-known DevOps paradigm—will help you become a multidisciplinary developer who is empowered to own each stage of the software delivery process. After reading this book, you’ll understand how to apply the BEAMOps principles in your daily work, creating reliable, scalable, and easy to understand applications in a cooperative team environment.

• Elixir 1.16.0 (compiled with Erlang/OTP 26)
• Phoenix framework 1.7.0
• Erlang 26.2.1
• Terraform v1.7.1
• Docker v20.10.22
• Packer v1.9.0
• Docker Compose v2.21.0
• SOPS 3.8.1
• Age 1.1.1
• An AWS account
• AWS CLI v2.15.15
• A GitHub account
• GitHub CLI v2.42.1
• jq v1.7

• Errata, typos, suggestions

• Source Code (zip file)

Releases:

• P1.0 2024/12/18
• B7.0 2024/11/06
• B6.0 2024/07/30
• B5.0 2024/07/15

• Foreword
• Acknowledgments
• Going Back to the Root of the DevOps Paradigm
  • Is This Book for You?
  • What’s in This Book?
  • Online Resources
• Introduction to the Journey
  • Introducing the BEAMOps Paradigm
  • The Book’s Tech Stack
  • Set Up Your Environment
• Use Terraform to Create GitHub Issues and Milestones
  • Get to Know Terraform
  • Create Your First Resource Using the GitHub Provider
  • Examine the Terraform State
  • Terraform Variables
  • Resource Dependencies excerpt
  • What Have You Learned?
• Build and Dockerize a Phoenix LiveView Application
  • Releases and the mix release Command
  • Docker Fundamentals
  • Build Your First Docker Image
  • Run Your First Docker Container
  • Write Multistage Dockerfiles
  • What Have You Learned?
• Set Up Integration Pipelines with GitHub Actions excerpt
  • Mandatory CI Steps for a CI Pipeline
  • Build a Docker Image and Push to the GitHub Registry
  • Amend Your GitHub Workflow Triggers
  • What Have You Learned?
• The Dev Environment and Docker Compose
  • Rebuild Your Phoenix Application with Ecto
  • Run Your Multiservice Application Manually
  • Get to Know Docker Compose
  • Create Your First Docker Compose File
  • Get to Know Docker Swarm
  • What Have You Learned?
• The Production Environment and Packer
  • Create Your AWS Production Environment Manually
  • Import an Existing Infrastructure Resource with Terraform excerpt
  • Get to Know Packer
  • Build Your First Amazon Machine Image (AMI)
  • Putting Packer and Terraform Together
  • What Have You Learned?
• Continuous Deployment and Repository Secrets
  • Handle Sensitive Data with Docker Secrets
  • Encrypt Secret Data with SOPS
  • Deploy Manually to Understand Requirements
  • Turn Deploy Commands into a Convenience Script
  • Enable Continuous Deployment
  • What Have You Learned?
• Revise Your AWS Stack to Create a Multinode Swarm
  • Create and Distribute Multiple EC2 Instances
  • Create a Multinode Swarm
  • Automate the Initial Deployment of Your Application
  • What Have You Learned?
• Distributed Erlang
  • Get to Know Distributed Erlang
  • Manually Implement a Local Distributed Erlang Cluster
  • Automate the Creation of a Distributed Erlang Cluster in a Remote Docker Swarm
  • What Have You Learned?
• Autoscaling and Optimizing Your Deployment Strategy
  • Autoscale Your Phoenix LiveView Application
  • Add a Load Balancer to Your Auto Scaling Group
  • Minimize Downtime with Automatic Rollbacks
  • Clean Up Dangling Docker Resources
  • What Have You Learned?
• Instrument Your Application with Logs and Metrics
  • Design Your Instrumentation Architecture
  • Collect Your Application Logs
  • Configure Default Monitoring Metrics
  • What Have You Learned?
• Create a Custom PromEx Metric and Grafana Alert
  • Implement Your Own PromEx Metric and Grafana Dashboard
  • Add Alerts to Your Metrics Configuration
  • Deploy the BEAMOps Kanban Image to AWS in Production
  • What Have You Learned?
  • Wrapping Up the Journey

Ellie Fairholm is a full stack developer on the path to becoming a solutions architect. She excels at communication and believes that no topic is too difficult to learn. She is deeply passionate about making technology available to all, especially to those often underrepresented in the industry. She currently lives in the UK and is committed to working on projects that will one day change the world.

Pep G. D’Lacoste is the founder of BeamOps Software Consultancy, a company that takes a hands-on approach to simplifying BEAM projects and ensuring lasting sustainability by involving the whole team. He has been working with Elixir for 10 years and helped set up the Elixir Meetup in his home town of Barcelona. He now lives in the UK and is on a mission to empower developers and revolutionize the way technical teams operate.