Engineering Elixir Applications

Navigate Each Stage of Software Delivery with Confidence

by Ellie Fairholm and Josep Giralt D'Lacoste

The days of separate dev and ops teams are over—knowledge silos and the “throw it over the fence” culture they create are the enemy of progress. As an engineer or developer, you need to confidently own each stage of the software delivery process. This book introduces a new paradigm, BEAMOps, that helps you build, test, deploy, and debug BEAM applications. Create effective development and deployment strategies; leverage continuous improvement pipelines; and ensure environment integrity. Combine operational orchestrators such as Docker Swarm with the distribution, fault tolerance, and scalability of the BEAM, to create robust and reliable applications.

BEAMOps starts by building a solid foundation for your project. On the infrastructure side, see how to use Terraform before you even start coding to automate your deployment and operations. On the project management side, we show you how to use issues and milestones to simplify tracking; you’ll use this foundation as you go through the book, implementing each of the steps required to deploy a scalable Elixir application.

Now that you have a foundation, you can start building. Create a Phoenix LiveView application and explore mix releases. Make your deployments reliable with Docker. Continuously improve your codebase by implementing an efficient continuous deployment/integration pipeline with GitHub Actions. Scale the Phoenix LiveView application and operate a distributed BEAM system in production using AWS EC2 nodes, AWS load balancers, and a remote Docker swarm. Recognize which application metrics should be collected and monitored, and set alerts when certain thresholds are met to ensure that your application auto-scales. Ship less code more often, and ensure it works!

The pragmatic BEAMOps approach that we teach in this book—an extension of the well-known DevOps paradigm—will help you become a multidisciplinary developer who is empowered to own each stage of the software delivery process. After reading this book, you’ll understand how to apply the BEAMOps principles in your daily work, creating reliable, scalable, and easy to understand applications in a cooperative team environment.

• Elixir 1.16.0 (compiled with Erlang/OTP 26)
• Phoenix framework 1.7.0
• Erlang 26.2.1
• Terraform v1.7.1
• Docker v20.10.22
• Packer v1.9.0
• Docker Compose v2.21.0
• SOPS 3.8.1
• Age 1.1.1
• An AWS account
• AWS CLI v2.15.15
• A GitHub account
• GitHub CLI v2.42.1
• jq v1.7

• Errata, typos, suggestions

• Source Code (zip file)

Releases:

• B7.0 2024/11/06
• B6.0 2024/07/30
• B5.0 2024/07/15
• B4.0 2024/06/24

• Foreword
• Going Back to the Root of the DevOps Paradigm
  • Is This Book For You?
  • What’s In This Book?
  • Online Resources
• Introduction to the Journey
  • Introducing the BEAMOps Paradigm
  • The Book’s Tech Stack
  • Setting Up Your Environment
• Using Terraform to Create GitHub Issues and Milestones
  • Getting to Know Terraform
  • Using the GitHub Provider and Creating Your First Resource
  • Examining the Terraform State
  • Using Variables
  • Understanding Resource Dependencies excerpt
  • What Have You Learned?
• Building and Dockerizing a Phoenix Live View Application
  • Understanding Releases and the mix release Command
  • Understanding Docker Fundamentals
  • Building Your First Docker Image
  • Running Your First Docker Container
  • Writing Multi-Stage Dockerfiles
  • What Have You Learned?
• Setting Up Integration Pipelines with GitHub Actions excerpt
  • Understanding the Mandatory CI Steps for a CI Pipeline
  • Building a Docker Image and Pushing to the GitHub Registry
  • Amending Your GitHub Workflow Triggers
  • What Have You Learned?
• The Dev Environment with Docker Compose
  • Rebuilding Your Phoenix Application with Ecto
  • Manually Running Your Multi-Service Application
  • Introducing Docker Compose
  • Creating Your First Docker Compose File
  • Introducing Docker Swarm
  • What Have You Learned?
• The Production Environment and Packer
  • Creating Your AWS Production Environment Manually
  • Importing an Existing Infrastructure Resource with Terraform excerpt
  • Introducing Packer
  • Building Your First Amazon Machine Image (AMI)
  • Putting Packer and Terraform Together
  • What Have You Learned?
• Continuous Deployment and Repository Secrets
  • Handling Sensitive Data with Docker Secrets
  • Encrypting Secret Data with SOPS
  • Deploying Manually to Understand Deployment Requirements
  • Turning Your Deploy Commands into a Convenience Script
  • Enabling Continuous Deployment
  • What Have You Learned?
• Revising Your AWS Stack to Create a Multi-Node Swarm
  • Creating and Distributing Multiple EC2 Instances Across Multiple Availability Zones
  • Creating a Multi-Node Swarm
  • Using Terraform to Automate the Initial Deployment of Your Application
  • What Have You Learned?
• Distributed Erlang
  • Understanding Distributed Erlang
  • Manually Implementing a Local Distributed Erlang Cluster
  • Automating the Creation of a Distributed Erlang Cluster in a Remote Docker Swarm
  • What Have You Learned?
• Autoscaling and Optimizing Your Deployment Strategy
  • Autoscaling Your Phoenix Live View Application
  • Adding a Load Balancer to Your Auto Scaling Group
  • Minimizing Downtime with Automatic Rollbacks
  • Executing Housekeeping Tasks When Deploying Your Application to Clean Up Your Dangling Docker Resources
  • What Have You Learned?
• Instrumenting Your Application with Logs and Metrics
  • First Steps in Designing Your Instrumentation Architecture
  • Collecting Your Application Logs
  • Adding Default Monitoring Metrics with Promex and Prometheus
  • Implementing Your Own Metric and Uploading it to Grafana Using PromEx
  • Adding Alerts to Your Metrics Configuration
  • What Have You Learned?
• Conclusion to the Journey

Ellie Fairholm is a full stack developer on the path to becoming a solutions architect. She excels at communication and believes that no topic is too difficult to learn. She is deeply passionate about making technology available to all, especially to those often underrepresented in the industry. She currently lives in the UK and is committed to working on projects that will one day change the world.

Pep G. D’Lacoste is the founder of BeamOps Software Consultancy, a company that takes a hands-on approach to simplifying BEAM projects and ensuring lasting sustainability by involving the whole team. He has been working with Elixir for 10 years and helped set up the Elixir Meetup in his home town of Barcelona. He now lives in the UK and is on a mission to empower developers and revolutionize the way technical teams operate.