It’s not the holidays yet, but the RabbitMQ community has presents for you anyway! The RabbitMQ Kubernetes cluster operator is now open-sourced and developed in the open in GitHub. Also, Gavin Roy has a new Python app that migrates queues between types. Finally, a webinar on RabbitMQ consumers from Ayanda Dube, Head of RabbitMQ Engineering at Erlang Solutions.
Over time, we have seen the number of Kubernetes-related queries on our community mailing list and Slack channels soar. In this post we'd like to explain the basics of a DIY deployment of RabbitMQ on Kubernetes: what Kubernetes resources will be necessary, how to make sure RabbitMQ nodes use durable storage, how to approach configuration of sensitive values, and so on.
This month in RabbitMQ features the release of the RabbitMQ Cluster Kubernetes Operator, benchmarks and cluster sizing case studies by Jack Vanlightly (@vanlightly), and a write up of RabbitMQ cluster migration by Tobias Schoknecht (@tobischo), plus lots of other tutorials by our vibrant community!
| Be aware this post has out of date information |
|---|
| RabbitMQ now has Disaster Recovery capabilities in the commercial editions via the Warm Standby Replication feature |
In this post I am going to cover perhaps the most commonly asked question I have received regarding RabbitMQ in the enterprise.
How can I make RabbitMQ highly available and what architectures/practices are recommended for disaster recovery?
RabbitMQ offers features to support high availability and disaster recovery but before we dive straight in I’d like to prepare the ground a little. First I want to go over Business Continuity Planning and frame our requirements in those terms. From there we need to set some expectations about what is possible. There are fundamental laws such as the speed of light and the CAP theorem which both have serious impacts on what kind of DR/HA solution we decide to go with.
Finally we’ll look at the RabbitMQ features available to us and their pros/cons.
This month, Jack Vanlightly continues his blog series on Quorum Queues in RabbitMQ. Also, be sure to watch the replay of his related webinar.
Finally, Episode 5 of TGI RabbitMQ is out -- Gerhard Lazu walks us through how to run RabbitMQ on Kubernetes. Don’t miss!
The team was recently asked about whether and how quorum queues can offer the same message ordering guarantees as classic queues given that they will deliver messages from a local queue replica (leader or follower) when possible. Mirrored queues always deliver from the master (the leader), so delivering from any queue replica sounds like it could impact those guarantees.
That is the subject of this post. Be warned, this post is a technical deep dive for the curious and the distributed systems enthusiast. We’ll take a look at how quorum queues can deliver messages from any queue replica, leader or follower, without additional coordination (extra to Raft) but maintaining message ordering guarantees.
In the last post we started a sizing analysis of our workload using quorum queues. We focused on the happy scenario that consumers are keeping up meaning that there are no queue backlogs and all brokers in the cluster are operating normally. By running a series of benchmarks modelling our workload at different intensities we identified the top 5 cluster size and storage volume combinations in terms of cost per 1000 msg/s per month.
There are more tests to run to ensure these clusters can handle things like brokers failing and large backlogs accumulating during things like outages or system slowdowns.
All quorum queues are declared with the following properties:
The x-max-in-memory-length property forces the quorum queue to remove message bodies from memory as soon as it is safe to do. You can set it to a longer limit, this is the most aggressive - designed to avoid large memory growth at the cost of more disk reads when consumers do not keep up. Without this property message bodies are kept in memory at all times which can place memory growth to the point of memory alarms setting off which severely impacts the publish rate - something we want to avoid in this workload case study.
In a first post in this sizing series we covered the workload, the tests, and the cluster and storage volume configurations on AWS ec2. In this post we’ll run a sizing analysis with quorum queues. We also ran a sizing analysis on mirrored queues.
In this post we'll run the increasing intensity tests that will measure our candidate cluster sizes at varying publish rates, under ideal conditions. In the next post we'll run resiliency tests that measure whether our clusters can handle our target peak load under adverse conditions.
All quorum queues are declared with the following properties:
The x-max-in-memory-length property forces the quorum queue to remove message bodies from memory as soon as it is safe to do. You can set it to a longer limit, this is the most aggressive - designed to avoid large memory growth at the cost of more disk reads when consumers do not keep up. Without this property message bodies are kept in memory at all times which can place memory growth to the point of memory alarms setting off which severely impacts the publish rate - something we want to avoid in this workload case study.
In the last post we started a sizing analysis of our workload using mirrored queues. We focused on the happy scenario that consumers are keeping up meaning that there are no queue backlogs and all brokers in the cluster are operating normally. By running a series of benchmarks modelling our workload at different intensities we identified the top 5 cluster size and storage volume combinations in terms of cost per 1000 msg/s per month.
There are more tests to run to ensure these clusters can handle things like brokers failing and large backlogs accumulating during things like outages or system slowdowns.
In a first post in this sizing series we covered the workload, cluster and storage volume configurations on AWS ec2. In this post we’ll run a sizing analysis with mirrored queues.
The first phase of our sizing analysis will be assessing what intensities each of our clusters and storage volumes can handle easily and which are too much.
All tests use the following policy:
