I really like RabbitMQ!
If you have not come across RabbitMQ before, it's a messaging system that enables you to seperate your system out to communicate and send data between services or even other systems. If you want to ensure your system meets your reliability, scalability and performance requirements for today but also for tomorrow, then a messaging system like RabbitMQ is a great shout.
In a nutshell, RabbitMQ provides a way of architecting your system so that you can write small services that focus on one job that are scalable to meet your performance requirements.
What does that mean?
Well, when you are designing a large system, there are going to be different parts of the system that will want to communicate with each other. For example, sending commands/actions from one system to another or even just sending notifications.
So, instead of writing your traditional, monolithic application, which is just one big codebase, message queues enable you to write small services. These services are usually lightweight and focus on one job! So, for example, you could have a ProcessOrder Service and an AlarmHandler Service, etc.
But how do these services talk to each other? Well, that’s where message queues and RabbitMQ comes in.
RabbitMQ is the glue that enables asynchronous communication between each of our business layer components. At its core, RabbitMQ is a FIFO (First in, First out) message queue and its fast! - The messages are transient meaning they are not stored forever and at some point are picked up for processing. In addition to all this, RabbitMQ can guarantee that messages will be delivered to the destination no matter what happens!
Writing software this way has got lots of benefits. If one of the services is under heavy load, we can spin up extra copies of the service (even across more machines) to cope with the load (scalability). This is called horizontally scaling and means each service can scale independently of one another.
More importantly, because these services are independent from one another (and not in some monolithic application), we can alter the functionality of one service in isolation of the others knowing our change will only affect this service. This makes for a more maintainable system.
If you need to connect lots of things together, have great performance and need something which is robust and production-ready you can't go wrong. As I said, I really like RabbitMQ!
Best of all, and this is the amazing bit... it's free!
Installing RabbitMQ on Windows
RabbitMQ is available for install on both Linux and Windows. As I have recently installed RabbitMQ on a Windows system, I thought I would write up the installation process while it was still fresh! My perception is that RabbitMQ is better supported on Linux than it is on Windows and there are a couple of gotchas when installing on Windows.
Step 1 - Set Base Directory
Okay, before even installing RabbitMQ, the first thing we need to set up is the RABBITMQ_BASE environment variable. The reason for this, is by default the RabbitMQ database/queues are saved to disk on the C: drive, in the directory %APPDATA%\RabbitMQ. As I mention in this StackOveflow answer, this just seems wrong to me. So, just to make sure we have space, we set this variable up to force the database to be held on the D: drive.
Step 2 - Installation
With the RABBITMQ_BASE environment variable setup, we can now actually kick off the installer process. Now, RabbitMQ depends on Erlang which means we are going to need to install Erlang fist before RabbitMQ.
At the time of this blog post, I am downloading RabbitMQ Version 3.7.8 and this version supports Erlang 19.3.6 up to 21.0. You can find out which version of Erlang RabbitMQ supports by going here.
Once installed, you will be able to fire up the RabbitMQ Command Prompt
To confirm everything installed fine, run the status command from the RabbitMQ command line
rabbitmqctl status
Step 3 - Setup Admin User
By default, RabbitMQ comes with the guest/guest credentials. Let's first take the time to setup an administrator account. From the RabbitMQ command line run the following commands:
rabbitmqctl add_user adminuser password1 rabbitmqctl set_user_tags adminuser administrator rabbitmqctl set_permissions -p / adminuser ".*" ".*" ".*"
You will be able to change the password from the web interface, but if you need to change the password from the command line, type:
rabbitmqctl change_password adminuser password2
Step 4 - Enable the Web Management Interface
By default RabbitMQ gets installed with everything off. Next, we need to enable the web interface.
rabbitmq-plugins enable rabbitmq_management
From the same machine you should now be able to log onto the management interface by visiting http://localhost:15672 and logging on with the admin user we previously created.
If at any point you need to disable the web interface, you can do so with the following command:
rabbitmq-plugins disable rabbitmq_management
Step 5 - Open Ports on Windows Firewall
To be able to access the management interface externally, we need to open some ports on the firewall. Run the following commands:
netsh advfirewall firewall add rule name="RabbitMQ_15672" dir=in action=allow protocol=TCP localport=15672 netsh advfirewall firewall add rule name="RabbitMQ_15672" dir=out action=allow protocol=TCP localport=15672
Whilst we are here, we also need to open port 5672 to allow clients to access RabbitMQ and add or remove work to the queues:
netsh advfirewall firewall add rule name="RabbitMQ_5672" dir=in action=allow protocol=TCP localport=5672 netsh advfirewall firewall add rule name="RabbitMQ_5672" dir=out action=allow protocol=TCP localport=5672
If you need to delete these rules at a later date, run the following command:
netsh advfirewall firewall delete rule name="RabbitMQ_5672"name protocol=TCP localport=5672
The next thing you need to do is open the management port 15672 on the Windows firewall, so that you can access the management interface from an external machine.
Step 6 - Setting up the config file
The next thing I did was to setup the config file. By default, RabbitMQ doesn't come with a config file so you will have to set one up yourself. You can find an example of the new rabbitmq.conf file here. If you read the documentation, it talks about setting up a new environment variable named RABBITMQ_CONFIG_FILE. However, it looks like there is a bug in the windows version of RabbitMQ as I could not get it to pick up the config file from this location.
So, as a work-around copy the config file to the RABBITMQ_BASE directory (which in my case is D:\RabbitMQBase).
Now, to get RabbitMQ to pick up the config file on Windows, we also need to additionally stop and restart the RabbitMQ windows service using the following rabbit commands:
rabbitmq-service stop rabbitmq-service remove rabbitmq-service install rabbitmq-service start
When we then take a look at the log file after our restart, we can see that it has picked up our config file:
Now that we have the config file setup, we can specify things such as maximum memory usage and disk limits. For example, the following config setting pins the maximum memory usage to 1GB of RAM.
vm_memory_high_watermark.absolute = 1024MB
At this point, that's it! We have successfully installed RabbitMQ on our windows system!
RabbitMQ Diagnostics & Monitoring
Now, because a messaging system such as RabbitMQ is such a crucial component of the system, it would be good if we could be notified when there was a problem.
For example:
- One of the Rabbit MQ nodes goes down
- One of the queues stops receiving data
- There are no consumers moving messages off a queue
- The number of messages in a queue exceeds a certain number
That kinda thing. Thankfully, RabbitMQ has an API that allows you to query the service. In a browser, you can query the interface and get JSON data back about the service. For example:
Here is an example of the different endpoints you can programmatically call:
http://adminuser:password1@127.0.0.1:15672/api/ http://adminuser:password1@127.0.0.1:15672/api/queues http://adminuser:password1@127.0.0.1:15672/api/queues/%2f/queue-name http://adminuser:password1@127.0.0.1:15672/api/connections http://adminuser:password1@127.0.0.1:15672/api/consumers
For a full list of all the available API endpoints, see this page a href="https://pulse.mozilla.org/api/">here/a>
Wrapping Up
That's it! Job done!
