Tag Archives: rabbitmq

Things have gotten out of hand
Things have gotten out of hand

My desk today - this doesn't even take into account the 150 tabs I had open in multiple instances of 2-3 browsers on each machine.

What happened...

Well, it grew organically since yesterday, or maybe it was the day before. Hard to remember. At any rate, my goal was to install RabbitMQ. At least, that was the basic goal. First, just get it installed and running.

My secondary goal was to get it actually sending and receiving messages locally. By this I mean RabbitMQ installed on a computer, and a Rails app running on the same computer, communicating with each other.

The third and goal was to get it talking between 2 Rails apps on 2 different computers. So, I'll start with my primary goal. This I accomplished late yesterday actually, and really wasn't very difficult.

Phase 1

First, if you've never worked with RabbitMQ before, I highly recommend you read the docs. They have pretty solid documentation on the whole install process, along with several great tutorials to explain the language, behaviors, and concepts to beginners like myself.

I'm on a mac (obviously from above I suppose), and since I have set up my environment with Homebrew, I opt to install most new tech the same way, when and where possible. So I chose to install using Homebrew, though there is a mac tarball you can download and unpack if that's more to your liking.

The docs cover it well, but it's basically 2 commands:

brew update
brew install rabbitmq

 

Once the install has run, you should read the output from Homebrew. It's nice if you do, possibly a head-scratcher if you don't. They tell you neat things, like where to access the management interface, how to launch the server at startup, how to launch the server now, etc.

==> Caveats
Management Plugin enabled by default at http://localhost:15672 

Bash completion has been installed to:
 /usr/local/etc/bash_completion.d 

To have launchd start rabbitmq at login:
 ln -sfv /usr/local/opt/rabbitmq/*.plist ~/Library/LaunchAgents
Then to load rabbitmq now:
 launchctl load ~/Library/LaunchAgents/homebrew.mxcl.rabbitmq.plist
Or, if you don't want/need launchctl, you can just run:
 rabbitmq-server

 

Additionally, you should edit your .bash_profile or .profile file (read my other post on this), as that will enable you to run the server commands correctly from terminal.

Once that's done, start your RabbitMQ server up in terminal with the rabbitmq-server command. Then fire up http://localhost:15672. You should be able to access the management interface - the default user and password is guest/guest.

Screen Shot 2016-02-23 at 10.17.17 AM

You should see something similar to this, though there will most likely be no activity yet. All queues, connections should be zero.

So at this point you have the management interface up, and I tend to go in baby steps, to make sure all's well as it appears.

So first, head back to terminal, and tail the logs to see if there's any action on them.

 

tail -f /usr/local/var/log/rabbitmq/rabbit\@localhost.log

 

Next, in another terminal tab (hopefully you've got a couple open by now - one for logs, one for server, one or more for your application instances, perhaps one for miscellaneous - file tree stuff... maybe a dog running in another...) where your RabbitMQ server is running, try shutting it down. Verify you see something in the logs, and head back to the management interface and refresh it - it should come back with a blank page - not available.

So, with all that working, I decided I was good to move on to the next phase, which is implementing some code to interact with RabbitMQ.

 

Phase 2

Remember from above, my primary goal was to get RabbitMQ running. That goal complete, it was time to move on to the Rails part of the project. If you're into Python, Java, or some other flavor, this post probably won't be as interesting for you, though you may still get something out of it.

For this, I cd to a directory where I want my app to be stored - I have quite a few "site" folders (all projects wanting attention) - and I run some basic commands to get a Rails app up and running. One thing to note - make sure to include  gem 'bunny', '2.2.2'  in your Gemfile.

 

gem install rails --no-ri --no-rdoc
rails new rabbit_tester . && cd rabbit_tester
rvm use ruby-2.3.0
rvm gemset create rabbit
echo 'ruby-2.3.0' >> '.ruby-version'
echo 'rabbit' >> '.ruby-gemset'
rvm use ruby-2.3.0@rabbit
gem install bundler
bundle

So, that spits out a bunch of shit in between commands, while I grab some coffee and then continue. The docs say to set up 2 ruby files in isolation, which works fine for a basic litmus test (I did this too in fact), but I had wanted something where I could run either a rake command, or call a class method, or even better, load up a restful api in the browser which triggers the call to the RabbitMQ. I chose the last method, and grabbed the code from their tutorials to incorporate into my app.

My overall plan is to create a model with 2 methods, mapped to the send and receive functions. Then I'll create a couple of routes, map them to controller methods, and have these controller methods call the respective model's methods. This way I can fire up the page in the browser and watch the call go out in the logs, see it in the management interface, and see any output in my app in the browser as well.  My first step is to generate a model, run the required database migration.

rails g model Rabbitt name:string color: string language:string speed:integer
rake db:migrate

 

Now I am going to create 2 class methods, one for the sending of messages and one for receiving. Once created, I create an instance of my class (just for laughs), pull in some of the code from the RabbitMQ docs and tweak it a bit.

rails c
Rabbit.create(name: 'Brier Sr.', color: 'purple', language: 'High Rabbit', speed: 175)

 

class Rabbitt < ActiveRecord::Base

  class << self
    def send_amqp
      puts "beginning send method..."
      conn = Bunny.new
      conn.start

      ch = conn.create_channel
      q = ch.queue("rabbitt events")
      ch.default_exchange.publish("hello from #{Rabbitt.first.name} ....", routing_key: q.name)
      puts "Sent a notice from #{Rabbitt.first.name} ----------------"

      conn.close
    end

    def receive_amqp
      conn = Bunny.new
      conn.start

      ch = conn.create_channel
      q = ch.queue("rabbitt events")
      q.subscribe(block: true) do |info, props, body|
        puts "message received! #{body} ----------------"
        puts "props: #{props.inspect} ================"
        puts "info: #{info.inspect} ================"

        info.consumer.cancel

      end
    end

  end # end class methods ........
end

Cool, so that's done. Next is to generate a controller and the necessary view templates.

rails g controller rabbitts index send

For the send method, I modify the routes a bit:

resources :rabbitts do
  collection do
    get 'send/message', to: 'rabbitts#send'
  end
end

Right, at this point, I'm ready to fire up my server again and see if any of this generates anything. First I load up my rails app from terminal, and then in another terminal tab, I start my RabbitMQ server. Finally, I open a third terminal tab to tail the RabbitMQ logs.

rails s -p1234
rabbitmq-server
tail -f /usr/local/var/log/rabbitmq/rabbit\@localhost.log

In my first tab, I browse to localhost:1234/rabbitts/send/message , and success! In the Rails logs I see

beginning send method...
Rabbitt Load (0.2ms)  SELECT  "rabbitts".* FROM "rabbitts"  ORDER BY "rabbitts"."id" ASC LIMIT 1
CACHE (0.0ms)  SELECT  "rabbitts".* FROM "rabbitts"  ORDER BY "rabbitts"."id" ASC LIMIT 1
Sent a notice from Brier Sr. ----------------

In a second tab, when I browse  to localhost:1234/rabbitts, I see the refresh icon in the tab is incomplete, as if it's continually loading. This is because it's waiting for a response from RabbitMQ, and the connection has been left open. While there's nothing in the UI of the page (I didn't actually modify the view yet), in the Rails logs, I see what I was hoping for:

Started GET "/rabbitts" for 192.168.1.11 at 2016-02-22 20:15:33 -0800
Processing by RabbittsController#index as HTML
message received! hello from Brier Sr. .... ----------------
props: {:content_type=>"application/octet-stream", :delivery_mode=>2, :priority=>0} ================
info: {:consumer_tag=>"bunny-1456200933000-722471154864", :delivery_tag=>#<Bunny::VersionedDeliveryTag:0x007fd414073858 @tag=1, @version=0>, :redelivered=>false, :exchange=>"", :routing_key=>"rabbitt events", :consumer=>#<Bunny::Consumer:70274422964960 @channel_id=1 @queue=rabbitt events> @consumer_tag=bunny-1456200933000-722471154864 @exclusive= @no_ack=true>, :channel=>#<Bunny::Channel:70274422981440 @id=1 @connection=#<Bunny::Session:0x7fd4140b04d8 guest@127.0.0.1:5672, vhost=/, addresses=[127.0.0.1:5672]>>} ================
  Rabbitt Load (0.2ms)  SELECT "rabbitts".* FROM "rabbitts"
  Rendered rabbitts/index.html.erb within layouts/application (2.6ms)
Completed 200 OK in 398ms (Views: 381.5ms | ActiveRecord: 0.2ms)

My final two checks are to the RabbitMQ logs and the management interface. The logs show connection accepted,  accepting AMQP connection <0.8837.4> (127.0.0.1:62341 -> 127.0.0.1:5672) and the management interface is interesting. I notice that in the global counts, I see 1 queue, and 1 consumer, and 1 connection, yet no messages.

I rerun the test, this time watching the management interface, and sure enough, the message comes in, and gets queued (in the 'queued messages' chart at the top), and as soon as I refresh the /rabbitts view, the message is consumed and removed from the queue.

So far, it looks as though everything is functioning as I had hoped.

Phase 3

So the last piece is replicating this same behavior, but across multiple machines - a remote connection. This was my final goal, and came with a few hiccups.

My process was to keep my Rails app running on the RabbitMQ machine, given that it was already working. What I needed to do was disable either the receiving or sending code, and migrate that to another computer, in a separate app. As it turns out, you can have multiple consumers of the same queue on RabbitMQ, so in hindsight, disabling one or the other wasn’t really necessary.

That being said, I took a laptop, ran through the same process again - set up a sample Rails app using the Bunny gem, plugged in the code from the existing one which was working, and… spectacular fail. Nothing. Only shell errors “Cannot connect”, “connection refused”. The Rails logs looked like this:

E, [2016-02-22T19:56:02.250071 #33526] ERROR -- #<Bunny::Session:0x7fda69e84860 varker@192.168.10.11:15672, vhost=/, addresses=[192.168.10.11:15672]>: Got an exception when receiving data: IO timeout when reading 7 bytes (Timeout::Error)
W, [2016-02-22T19:56:02.251043 #33526] WARN -- #<Bunny::Session:0x7fda69e84860 varker@192.168.10.11:15672, vhost=/, addresses=[192.168.10.11:15672]>: TCP connection failed, reconnecting in 5.0 seconds

I checked my firewall settings on the computer running RabbitMQ - all looked good. Just in case, I turned it off completely, hoping that would solve the issue. It didn’t.

So let’s step through the process a bit and I can explain how it’s different from Phase 2.

First, there are now 2 clients in the game, so instead of ‘localhost’, ‘0.0.0.0’, or ’127.0.0.1’, we have to use the proper IP address for each.

So I ran ifconfig on each machine, giving me a 192.168.10.11 and 192.168.10.5 for the RabbitMQ and remote machines respectively.

So the first change that needs to be made is in the management interface of RabbitMQ, or in the config file. We’ve been leveraging the default user, ‘guest’. Guest is not allowed to connect across machines in RabbitMQ however, unless the config file is modified.

So there are two options here - modify the config file, or create a new user. There’s some pseudo code in the docs about the config settings, so first I followed this path, modifying /etc/rabbitmq/rabbitmq.conf. Then, just to be on the safe side, I went into the management interface in the browser and created a new user, ‘varker’, with admin priveleges.

The next change is in our Ruby model. We can no longer call Bunny.new without arguments. This was basically running off 'localhost', the default port '5672', and the default user/password combo of 'guest'/'guest' - all implicit. Now we have to do Bunny.new(:host => '192.168.10.11', port: 5672, user: 'varker', password: ‘xxxxx’ )

The other change needed for our rails apps is in the server instantiation. We actually need to bind to the correct IP and port. This means on the RabbitMQ machine I need to run

rails s -b 192.168.10.11 -p1234

And on the remote machine, I use it’s IP in the same fashion:

rails s -b 192.168.10.5 -p3000 (port is your choice..)

Then, the final change I missed the first time around, and what caused me endless anguish with the IO timeouts, is to modify the RabbitMQ env config file. Right, there’s a second config - /etc/rabbitmq/rabbitmq-env.conf. Its pretty basic out of the box - just a couple of variables: NODE_NAME, and NODE_IP_ADDRESS. This last one is most likely set to ’127.0.0.1’ by default, or ‘localhost’. To use it remotely, you need to modify this to point to the machine running the instance of RabbitMQ. In my case, ‘192.168.10.11’.

Then, the absolute last change - if you were accessing your management interface on http://localhost:15672, you’ll need to change the url to point to that same machine, in my case http://192.168.10.11:15762.

That done, I go into my final test workflow:

  1. Restart my RabbitMQ server
  2. Open another tab to tail the RabbitMQ logs
  3. Start both Rails apps on each machine.
  4. Fire up the Rails apps in browsers, and watch the Rails logs and RabbitMQ logs for output.
  5. Check the management interface for activity. One interesting point here, (see image below), the bind address listed should now be the IP you set in the env conf file.

bind address updated

For my test, I used the machine running the RabbitMQ instance, (the 10.11), as the sender, while my remote machine was the receiver. So I hit refresh on 192.168.10.11:1234/rabbitts/send/message, and watched as magically, the logs in my Rails app on 10.5 showed:

Started GET "/" for 192.168.1.5 at 2016-02-23 19:48:43 -0800
Processing by HomeController#index as HTML
waiting for messages from the big mac where the bunny queue 'rabbitt events' was set up ....
Message received! ------- hello from Brier Sr. .... ---------
props: {:content_type=>"application/octet-stream", :delivery_mode=>2, :priority=>0} ===================
info: {:consumer_tag=>"bunny-1456285723000-548533168304", :delivery_tag=>#<Bunny::VersionedDeliveryTag:0x007fdc745f5930 @tag=1, @version=0>, :redelivered=>false, :exchange=>"", :routing_key=>"rabbitt events", :consumer=>#<Bunny::Consumer:70292411003800 @channel_id=1 @queue=rabbitt events> @consumer_tag=bunny-1456285723000-548533168304 @exclusive= @no_ack=true>, :channel=>#<Bunny::Channel:70292424556060 @id=1 @connection=#<Bunny::Session:0x7fdc75fbd958 varker@192.168.10.11:5672, vhost=/, addresses=[192.168.10.11:5672]>>} ===================

Success! Now that I know how to set this up, it’s really fairly straightforward. The RabbitMQ docs are a great help and really well-done. Hope this helps someone. Ping me if you have questions or find errors/bugs - happy to help if I can, and I always welcome feedback.

 

 

 

I was playing around with RabbitMQ today. For those who don't know what it is, RabbitMQ is a message broker - or what some might call a bus, message queue, or message-oriented middleware. In the most simplistic terms, it's a middleman. It sits between 2 or more applications, receiving and sending messages back and forth, which in turn most likely trigger some action within one or more of the applications.

One of the nicest things about this is you logically decouple the application instances from each other, so they don't have to know about each other's structure, objects, or even language. A Python script can send a message to RabbitMQ, and on the other end of the queue, a Ruby app can consume it. Another nice point is scalability - if you find performance of the queues is suffering, you can throw more hardware (or virtualized hardware) at the RabbitMQ instance, without needing to touch either of your applications.

So intros aside, on to my troubles. One issue I was having is attempting to start the server. I had installed RabbitMQ with Homebrew, and so there are a couple of methods to interact with the server. You can run these two commands, launchctl load ~/Library/LaunchAgents/homebrew.mxcl.rabbitmq.plist
to start it, and launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.rabbitmq.plist
to stop it.

Drawbacks should be obvious. The other command available is rabbitmq-server. Whenever I attempted to run this command, I kept getting an error message: rabbitmq-server command not found

Upon re-reading the docs, I found the cause of the problem. They literally tell you, "The RabbitMQ server scripts are installed into /usr/local/sbin. This is not automatically added to your path..."

So, to fix it, simply open up your .profile or .bash_profile, and add the recommended line. In my case, I use vi, so:

$ vi ~/.bash_profile

On lines 5 and 6 I had:

PATH=$PATH:$HOME/.rvm/bin
PATH="/usr/local/bin":$PATH

On line 7, I added:

PATH=$PATH:/usr/local/sbin

Then I wrote, quit, and did a

$source ~/.bash_profile 

in my open Terminal tabs, and I was off and running with "rabbitmq-server"