EventMachine and Passenger

In order to fully explain this post, we first need to cover some back story. Originally, Gaug.es was hosted on Heroku. Recently, we moved Gaug.es to RailsMachine (before the great AWS outage luckily), where we are already happily hosting Harmony.

At Heroku, we were running on 1.9.2 and thin. The most common RailsMachine stack is REE 1.8 and Passenger. Sticking with the common stack meant it would be a far easier and faster transition to Railsmachine, so we tweaked a few things and switched.

Heroku, Thin, and EventMachine

While at Heroku, we had been testing using PusherApp for live updating of analytics as they occurred. The pusher gem has two ways to trigger notifications, trigger (net/http) and trigger_async (em-http-request).

Since Heroku runs on thin, we used trigger_async. This meant that sending the PusherApp notifications in the request cycle was fine, as they did not block.

One of the changes when moving to RailsMachine was switching from trigger_async to trigger. Obviously, having an external HTTP request in your request path is less than ideal, but backgrounding it seemed to go against the whole idea of “live”.

Our response times for Gaug.es average around 5ms, so even with 75-100ms for each PusherApp request, we were still in a normally acceptable response time range (not ok with me, but ok for now).

Pusher Conversation

I contacted the fine folks at Pusher and asked if they had any suggestions. One suggestion Martyn mentioned was Thread.new { EM.run }.

Given my lack of experience with threads and event machine, this at first this struck me as dirty/scary and I was not sure if he was serious.

I did a bit of research and discovered he was not only serious, but people we doing it. The AMQP gem even recommends it in the Readme.

Hmmm, This Might Actually Work

After a bit of googling and scouring code on Github I found a few different solutions. I started hacking and got something that was “working” pretty quickly. Quite intrigued I decided to hit up someone smarter than I, Aman Gupta, who maintains the EventMachine and AMQP gems.

He confirmed that it would work and recommended a few tweaks. Yesterday, I pushed it to production and thus far it is working great. Below is the code needed to make the magic happen.

module GaugesEM
  def self.start
    if defined?(PhusionPassenger)
      PhusionPassenger.on_event(:starting_worker_process) do |forked|
        if forked && EM.reactor_running?
          EM.stop
        end
        Thread.new { EM.run }
        die_gracefully_on_signal
      end
    end
  end

  def self.die_gracefully_on_signal
    Signal.trap("INT")  { EM.stop }
    Signal.trap("TERM") { EM.stop }
  end
end

GaugesEM.start

Gaug.es is 100% Sinatra, so I just put this in the file in Gaug.es that works similar to environment.rb or an initializer would in Rails.

There are two key parts. First, if we are running on Passenger and using smart spawning, we need to stop the event machine if it is started. Second, we create a new thread and start the event machine loop.

Now, in the Notification class that we have in Gaug.es, I can do the following to make the Pusher request not block the main request.

EM.next_tick {
  Pusher[channel].trigger_async('hit', doc)
}

The main request carries on as usual and does not wait for the Pusher to request to finish. In the background, event machine is sending all these notifications. Once again, even on Passenger, we now have non-blocking pusher notifications.

Hmm, This Does Work

Since it took me a bit to figure it out, I thought I would post it here for everyone to benefit from and maybe to start some discussion. If you have suggestions or see glaring issues, please let me know.

I have no assumptions that I am wise or that this is perfect, but thus far it is getting the job done with no adverse affects.

Misleading Graph of Proof

Below is a graph of response times for Gaug.es thanks to New Relic. Seriously, where would we be without New Relic! Green is the time spent in external requests. I am sure you can tell at which point I pushed out the event machine integration.

That said, don’t think that all that time is instantly gone. It is still happening, just in a thread in the background without much affect, if any, on our normal response times.

Demo, Plz!

If you are curious about what the live updating looks like currently in Gaug.es, I posted a short video a few weeks back.

Also, if you too are addicted to analytics, you should definitely sign up and try it out. Lots of good stuff coming down the pipe!

10 Comments

Keith DeLong
May 04, 2011

I love the live updating. Never thought I was addicted to analytics but watching the real time demo spawned a ‘too cool’ moment! I’m on my way over to Gauge.es…
John Nunemaker
May 04, 2011

@Keith DeLong: Sweet. Hope you like it.
Dr Nic Williams
May 05, 2011

Mate, can you be a champion and throw these ideas into a little stand alone Sinatra demo app?
Charles Roper
May 05, 2011

Kind of tangential, but, given this how come you’ve moved to RailsMachine?
Christoffer S.
May 05, 2011
Wouldn’t it be simpler if you got rid of EventMachine and ran trigger in another thread? Something along the lines of…
```
Thread.new {
  Pusher[channel].trigger('hit', doc)
}
```
…but perhaps with a queue.
Jonas Schneider
May 07, 2011

@Christoffer S: This could potentially lead to a large backlog of threads given the HTTP requests start stacking up and taking too much resources…
Julien Nakache
May 27, 2011

Hi John,
this is an interesting mix of threading and eventmachine here. I’ve got one question though.

Even though the user request is processed normally without waiting for the em-http request to complete, I think Passenger won’t be able to use that spawning process until em-http returns ? I’m not 100% sure, so it would be nice to have your view here.

For that reason, I had a similar problem and I created a fully asynchronous server based on event-machine and redis BLPOP (get new element in real time) to deal with long running I/O requests. The advantage here is that you only need one rails environment loaded up in memory (unlike Resque) and you process your request in real time. The disadvantage is you need to have the server running all the time and it’s one new point of failure (need to monit the process).

So, what do you think ?

Cheers
Milan Dobrota
Jun 27, 2011

EventMachine is awesome, I have been using it for some of my projects.
Btw “without much affect” should be “without much effect”.
Michael
Jun 14, 2011

There is nothing dirty about running EventMachine in a separate thread. On non-EM based app servers you cannot run EM event loop on the main thread because it will block forever.

There is one caveat (not EM-specific, by the way): with fork(2), child process only inherits one thread. Which means that EventMachine thread will not be inherited and EM will be very confused about that.
So the solution (described in the amqp gem documentation) is to start EventMachine reactor after Unicorn, Passenger or what have you forks.

Once you do that, on 1.9.2 and JRuby and Rubinius you are all set. On 1.8.7 it will work but 1.8.7 thread scheduler is not reliable (it will give threads different time slices) and you will see “publishing delays” if you are trying to publish a lot of messages at once. That is because EventMachine thread isn’t given fair time share to run.

But other than the forking behavior which is an OS “feature”, there is nothing “dirty” or “scary”. In fact, as more and more Ruby implementations drop the GIL, understanding threading becomes a necessary skill.
CESKEETTESESK
Sep 11, 2011

скачать лего стар варс ланарте схемы скачать бесплатно Kms активация windows 7 enterprise сервер обновления nod32 v4 stalker 1 часть скачать скачать total network inventory скачать мультфильм вуншпунш инструкция 5 нок Скачать Форрест Гамп скачать бесплатно клипы зары кипелов власть огня скачать скачать тексты про любовь ключ к игре храм инков камасутра рассказы pc viewer d6 series скачать скачать русификатор для iconpackager скачать баста город в огне скачать песню небо пополам ww torrentino com jvc kd lhx502 инструкция

секреты проституток рецепт морковных котлет диета нанометодика похудение фенотропил похудеть бесплатно скачать хиты радио acronis disk director suite кряк one more try скачать убрать живот с помощью диеты диета похудеть соки суть кремлевской диеты проститутки метро новогиреево aha aha i like it скачать скачать эмулятор n64 скачать патч star wars battlefront диета южного побережья диета розовый лишай интим досуг в смоленске холестерин повышен диета crysis wars crack скачать samp сервер beyonce baby boy скачать бесплатно оптимизация производства фруктово творожная диета посоветуйте спортивное питание скачать песню из фильма леон

кейген corel x4 delphi serial number
брокколи диета
сырая диета
секс знакомства геев
диета софии ротару
как похудеть за полгода

скачать через торрент музыкальные видеоклипы Скачать Джек-потрошитель камасутра бесплатная в картинках действенная диета похудения масло перца чили похудеть работа в малоярославце самый верный способ похудеть скачать rise of the triad скачать counter strike letitbit скачать гуф снег убираем животик эффективные упражнения андреева похудение силой мысли антицеллюлитный массаж ложкой похудеть после 40 лучшие обои для стола easy gif animator 4 crack александр заборский скачать лечебное похудение санаторий вакансии бухгалтера в кирове знакомства в ялуторовске магазин интим в сыктывкаре 1c postgresql битва титанов скачать бесплатно 2010 nodvd для 18 стальных колес хлебная диета отзывы похудеть быстро без таблеток календарь похудел на декабрьской диете проститутки казахстана петропавловска

batman arkham кряк металлочерепица монтаж инструкция swedish house mafia скачать бесплатно диета ольги дроздовой шлюхи орёл психотерапия похудения сильно похудевшие звезды сократ переводчик скачать торрент электронный учебник по химии скачать касперский скачать бесплатно триал ресет самый эффективный препарат для похудения диета доктора мухиной руки вверх скачать песню утром интернет магазин товаров для похудения драйвер двд скачать бесплатно add2board 4 кряк скачать скачать песню коди симпсона