RailsTips.org

I’m not going to lie and tell you that I’m great at task management. Frankly, I suck and I blame it on technology. If only there were a task system that could light a fire under my butt. We all know there isn’t, but the system that I’ve found to work best for me is Todoist. Todoist is really light and fast. Also, it has a simple API so it’s very extensible and already has a lot of tools surrounding it (quicksilver, launch, dashboard widget, mobile, gmail integration, etc).

The Problems

The problem that I have with task management is two fold. First, I need reminders. My programming brain is sharply trained to scoot information in and out, only keeping what is immediately necessary. It’s great to have a to do list but if it doesn’t remind me, it barely works. The good news is that todoist has a premium service that is only $3/month (yes only $3) that allows for reminders in any form you like (twitter, sms, email, jabber, etc). First problem solved.

The second problem that I have is that sometimes a reminder is not enough. I personally need to see my upcoming projects, tasks and calendar items in front of me anytime I can. It hit me the other day that I look at my desktop a lot. This got me wondering if I could use the todoist API to put my upcoming, date sensitive tasks right there, on the desktop, in front of me whenever I’m on my laptop.

The Solution

I did some googleing and found Geektool, which I had installed/uninstalled before but this time seemed to have a purpose. The end result is in the screen cap below.

To do this, I created a quick and dirty ruby script to hit the todoist api and put out a simple list of all time sensitive tasks. 1) I put the following script at ~/bin/todoist.rb.

require 'yaml'
require 'rubygems'
require 'rio'
require 'json'
require 'active_support'

def todoist_query(*args)
  token   = args.pop
  queries = %Q{["#{args.flatten.join('","')}"]}
  url     = "http://todoist.com/API/query?queries=#{queries}&token=#{token}" 
  JSON.parse(rio(URI.encode(url)).read)
end

lines, now   = [], Time.now
config = YAML::load(rio(File.join(ENV['HOME'], '.todoist')).read)
format = '%Y-%m-%dT%H:%M'
days   = (now..now.advance(:days => config['days'] || 3)).step(1.day)

todoist_query("overdue", days.map { |d| d.strftime(format) }, config['token']).each do |type|
  next if type['data'].size == 0
  lines << (type['type'] == 'date' ? Time.parse(type['query']).strftime('%b %d').gsub(/\s0/, ' ').upcase : type['type'])
  type['data'].each { |i| lines << " - #{i['content']}" }
  lines << "\n" 
end

puts lines

Like I said, it’s quick and dirty. 2) Now setup geektool to run the script at a given interval (I chose 2 minutes).

3) I created a file in my home directory to store my todoist token and the number of days I want to show. The main reason I separate the token from the ruby script is in case I feel like using it somewhere else on my system with another script. If my token ever changes, I can update it in one place and all the scripts will still use correct one. Anyway…I put the file at ~/.todoist and it looked something like this:

token: foobarbaztoken
days: 4

Now every two minutes the script runs and updates based on how I am updating my tasks at todoist.com. My tasks are continually in front of me, which helps keep them in my mind so that I actually get stuff done.

Other Uses

Some other uses I could see valuable for this would be calendaring. Google Calendar has an API so you could hit that and show your upcoming events right by your tasks. Maybe there is some site that posts a lot of updates and it’s hard to keep up with by feeds? Just use feedtools and a tinye ruby script to show the latest posts for your perusal when you get a few seconds. Just some thoughts.

Anyone else do something along these lines? Have some code to share or thoughts on how to do it better? Let me know.

I have a dream. A dream wherein I can update a buddy’s iChat status remotely. It’s not a change the world dream, but it would be fricken sweet, right?. I mean imagine your fiend coding up a storm on his computer and all of a sudden noticing that his iChat status says ‘I have a man-crush on DHH.’ I know you are feelin’ me.

Anyway, I guess I should drop some history. The dream started a year back with Steve, Apache and PHP, but quickly faded as we lost interest. A few days back, the dream was rekindled, this time with Chas, Twitter and Ruby and it has now ended in success.

The Goal

The goal is very unselfish. I don’t want this power all to myself. I want to share it with all my friends, that is with the exception of who we shall now refer to as “the victim.” We wanted to be able to allow ourselves and a few friends the ability to update one of our other friend’s iChat status (“the victim”) from anywhere. What is a tool we all use, all the live long day? Twitter. The solution is actually pretty simple. It’s only 3 steps. Create a private twitter account that you can all direct message. Install a ruby script on “the victim’s” computer. Setup scheduled task to run ruby script every few minutes. I’ll now detail these steps with code samples but remember, with great power comes great responsibility (and laughter).

Step 1 – Create Private Twitter Account

Create a twitter account and make it private. All the friends that would like to have power over “the victim” should then request to follow the new account and you should approve them. This means that any friend who follows that account, and that you approve, can now direct message that account. The general public cannot. Now the group of controlling friends needs only to twitter ‘d newtwitteraccount some new ichat status’ and the next status for “the victim” is ready to be set.

Step 2 – Install Ruby Script

Ok, so I could care less about windows. Knowing that, I’m only dealing with Mac’s. What does every mac come with? Ruby. What language do I make love to every night? Ruby. Now beings that I wrote the twitter gem, you might assume that I used that in the script. You’d be wrong. You know what they say about assuming, right? It makes an ass out of ‘u’ and ‘ming’ or something like that.

More than likely you will only have a few bathroom break minutes to setup everything on your friends computer. This means we need the least amount of dependencies to slow us down. Below is the quick and dirty script I wrote to get the latest direct message from a twitter account using only core ruby libraries. Basically it grabs the latest direct message and if it is new, it executes some applescript to update the iChat status.

#!/usr/bin/env ruby
%w[open-uri rexml/document fileutils].each { |x| require x }

Debug = true
Username = 'newaccount'
Password = 'password'
dms = []
url = "http://twitter.com/direct_messages.xml" 
lines = ['tell application "System Events"', 'if exists process "iChat" then', 'tell application "iChat"', 'set the status message to "#{latest}"', 'end tell', 'end if', 'end tell']
puts 'Doing some magic' if Debug
data   = open(url, :http_basic_authentication => [Username, Password]).read
doc    = REXML::Document.new(data)
doc.elements.each('direct-messages/direct_message/text') { |el| dms << el.text }
latest = dms.first
if latest
  puts 'we have a latest message' if Debug
  open('latest.txt', 'w+') do |f|
    last = f.gets
    if latest != last
      puts 'we are changing ichat status now' if Debug
      f.puts latest
      cmd = "-e '#{lines.collect { |l| l.gsub('#{latest}', latest) }.join("' -e '")}'" 
      system "osascript #{cmd}" 
    end
  end
end

This script should be located at ~/Library/TwitteriChatHack/twitter-dm.rb. Be sure to update Username and Password to the private account you just created in step 1.

Step 3 – Schedule Task to Run Ruby Script

We have our script, but it does nothing without being told to. That said, we have to schedule it to run on an interval. Mac OSX has a great scheduling tool called launchd. It appears to be a black art, but is actually really easy to use. Justin just wrote up a great article on how to automate rick-rolling your friends with launchd. Below is the sample launchd plist item that instead of rick-rolling, will run our ruby script from step 2.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.addictedtonew.twitter.ichat.hack</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/env</string>
        <string>ruby</string>
        <string>/Users/MACOSXUSERNAME/Library/TwitteriChatHack/twitter-dm.rb</string>
    </array>
    <key>RunAtLoad</key> <false/>
    <key>StartInterval</key> <integer>60</integer>
</dict>
</plist>

This script should be located at ~/Library/LaunchAgents/com.addictedtonew.twitter.ichat.hack.plist. Be sure to change MACOSXUSERNAME with your friends max osx username.

Now the only thing left to do is to actually load the task into launchd. If you don’t do this, your script won’t work until “the victim” logs out and back in. Just run the following command:

launchctl load ~/Library/LaunchAgents/com.addictedtonew.twitter.ichat.hack.plist

Summary

That’s it. Once you create the private twitter account, add the ruby script onto “the vicitim’s” computer and setup the launchd task, you can can ‘d twitteraccount i love britney spears’ and within a minute your friend’s iChat status will be updated appropriately. Below is an example on my computer of the direct message and iChat status update.

Notes

• The interval at which the status can be udpated at is set with the StartInterval number. Mine defaults to 60 but you might want to set it to 300 (every 5 minutes) so you aren’t continually checking for updates.
• The ruby script checks to see if iChat is running and only attempts to udpate status if it is so it won’t pop open iChat on your friends computer every minute.
• The ruby script also saves the last direct message that was used so if you direct message and your buddy changes his status back to something else, it will only change it again if you put through a new direct message.

So that is all. What kind of crazy things like this have you all done?

Jeremy, from my local ruby group, had a sweet idea. He was piping crontab commands into mail and thought “why not use twitter?” He hacked around a bit and sent me some code. I tweaked a couple things and now, as of version 0.2.6, the twitter gem can take stdin when posting from the command line. Below are a few examples from the changelog.

$ twitter post 'test without stdin' 
# => twitters: "test without stdin" 

$ echo 'test with stdin' | twitter post 'and an argv[1]' 
# => twitters: "test with stdin and an argv[1]" 

$ echo 'test with stdin without any argv[1]' | twitter post
# => twitters: "test with stdin without any argv[1]"

If you are curious about the implementation, see the bottom of the bin/twitter file.

A Recent Cool Use of the Twitter Gem

Also, while on the topic of twitter, some pals of mine are repurposing the twitter gem to do reviews (and they were even so kind as to update the gem and send a pull request). If you are a twitterer, check out microrevie.ws.

There are a few view related things of late I’ve been using to make my code a bit more readable. Trust me, they are nothing new, but for whatever reason I hadn’t jumped on the train.

div_for

This doesn’t do anything special but saves some typing and I tend to think it makes things a bit more aesthetically pleasing in TextMate. Using div_for, the following:

<div id="<%= dom_id attachment %>" class="<%= dom_class attachment %>">
    ... removed for clarity ...
</div>

becomes..

<%- div_for attachment do -%>
    ... removed for clarity ...
<%- end %>

You can also use it’s companion, content_tag_for in the case where you are creating table rows or something:

<%- content_tag_for :tr attachment do -%>
    <td>... removed for clarity ...</td>
<%- end %>

local_assigns

Want to pass in optional parameters to a partial rendering? local_assigns is the key to your problems.

<%- if local_assigns.has_key?(:show_project) -%>
    <h3><%= time_entry.project.name %></h3>
<%- end -%>

If you did <%= render :partial => 'foo', :locals => {:time_entry => time_entry, :show_project => true} %>, the project heading would be shown but if you just did <%= render :partial => 'foo', :locals => {:time_entry => time_entry} %> without the :show_project hash key, the heading would not be printed. That might be a bad example, but you get the idea.

content_tag

Obie mentioned a good use for this in the Rails Way, which is in combo with an if statement for some one line goodness. Take the following for example:

<%- if current_user.authorized_to_edit?(user) -%>
    <span class="actions">
        <%= link_to 'Edit', edit_user_path(user) %>
    </span>
<%- end -%>

That works ok, but using content_tag you can easily put it on one line like so:

<%= content_tag 'span', link_to('Edit', edit_user_path(user), :class => 'actions') if current_user.authorized_to_edit?(user) %>    

The two examples put out the same html with the same permissions but the latter is much more succinct.

debug

In PHP, it was var_dump. In ColdFusion is was <cfdump>. Somehow, I overlooked debug in rails for almost a year. Simply call <%= debug @projects %> in your view and you will get a to_yaml representation of the variable wrapped in pre tags.

I’m sure there are tons of others that I’m not super aware of, but these are a few that I’ve found helpful of late. Anybody have other favorites? Maybe concat mixed with a block? Maybe some capture action?

Doesn’t it stink that google analytics doesn’t have an API? Yeah, I know. It seems nearly every post I mention conductor nowadays so I’ll bring it up again. We have about 15 sites live in conductor and darn it if I’m not interested in how they are doing traffic-wise.

The Original Sin

What I thought would be handy, is one simple dashboard where I could see all the pageviews, visits, etc. for each of the live sites in conductor, both separately and collectively. Naturally, I went searching for an analytics API and found nothing. I did a bit more digging and found that there were a few rogue hackers out there who had some form of an API working. It intrigued me so I started to check out the analytics source code. You know all those fancy charts and such? Yep, they’re powered by XML. Woohoo!

Log Me In Scotty

Now, the only thing between my glorious dashboard and me was a dainty browser session. For this, I reverse engineered Google’s login, discovered their cookie monster and created an authentication base class. This made it drop dead easy to make authenticated requests to google for data (I’m also using it in a GReader wrapper I’m working on ).

If you want to make an authenticated request to google it’s as simple as gem install googlebase and the following:

require 'google/base'
Google::Base.establish_connection('username', 'password')
Google::Base.get('http://www.google.com/reader/api/0/token')

Statwhore

Ding, ding, ding. Dainty browser session down, it was now time to start wrapping up those xml responses into ruby objects that are more fun to play with. It’s pretty much just a lame start, but it is aptly named Statwhore and it’s on a github near you. For now it just has really basic stuff, like pageviews and visits for a given time period. Take the irb session below for example:

>> require 'statwhore'
=> true
>> Google::Base.establish_connection('username@gmail.com', 'tehs3cr3t')
=> #<Google::Base:0x199ae84 @password="tehs3cr3t", @email="username@gmail.com", @sid="areallylonghashofkeysthatyoucantsee">
>> Statwhore::Google::Analytics::Account.find_all
=> [#<Statwhore::Google::Analytics::Account:0x1943300 @account_id=85301, @name="addictedtonew.com">, #<Statwhore::Google::Analytics::Account:0x1943058 @account_id=344381, @name="webgroup.nd.edu">]
>> Statwhore::Google::Analytics::Profile.find_all(85301)
=> [#<Statwhore::Google::Analytics::Profile:0x12a0304 @account_id=85301, @name="addictedtonew.com", @profile_id=47912>, #<Statwhore::Google::Analytics::Profile:0x129edd8 @account_id=85301, @name="googlebase.rubyforge.org", @profile_id=5867643>, #<Statwhore::Google::Analytics::Profile:0x129e70c @account_id=85301, @name="googlereader.rubyforge.org", @profile_id=5868472>,  #<Statwhore::Google::Analytics::Profile:0x129cde4 @account_id=85301, @name="lorem.rubyforge.org", @profile_id=3027809>, #<Statwhore::Google::Analytics::Profile:0x129a080 @account_id=85301, @name="mirrored.rubyforge.org", @profile_id=5211778>, #<Statwhore::Google::Analytics::Profile:0x12999c8 @account_id=85301, @name="railstips.org", @profile_id=1897579>, #<Statwhore::Google::Analytics::Profile:0x12995b8 @account_id=85301, @name="scrobbler.rubyforge.org", @profile_id=3293580>, #<Statwhore::Google::Analytics::Profile:0x12991bc @account_id=85301, @name="snitch.rubyforge.org", @profile_id=2800544>, #<Statwhore::Google::Analytics::Profile:0x12989c4 @account_id=85301, @name="twitter.rubyforge.org", @profile_id=2865122>]
>> profile = Statwhore::Google::Analytics::Profile.find(85301, 1897579)
=> #<Statwhore::Google::Analytics::Profile:0x1c3af18 @account_id=85301, @name="railstips.org", @profile_id=1897579>
>> profile.pageviews 
=> 1743
>> profile.pageviews(:from => Time.mktime(2008, 3, 1), :to => Time.mktime(2008, 3, 10))
=> 2416
>> profile.visits(:from => Time.mktime(2008, 3, 1), :to => Time.mktime(2008, 3, 10))
=> 1862

Cool stuff, eh? So I didn’t write this post to say go use this in production for a paying client. I just thought it was fun playing around and thought I would put it out there with some explanation on the off chance other people might feel like helping the project along.

The Future

Eventually, I see Statwhore wrapping a crap load of the analytics api’s that I’m sure are out there. Feedburner has an awareness api and a feed management api. Rest assured I’ll be wrapping those up eventually, as I have a dream of only Feedburner urls being spewed from Conductor’s news section. I guess my question is, what are some other good analytics services that also have api’s?

Testing Live Web Services

Also, if you are working on a web service gem and are wondering what is the best way to test it, I don’t have the answer. :) I have tried a few different ways. My recommendation is to mock or stub in some way the response and then let your class go to town. Below are a few different ways I have done it on previous gems.

1. Live. Don’t do this as your only tests. It sucks. Good idea to have around to see if API changes though. I now horribly regret the tests I put in for the twitter gem. They are horribly fragile.
2. Mock the class that does the outside connections and instead return local fixture files of some sort.
3. Do the same thing as the previous one but use the FakeWeb gem
4. And most recently, use rSpec to stub the fetching of the response with a local file.

Anyway…it’s late and I’m heading to bed. Just felt like getting this post out before the project died a horrible open source death.

Ok. Back on track for keeping your tag cloud running. We talked about backups and short/long term problem resolution, so now it’s on to resource reporting. I really see resource reporting more at a macro level. What I’m talking about are things like memory, CPU and filesystem usage, apache processes and volumes and mysql queries and throughput. These stats give you information on how your application is running and where possible rough spots are (spots that need tweaking).

Munin

There are a lot of tools out there to get the job done but the one that I picked is munin so that’s what I’ll talk about today. From the munin website:

Munin the monitoring tool surveys all your computers and remembers what it saw. It presents all the information in graphs through a web interface. Its emphasis is on plug and play capabilities. After completing a installation a high number of monitoring plugins will be playing with no more effort.

Server/Node

That pretty much explains it. What I did is I have one small VPS that I use for version control and as a munin server. I guess before I go to far into that I should explain some quick theory. Munin has a node/server setup. That means you can have all your servers act as nodes which simply make information available to a configured server. The server is the machine that goes out every minute (or how ever often you say), grabs the information from the nodes and collects it into one place. This is nice because I can get a quick daily, weekly, monthly or yearly overview of all the nodes at once on one page. I only keep track of about seven so the page is not ridiculously full of information.

Installation

My gauge for easy installation is, if I can do it, anyone can do it. I’m far from a linux guru, trust me. On Ubuntu, it’s nearly as easy as sudo apt-get install munin and then a few tweaks. On CentOS, once you get the RPMForge and Dag hooked up, installation is a simple sudo yum install munin and, again, a couple of tweaks and you are good to go. I’ll list some links at the end that got me going.

The Payday

So I’ll be the first to admit that Munin could use some UI love but the most important thing is seeing the data in a graph and it accomplishes that well enough. Below are a few samples of real graphs (all daily) pulled from Conductor, our home grown, rails, multi-site website management tool at the University of Notre Dame.

Apache Access: This is helpful just to see the wax and wane of traffic on your site. I find it interesting that most of the sites in Conductor go nearly dormant at night. Leads me to believe we have mostly U.S. traffic (which makes sense).

File System: Now I should have stuff set in to automatically warn me if I get up in file system usage but I’m so low right now, I don’t worry about it too much.

Memory Usage: This is really important for Rails apps. They are known to use a fair share of memory and you need to keep an eye on this for leaks. If you see steady growth, you probably have some issues. One thing to note from the spike above is that it is caused each night by a backup task that I have to shuffle my database, assets and themes off-site. I need to use rsync or something but it is working and who has time, right? Plus if things get out of control, I have monit to get things back in order.

MySQL Queries: Now obviously, more traffic equals more queries so this graph has tended to show the same trends as apache access. I think it’s pretty interesting to see the number of queries per second so I tend to check it out.

Real Life Ways Munin Helped Me

So I show this to give a valid example of how Munin helped me. I avoided premature optimization when building Conductor, and just focused on how the app worked. Once I got Munin tracking all this stuff, I noticed that it seemed like a lot of queries, for the amount of traffic we had.

I did some tailing of log files, tweaking of queries and some htaccess trickery, and you can see the dramatic drop. That drop was not a drop in traffic. In fact, traffic and the number of sites in conductor has only risen, but the queries are still down quite a bit, simply do to a bit of analyzation (which was prompted by the queries per second graph). Also, ignore the blank spots. :) We had some issues where server reboots were required and I forgot to set Munin to start on boot and also forgot to check to see if it was running. Like I said, I’m not a server admin. I’m like you (unless you are a server admin).

Another way that Munin helped out, was by showing me the number of slow queries. This particular slow query, again, happens when the memory jumps as ruby tries to FTP a crap load of backups off-site. How this was helpful, was it prompted me to do more research on indexes and analyzing queries. I was noticing a lot of slow queries as the tally of pages grew in conductor. I did some grepping of the slow query log, based on what Munin was telling me, then some EXPLAIN’in in MySQL, and quickly added two or three indexes which fixed all the issues (more on that in another post).

These are just two examples, and I’m sure I could come up with more but I think I got the point across. Munin is pretty easy to install, has lots of documentation/articles on the net and has come in really handy for me.

Related Links

• How to install Munin on CentOS
• How to get RPMForge and DAG on CentOS
• Password protecting your Munin install
• Nagios
• FiveRuns Server and Rails Application Monitoring

The ‘Keep Your Tag Cloud Running’ Series

• Introduction
• Part 1: Backups
• Part 2: Short Term Problem Analysis and Resolution
• Part 3: Automating Problem Resolution
• Part 4: Resource Reporting
• Part 5: Log rotation and analysis
• Part 6: Slow Queries and Indexing
• Part 7: General Closing Thoughts

So I nearly threw my laptop across the room in a violent rage trying to figure this out. Thankfully I didn’t and rather scoured the googles for a solution. Upon finding it, beautiful music rang from the heavens and a solitary ray of sunshine beamed down on me.

I don’t know if it’s something people rarely try to do or if it’s just hard to find but I thought I would post it here in hopes that the next poor sap who attempts to do so has a bit easier going.

The Tricki-Tikki-Tavi

The trick (if it can be qualified as that) is to use !xml | and some proper spacing like so:

law_news_category:
  theme_id: 1
  filename: news.category.liquid
  data: !xml | 
        <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
        <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
        <head>
            <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
            <title>{{ site.name }} // University of Notre Dame</title>
        </head>
        <body>
        <div id="news">
            <h1><a href="/news/">News</a> &raquo; <span class="category_name">{{ category.name }}</span></h1>
            {% for item in news %}
                <div class="news">
                    <h2>{{ item | link_to_news }}</h2>
                    <h3>{{ item.published_at | strftime: '%B %d, %Y'}}</h3>
                    <div class="excerpt">
                        {{ item.excerpt }} 
                    </div>
                </div>
            {% endfor %}
            {{ news | will_paginate: path }}
        </div>
        </body>
        </html>
  creator_id: 3
  updater_id: 3

If you line up the html with the !xml, you will be good as gold. Phew. Now back to my regularly scheduled programming.

So each time I go to code in a rails app I open up three tabs: one for script/server, one for autotest and one for general commands like script/generate and such. Let’s say I want to open up tinye, a project I’m working on at work. I have to do the following:

cd sites/nd/tinye.git
script/server
# new tab
cd sites/nd/tinye.git
a #that is my shortcut for autotest -rails
# new tab
cd sites/nd/tinye.git

I got a bit tired of this, so based on some scripts I saw around the interwebs, I put together the one below:

#!/bin/sh

if [[ $# == 0 ]]; then
  PROJECT_DIR=$PWD
elif [[ $# == 1 && -d "$1" ]]; then
  PROJECT_DIR="$@" 
else
  print "usage: iterm.sh [rails project directory]" 
  return 1
fi

# how to make new terminal (make new terminal)

osascript <<-eof
  tell application "iTerm" 

    tell the last terminal

      activate current session
      tell the last session
          set name to "server" 
          write text "cd \"$PROJECT_DIR\"" 
          write text "ss" 
      end tell

      launch session "Default Session" 
      tell the last session
          set name to "autotest" 
          write text "cd \"$PROJECT_DIR\"" 
          write text "a" 
      end tell

      launch session "Default Session" 
      tell the last session
          set name to "app" 
          write text "cd \"$PROJECT_DIR\"" 
                    write text "open http://localhost:3000/" 
      end tell

    end tell
  end tell
eof

I put this in _{/bin, named iterm.sh, and made sure it was executable (chmod u+x ~/bin/iterm.sh). Then I threw the following in as aliases (}/.bash_profile for me):

alias r='iterm.sh $1'
alias ss='script/server'
alias sc='script/console'
alias a='autotest -rails'

This allows me to open the same project I talked about earlier by simply typing:

r sites/nd/tinye.git

Instantly, I get three tabs in iTerm (server, autotest and blank) and an open browser window on localhost:3000. Thought I would share it here to spark other ideas like this. Also, I’m curious if others have similar things (or cooler things). Below is quick and dirty screencast that shows it in action.

I held out as long as I could, but on Wednesday, I grabbed a ham sandwich, nestled into my office chair and hit play in Quicktime for a lunch filled with my three favorite G’s—Geoffrey Grosenbach and Git. An hour later, I was kind of blown away and feeling stupid for not checking it out sooner. I cashed in my GitHub invite and started gitifyin’ some of my svn projects.

Why?

First off, it’s clean. You know all those .svn folders inside every directory of your subversion project? Well, you can have one .gitignore file in the root of your project and you are good to go.

Second, you can use it offline. Ever been away from a net connection? Yeah, me neither. But if it did every happen, wouldn’t offline mode be handy? Git is made for distributed development. This means you can do it all locally but it doesn’t hold you to just that. If you want (and who wouldn’t), you can setup remote git repositories which allows you to push and pull your project and share easily amongst a team or open source community.

Third, straight from the git website, Git has “strong support for non-linear development.” You know that branch feature of svn that you never use because you can’t remember the command? Well, git makes it gosh darn easy. Did I mention I taught my cat Bob to do it? Just be sure to give your animals a treat after they make a branch for you. You can create a new branch…do some crazy experimental thing…and only merge your crazy code into your master branch once you have it all working and fully tested (like a good little programmer).

Fourth, sweet svn integration. Yep, anytime you make a new tool, you have to give your new cult following some way to import all the crap they have in their current tool. I liked git for the first three reasons but I love git for this fourth. cd into your local svn project, use drnic’s gitify and you have a git repo that can also commit to your svn repo (for all of those still kickin’ it old school). Branch it up, make changes and when you are done git committing, just git svn dcommit and it transfers all your git commits to the remote svn repository. But what if someone updates the svn repo while you are gittin’ your git on? Well, Linus thought of that too. You can run git svn rebase and your git repo will update with the remote changes in your svn repo. This dude even has some sake tasks to make it easier to fool your svn friends into thinking that you aren’t using git.

How can I get started?

I would highly recommend breaking the bank for the $9 git screen cast by Geoff. He covers all the stuff you need to git going, including installation. Also, there are a few places where you can host your git projects for free (repo.or.cz, gitorious, and github, which I recommend). Pretty cool stuff. All I’m trying to get at in this article is if you are thinking, “Learn something new? But I already know subversion and it works for me…” stop thinking that, buy the peepcode and enjoy version control again.

How am I using it?

I’m now managing most of my gems on GitHub and loving it. After Jesse Newland whipped together a github-campfire post-commit, I copied him and took the credit, making a github-twitter post-commit soon after. On the topic, that was my first use of Sinatra and I definitely like it for little things like this.

Some Git Links

• Using git within a project
• Git is the next Unix
• Deploying with capistrano
• All that git talk in the rails world, what gives?
• Git cheat sheet
• Why distributed version control matters to you, today
• Getting git, part 1
• Using git without feeling stupid
• Grit, a ruby git lib
• Using git for rails development
• git-rails
• git post-commit for profit

In the interest of kicking my butt into posting more, I’m going to put up a sample deploy file for a merb app we launched about a month ago. Below is an example capistrano 1.4 file that I use for a merb app that runs on active record and sqlite3. It’s a small stat program that grabs analytics in the flash carousel on nd.edu.

set :application, "myapp" 
set :user, "deploy" 
set :domain, "yourdomain.com" 
set :repository, "svn+ssh://#{user}@#{domain}/var/www/apps/repos/#{application}/trunk" 
set :deploy_to, "/var/www/apps/#{application}" 

role :app, domain
role :web, domain
role :db, domain, :primary => true

desc "Link in the production extras" 
task :after_symlink do
    # symlink log path
  run "ln -nfs #{shared_path}/log #{release_path}/log" 
    # symlink path to database, this is only needed if you are using sqlite (which is ok for little things and is really easy)
  run "ln -nfs #{shared_path}/db/#{application}_production.sqlite3 #{release_path}/db/#{application}_production.sqlite3" 
end

desc "Merb it up" 
task :restart do
  run "cd #{current_path};./script/stop_merb" 
  # run "cd #{current_path};env EVENT=1 merb -e production -c 1" # for evented mongrel
  run "cd #{current_path}; merb -e production -c 1" # plain old mongrel
end

Once you have everything configured, deployment is as easy as cap setup and cap deploy (you’ll probably need to migrate your db as well).

We (the notre dame web group) are now using Merb for two production applications and are finding it a nice change of pace from Rails. If you want to learn more, peter cooper has a crap load of links for merb at ruby inside.

You didn’t think I’d leave you hanging forever, right? I knew everyone was waiting with bated breath so here comes part 3. You have backups. You know how to fix those pesky simple problems that plague us all. Now it’s on to bigger and better things. Next stop…the monit ho train. The whole point of this post is to give you some pointers on how to automate recovering from bad things. It’s nice to know how to fix things manually but that won’t get you anywhere. The best pets are those that mostly take care of themselves. Below are some tips for turning your servers into good pets.

Installation

If you are on a slicehost server, you can use apt-get to install monit (sudo apt-get install monit) and if you are on a Railsmachine slice, you can hookup the dag repository and then use yum to install it (sudo yum install monit).

Either route you take, you will then have to edit the config file (look for /etc/monit/monitrc or /etc/monit.conf). The settings in the config file are pretty well documented. I pretty much just use monit to monitor my mongrel instances. The annoying thing is that you have to completely duplicate the entire config block for each instance of mongrel you want to monitor. I put a simple example that I use on pastie for those interested.

Once you have everything configured, you can start it up using the monit command (monit -c /etc/monit.conf or whatever the path is to your monit conf file).

Now comes the fun part. I’m assuming you have a few instances of mongrel running and you’ve installed and configured monit. If you have those basics covered, you can play with monit to confirm that it is actually working.

# get a list of your mongrels and the command that started them
ps aux | grep mongrel
# kill one of them by pid
sudo kill 3453 # or whatever the pid number is

Now just wait and you should get an email and see the process you just killed fire back up on its own (ps aux | grep mongrel). Pretty cool, eh?

Honestly, there isn’t a whole lot more to monit. It has an optional web interface. I threw a few screenshots in below that you can check out.

Monit Start Page

Monit Individual Process Page

Other Options

The new kid on the monitoring block is God. It was created by Tom Preston-Werner and is a pure ruby solution. The sweetness of this is that you can use ruby in your config file which makes for less repitition (think looping through an array of mongrel ports). The only downside is I’m sure it uses more memory (though it’s probably so little that who cares?). Anyway, I can’t vouch for it as I haven’t used it but I talked to Tom at RubyConf and he said he uses it in production on several servers with great success. I’ll be trying it soon.

In Conclusion

Whether you go with old faithful, Monit, or the new kid, God, you need to use something to monitor your tag cloud. If you are like me, the only way you want to be up at 3AM is if you are hacking on something fun, not bringing mongrel instances back online.

Related Links

• Monit Makes Mongrel Play Nice
• Daemonizing Ruby Scripts For Monit
• Mongrel Cluster and Monit
• Good Starter Monit Config File

The ‘Keep Your Tag Cloud Running’ Series

• Introduction
• Part 1: Backups
• Part 2: Short Term Problem Analysis and Resolution
• Part 3: Automating Problem Resolution
• Part 4: Resource Reporting
• Part 5: Log rotation and analysis
• Part 6: Slow Queries and Indexing
• Part 7: General Closing Thoughts

Ack is a tool for programmers similar to grep but made for heterogeneous source code trees. What I was impressed with upon downloading and installing it was the highlighting it does all in the terminal.

I’ve used it a few times searching through rails trunk as TextMate seems to dog when you get a bunch of files in one project. It also prominently displays the file path which you can then copy and open with the mate command.

A few reasons you might want to use ack (from the ack site, there are more there as well):

• it’s really fast
• it runs on windows
• the standalone version has no dependencies so you can throw it in ~/bin
• searches recursively through directories by default, while ignoring .svn, CVS and other VCS directories
• let’s you specify file types $ ack --perl pattern

Installation is really easy on a mac. Just grab the standalone version and throw it in ~/bin/ack. Note: make sure that it is executable (chmod u+x) and that ~/bin is in your path. Enjoy.

Pro Active Record “helps you take advantage of the full power of your database engine from within your Ruby programs and Rails applications.” It is authored by Kevin Marshall, Chad Pytel, and Jon Yurek and published by Apress. For this review, I’ll start with the bad and then finish on a positive note.

The Bad

I feel the title of this book is very misleading. It should be “Learning Active Record” or “More Documentation for Active Record” or “On Your Way to Being a Pro in Active Record.” Why is a “Pro” level book teaching you how to set up your database in chapter 3. Heck, why is it talking about setting up your database at all? I would think that a book on the pro level would literally be chocked full of advanced code samples and such, not simply above average documentation for all the the methods in the Active Record module. Teach me how to model complex things. Start right off with an example of how to do something that you don’t learn in the first few months of rails.

There were a few places in the book where I felt there were contradictions with idiomatic ruby/rails and even with other parts in the book. For example, page 43 recommends not to use upper case database columns. I agree. Rails agrees. It doesn’t cause any harm but Ruby is case sensitive so it’s better to stick with lower. Also, upper case typically denotes constants and classes. Method names in ruby (and thus column names) should always be lower case, yet several places in the book use examples with column names in mixed case (ie: Account_Username instead of account_username).

Also along these lines, typically, method names are words separated by underscores (ie: my_method_name instead of mymethodname). More than a few examples in the book were multiple word methods or column names but didn’t separate with underscores.

I realize that I am being really picky and this seems kind of harsh but I see this book as a beginner to intermediate level and would hate for someone in that range to see something generally frowned upon in the community and unknowingly accept it as an “ok” practice.

The Good

Ok. Enough ripping, now on to the good in this book. First off, it reminded me of how Active Record is capable of handling locking by use of the lock version column. I had read this a while ago but forgot. Not too long ago I spent 30 minutes at the whiteboard coming up with a solution (purely for academic purposes), not thinking to see if Active Record handled this in any way.

Second, it had a few examples of how and when to use the straight up connection methods for executing raw sql (execute, insert, update, select_on, select_values, select_all, etc.; more on this on Josh Susser’s blog). Just yesterday, I used select_all to do some sql for the news archives area of Conductor, the CMS I am building at work for our University.

Third, it mentioned the convertible to csv gem/plugin which adds a to_csv method to your active record models in a similar fashion to the existing to_xml, to_json, etc. that you can do in Rails.

Lastly, it pointed out the uuidtools gem. I had forgotten about it.

Conclusion

I would recommend this book to someone who has some grasp on Active Record but wants to learn more about all the options it has. Don’t expect any complex examples but the book does work great as a documentation source for Active Record. It has a few more examples and explanation than the AR docs themselves.

Other Reviews

Don’t take just my word for it, check out these other bloggers who have written up reviews and make an informed decision.

• Josh Susser
• James Stewart
• Rick DeNatale
• John G. Norman
• Justin Pease
• Ilya Grigorik
• Amazon Customers

Overall, I had a great time at my first RubyConf. I found myself complaining about a few things so I figured I should at least attempt to offer some constructive criticism, rather than just whining. :) Below are some ideas for RubyConf 2008.

• Enlarge the individual’s name on the name tags. This year the logo and design was huge and the person’s name almost disappeared. Everyone knows we are at RubyConf, but we don’t know everyone’s names. Make names huge and easy to read, you know, so stalking is a bit easier. The 40px square avatars that people use on twitter aren’t enough to recognize them at a conference.
• Good wifi. Is this not possible? Can we not pair with a university or something to get sweet conference room setups and good wifi. Hotels and conference centers may be able to handle the interweb load of some suit on a blackberry but not a geek installing a gem that a presenter is talking about or checking out some code to hack on.
• Start at no earlier than 10. The best part about any conference is meeting all the people. When the talks and keynotes go until 8 or 9 at night that only leaves the nightlife as a time to meet-up. This leads to some late nights and makes it pretty hard to function at 8 or 9AM (not that I function well at that hour anyway).
• Less talks per day. I would say four or five talks is plenty for a day. The sessions start talk. Use them as that, not as a full day of education. Use the sessions to get everyone primed to fight over interpreters at the bar later. I found myself a bit “talked” out by afternoon of day 2. Or maybe weight the number of talks by day? Day 1 has 7 or 8, day 2 has 4 or 5 and day 3 has 2 or 3.
• Don’t get any bigger. RubyConf was much easier to meet people than RailsConf. I think this is largely due to the fact that RailsConf is 3x as large. Stay small RubyConf, stay small.

Those are my suggestions. Anyone else agree or disagree with any of my points? Have some other constructive suggestions?

Again, I would like to repeat that this is not necessarily defacto or best practice, I’m just putting out what is working for me. If you think I’m stupid feel free to say so, but post an example of a better way or your comment will be deleted. Onward.

Your app is up and running. You have regularly scheduled backups. Oh crap. 503 Proxy Error. Downstream upstream. Clients and users are getting pissed. What do you do? No, calling ghostbusters is not an option. Most likely you are having memory problems or something.

Before I was using Monit (which we’ll discuss in part 3), these are steps I would take to figure out why things went down. First you have to get in your server so fire up terminal or iTerm and ssh in (be sure to check out I Can Has Command Line? for ssh shortcuts and other command line tips).

ps aux

Once you have ssh’d in, check the processes (ps aux) and see what is running (or not running) and what is pissing off your server.

From the pic above you can see the columns of information that are given to you. Most likely you will get a long list of processes but don’t worry that is normal. There are a few important things to look at. First, scan up and down the list in the CPU and MEM columns. Is anything over like 20-25? If so, there is a good chance that process is causing some issues. You can kill it if you want (and you know how to start it back up). sudo kill 206 will do the trick where 206 is replaced with the process id in the ps aux PID column. It is a good idea to check out processes every now and then when things are running fine so you have an idea of what is typical. That way when something is wrong, you’ll instantly notice what it is different and you can start your troubleshooting there.

The other column to note in this output is COMMAND. Command is typically what was called to startup this process. For example, often apache processes COMMAND output look something like this:

/usr/sbin/httpd.worker

Likewise, mongrel instance COMMAND output often looks like this:

/usr/bin/ruby /usr/bin/mongrel_rails start -d -e production -p 8000 -a 127.0.0.1 -P log/mongrel.8000.pid -c /var/www/apps/myapp/current --user username --group groupname

If your mongrel conf file has four instances set and you only see 3 when you run the following:

ps aux | grep ruby

That means that one of your instances of mongrel has crashed. If you have them running on 8000, 8001, 8002, and 8003 and 8002 is missing, you don’t need to restart all the dogs, you can just start the one. Simply copy the command of one of the running mongrels and replace the port with the port of the instance that crashed. So for example, if port 8002 is not showing up in your ps aux output, copy the port 8000 command, replace 8000 with 8002 near -p and -P and hit enter. You can ps aux | grep ruby again to make sure that it has in fact started up.

If you see a ton of apache workers, there is a good chance that it’s time to pimp your setup a bit. Most likely, in my experience, you have a ton of requests coming in and apache is overloading one or two of your mongrel instances and nearly forgetting about the others. Apache’s mod_proxy_balancer is easy to setup but it seems to not care if a mongrel instance is slow or not responding and keeps firing requests at it. If you have KeepAlive set to on in apache and a mongrel instance gets overloaded, you’ll end up with a crap load of httpd workers. Anyway…those are a few things that can go wrong.

Sometimes if things are running slow or stopped and all my instances of mongrel are still running a simple restart of them and an ensuing restart of apache will clear things up. On my machine, I setup alias in my bash profile to allow for easy restarting of the entire cluster and apache:

alias restart_web='sudo /etc/init.d/httpd restart'
alias restart_app='sudo  mongrel_rails cluster::restart -C /etc/mongrel_cluster/myapp.conf'

Your config and restart commands may be different, but you get the idea. You don’t want to have to think when something goes down. Typically, a restart will fix things so make it really easy to do.

free -m

Before you go cowboy and start restarting all your dogs, you might want to check out your memory and file system usage. To see how much memory you are using, you can use free -m.

The top row used value (243) will always be very close to the top row total value (254). This is because linux uses any spare memory to cache disk blocks (or so I’ve read online). The important number is the buffers/cache used value (138). This tells how much memory you are actually using. For best performance, it should be less than your total memory (254). If your buffers/cache used value (138) is higher than your total memory + your swap memory, you are going to get out of memory errors which I’m assuming would be really bad. :)

vmstat

Another way to analyze memory is with vmstat (which will be graphing in part 4).

There is no way I can explain vmstat better than this Rimuhosting article so I’ll direct quote:

The first row shows your server averages. The si (swap in) and so (swap out) columns show if you have been swapping (i.e. needing to dip into ‘virtual’ memory) in order to run your server’s applications. The si/so numbers should be 0 (or close to it). Numbers in the hundreds or thousands indicate your server is swapping heavily. This consumes a lot of CPU and other server resources and you would get a very (!) significant benefit from adding more memory to your server.

Some other columns of interest: The r (runnable) b (blocked) and w (waiting) columns help see your server load. Waiting processes are swapped out. Blocked processes are typically waiting on I/O. The runnable column is the number of processes trying to something. These numbers combine to form the ‘load’ value on your server. Typically you want the load value to be one or less per CPU in your server.

The bi (bytes in) and bo (bytes out) column show disk I/O (including swapping memory to/from disk) on your server.

The us (user), sy (system) and id (idle) show the amount of CPU your server is using. The higher the idle value, the better.

df and du

You’ll also want to keep an eye on total disk usage (until we start graphing it in part 4) so get familiar with df and du. df is for checking your total file size and du is for checking particular directories. Use the -h option with either to get more human readable file sizes.

Your best friend in times of crisis is a level head. Remember…outages happen. Computers messup. You messup. The most important thing is that you stay calm and attack logically. In the year or so that I’ve been managing rails apps, I haven’t ran into to many problems that a restart of the mongrel cluster and apache wouldn’t fix.

Did I miss anything? Have a better idea? Didn’t understand something? Leave comments below.

The ‘Keep Your Tag Cloud Running’ Series

• Introduction
• Part 1: Backups
• Part 2: Short Term Problem Analysis and Resolution
• Part 3: Automating Problem Resolution
• Part 4: Resource Reporting
• Part 5: Log rotation and analysis
• Part 6: Slow Queries and Indexing
• Part 7: General Closing Thoughts