# Don't generate fancy HTML for stack traces.
disable :show_exceptions
# Allow errors to get out of the app so Cucumber can display them.
enable :raise_errors

Both are required. If you just disable :show_exceptions, you’re spared the fancy HTML, but you’re also spared any useful information about what’s going wrong. (You just get “Internal Server Error”.) But if you just enable :raise_errors, well, the errors don’t get raised - they’re trapped by the exception-showing code.

Watchr is a tiny program I wrote five years ago. Ken and I were talking about how we’d like something that pops up images when they’re posted to Flickr. For reasons I don’t now recall, we then wrote two separate applications, his in Python and mine in Java. (By the way, that page is the only use so far of erikostrom.com, except as a mirror of drowning.org, the equally dated site about my singer-songwriter career.)

I’m a little surprised to find that it still basically works. Write once, run everywhere, I guess! Although it still has the same Mac bug noted in 2005, which bothers me more now that I’m a Mac user.

Today I got FireWatir working! It wasn’t that hard, but I had to consult a few different web pages, so here it is all in one place.

1. Via FireWatir Installation, install the gem:

   sudo gem install firewatir

   and the JSSh Firefox extension (Mac FF 3.5 version) to make Firefox steerable.

2. Set up a Firefox profile, so you can run tests without all your weird extensions in play, and without messing with your personal browser session and history. Mine is called “tester”.
3. Start up Firefox with JSSh and your new profile (you can do this while Firefox is running):

   /Applications/Firefox.app/Contents/MacOS/firefox -jssh -P "tester"

4. In an irb session, try some stuff - for example, on a new Spree instance:

   ff = FireWatir::Firefox.new
   ff.goto 'http://localhost:3000/'
   ff.link(:text, 'Clothing').click

Voila! Now I want to try Cucumber and Watir.

javascript testing is different from desktop/server testing
- cross-browser issues

there are a lot of javascript testing frameworks!
- long tail
- it’s easy to write one

basic components for a testing framework
- the usual, plus async tests
- minimum: assertions
(’glorified assertion-and-logging framework’)
- async tests
- pause (stop executing other tests) and resume

why write your own?
- a good way to understand what testing is for
- not too many cross-browser issues

popular frameworks:
- (none)
- JSUnit
- 2001, and looks it
- YUITest
- good code, supports async tests and event simulation
- (looks like nested contexts?)
- QUnit
- async testing, test timeouts
- no dependencies
- easy to use
- FireUnit
- firebug extension

standardization
- CommonJS: standard API across client and server environments
- including testing API, now adopted by QUnit (I think)

server-side testing of client-side code
- usually Java + Rhino
- Crosscheck (Java)
- Env.js (pure JS)
- BlueRidge (Env.js + Screw.unit + Rhino)

browser launching
- WebDriver (Java)
- Watir (Ruby)
- JsTestDriver (Java)
- Selenium RC (Java)

distributed (for scale)
- Selenium Grid - on Amazon servers
- TestSwarm - crowd-sourced

PART II: Accurately Measuring JavaScript

major use cases:
- same code, different platforms
- which browser is fastest?
- different code, same platform
- which framework is fastest?
- how should we implement this?

same code, different platform
- SunSpider (WebKit), V8 (Chrome), Dromaeo (Mozilla)

… statistics…

error rate
- faster tests => less accuracy
- faster *browsers* => less accuracy
- dromaeo/v8 test runs/sec instead of msec/run
- use harmonic mean to average

different code, same platform
- most solutions are bad
- few runs, inaccurate
- garbage collection
- use mode instead of mean
- less accurate than mean, but more consistent
- don’t discard “bad” results - GC is significant
- may reflect memory issues
- getTime() “very very imprecise”
- 15ms intervals on XP (except FF/Chrome)
- IE in Wine gives you precision, but it’s not IE on Windows

- browser tools
- Safari, Firebug, IE 8 profilers
- DynaTrace Ajax for IE 6-8
- Shark - watches internal function calls

PART III: jQuery 1.4
- complexity reduction
- bubbling change, submit, focus, blur
- required script loading

analyzing performance
- stop measuring absolute execution time
- improve code quality and flow

complexity analysis
- count function calls instead of seconds
- may indicate not slowness, but poorly written code
- remove() did a lot more work than it needed to
- and so did everything that used remove()

event bubbling in IE
- focus, blur, change, and submit don’t bubble
- alternative solutions: piggyback on events that DO bubble
- focus/blur: use focusin/focusout
- submit: watch for click… or ‘enter’ keypress…
- change: basically reimplement the whole event
- track previous value
- watch for keyup navigation, checkbox clicks
- beforeactive on radio buttons

jQuery.require()
- don’t try to load a script multiple times
- load synchronously if needed for dependent code
- load asynchronously if you can get away with it!

URL mapping
- require(’ajax’) => ajax.js
- namespaces expand to URLs

PART IV: HTML 5 Elements in IE

problem #1
- unknown elements are unstyled, and “inner contents escape”
- fix: document.createElement(’section’) makes section work
- html5shiv does that for all HTML5

problem #2
- unknown elements can’t hold other unknown elements
- even with html5shiv
- no fix

problem #3
- nodeName of unknown elements is treated as XML
- fix: don’t assume nodeName is uppercase

problem #4
- injecting unknown elements via innerHTML fails
- “fix”: parse HTML and build DOM in JS

Q&A:

… i zoned out…

Q: clojure?
A: framework… okay. compiler… pretty slick!
- but designed to compile your whole application
- can’t compile the framework separately

Last night I spent at least an hour trying to figure out why I was getting a nil.rewrite error from Rails’s url_for method. I’ve seen hard-to-debug errors from there before, so I dug right down into it, but I couldn’t figure out why the @url variable was set in some cases and not in others… and then I determined that it actually was being set in all cases, but then it was mysteriously blank by the time url_for needed it….

And then I realized it was because I had a @url variable in my own controller action. And then I went to bed.

Most of the time I really love how dynamic and open Ruby is as a language, but sometimes private instance variables are just what you’d like to have had.

should "display the suggestion as a link" do
  assert_select 'a[href=?]', @click_url, @display_url
end

… says there should be an <a> tag with the given href attribute and content. Clearer and less brittle than computing a string and searching for it.

But what if you want to assert something about the HTML structure of something that isn’t the controller response? Like, if you want to look for a link in the flash hash?

I just added this to my test_helper.rb:

class ActiveSupport::TestCase
  def assert_select_from(text, *args)
    @selected = HTML::Document.new(text).root.children
    assert_select(*args)
  end
end

Now in my tests I can use the full power of assert_select on an arbitrary string:

should "provide an undo/delete link" do
  assert_select_from(flash[:notice], 'a[href=?]',
    venue_path(assigns(:venue)), 'undo')
end

The idea is simple: RailsBridge has a lot of energy. The Rails Lighthouse has a lot of open tickets. With the help of some Rails Core team members, we’re going to see what we can do to cut down the number of open tickets, encourage more people to get involved with the Rails source, and have some fun.

Definitely a success - reportedly at least 120 tickets were closed, I’m sure a lot of new people got involved, and the people on #railsbridge seemed to be having fun.

I meant to get involved early - even thought about driving to San Francisco to join some other bug mashers face to face - but instead I spent most of the weekend tracking down an obscure interaction in HtmlUnit/Rhino, Yahoo! Maps, and Prototype. What with that and two Tin Cat shows, I didn’t get around to the BugMash until Sunday afternoon.

Still, I did get to it. And now I’m officially a Rails contributor! Those two contributions are actually one - the early version that was committed by accident, and the later, more thorough revision. (I also shepherded this new feature into the core.)

So that’s great! I’ve been wanting to help out somehow with Rails for a while, but it’s, you know, big. It helped that someone had gone through the open tickets and picked out good ones for us to work on. It helped that there were people around on IRC to talk about the code and how to recover from newbie git mistakes. And it helped that there was a bot ready to supply a new project every time I finished one (or thought I’d finished it). I’m a sucker for a steady flow of work - in my head I’m still typing “!gimme” at the IRC prompt.

You may want to trigger [reindexing] via a cron job on a recurring schedule (i.e. every day) to update the Xapian database. However it will only take effect after the Rails application is restarted because the Xapian database is stored in memory.

I see the Xapit Sync project to fix this has since ceased to be vaporware. Well, maybe next time.

Anyway, so RubyTrends tells me the cool kids use Thinking Sphinx, and I want to be cool, but I’m running Touring Machine on the cheap - on a shared DreamHost server - and they don’t want me to run server processes, and although some guy on the Internet says it’s probably fine, I’m leery of defying them. But last week they were running a discount offer on DreamHost PS, their quasi-VPS service - no root, but you can run whatever you want, within the resources (memory, CPU) you pay for.

Sounds like a fine place to run the search engine, but I didn’t want to run the whole Rails app there - since I’ve already got a place to run it that is effectively free. (DreamHost is very cheap, and I run a bunch of sites on it.) So I set out to run a distributed site, with the web app running on DreamHost shared hosting, and the search engine running on DreamHost PS.

It took some setup, but I think I’ve got it working. Here are some notes. As usual, this isn’t a tutorial. You should read everyone else’s instructions - especially the official Thinking Sphinx documentation, and J. Wade Winningham’s post about Capistrano tasks. (I didn’t use his deploy.rb, though.)

The Approach

As I said, I’ve got software installed on two machines - call them app (the web application) and search (the Sphinx server). (app has no special access to search - it’s open to DOS and other attacks. I’ll fix it later.) So Capistrano must be told about the new server and its new role. In deploy.rb:

role :search, "USERNAME@SEARCHHOST"

To keep things simple (sort of), I deploy the whole codebase to both hosts. app is configured via the DreamHost control panel to run Passenger out of the Rails directory; search is configured not to, and I manage the search server via Capistrano.

There doesn’t seem to be a way not to run a web server on the PS host, so I created a public/redirect directory, told DreamHost to use that as my web directory, and put this in public/redirect/.htaccess:

Redirect 301 / http://touringmachine.org/

The standard Capistrano recipes assume you want to deploy to the same directory on each host. Unfortunately DreamHost won’t let you have the same username on both a shared and a PS host - and you can’t install your app anywhere outside of your home directory. Stuff breaks if you use a relative path for installation. This worked for me, at the top of deploy.rb:

set :application, "touringmachine.org"
set :applicationdir, "`(cd ~; pwd)`/#{application}"

Thinking Sphinx

Thinking Sphinx’s Sphinx installer can’t be used as-is for DreamHost PS, for two reasons: It fails if PostgreSQL isn’t available, and it assumes you have sudo permission. I’ve submitted a revised version that fixes both problems. (It’ll still use sudo if you set :use_sudo to true.) For now, you’ll need to copy my capistrano.rb into your project if you want to use it, (Update: it’s now in the master repository, and should be in the next gem version) and add this to deploy.rb:

set :thinking_sphinx_configure_args, "--prefix=$HOME/software" # where to install Sphinx
require "#{File.dirname(__FILE__)}/thinking_sphinx_capistrano"

You’ll also need this in config/sphinx.yml:

production:
  ...
  bin_path: '/home/USERNAME/software/bin'

Thinking Sphinx lets you specify a path to keep the indexes in, and comes with a task to create the path when you set up the server. Unfortunately the task is assigned to the app role. I haven’t patched Thinking Sphinx for this yet - I just put this in deploy.rb (after the above snippet):

namespace :thinking_sphinx do
  # Override TS's task to use :search instead of :web, and not use sudo
  desc "Add the shared folder for sphinx files for the production environment"
  task :shared_sphinx_folder, :roles => :search do
    run "mkdir -p #{shared_path}/db/sphinx/#{rails_env}"
  end
end
 
after "deploy:setup", "thinking_sphinx:shared_sphinx_folder"

The other tasks assume you want them run on all machines; for now I’ve dealt with that by using, e.g., cap ROLES=search thinking_sphinx:install:sphinx to do Sphinx stuff.

Delayed Deltas

I used delayed deltas to keep the index up to date. The delayed_job plugin also comes with Capistrano tasks for managing a DJ daemon. (DJ is packaged with Thinking Sphinx, but you should install the latest version as a plugin.) cap delayed_job:start runs rake delayed_job:start on the server. This Rake task loads the whole Rails environment, then forks itself to run as a daemon. Combine this with Sphinx itself and an ssh session, and suddenly you’re using too much memory for a minimal DreamHost PS server (150M + swap). The forked daemon is killed before it can start.

I could upgrade, but it’s pretty ridiculous: You don’t need a whole Rails environment to start a daemon. I wrote a lo-fi replacement for rake delayed_job:* that does only what I need:

#!/bin/sh
PIDFILE=tmp/pids/delayed_job.pid
command=$1
cd `dirname $0`/..
case $command in
  start)
    nohup script/delayed_job run 2>&1 >>log/delayed_job.log &
    
  stop)
    if [ -f $PIDFILE ]
    then
      pid=`cat $PIDFILE`
      rm $PIDFILE
      kill $pid 2>/dev/null
    fi
    
  restart)
    $0 stop
    $0 start
    
esac

Put that in script/dj, and add another chunk to deploy.rb:

namespace :delayed_job do
  desc "Stop the delayed_job process"
  task :stop, :roles => :search do
    run "cd #{current_path}; env RAILS_ENV=#{rails_env} script/dj stop"
    sleep 1
  end
 
  desc "Start the delayed_job process"
  task :start, :roles => :search do
    run "cd #{current_path}; env RAILS_ENV=#{rails_env} script/dj start"
  end
 
  desc "Restart the delayed_job process"
  task :restart, :roles => :search do
    run "cd #{current_path}; env RAILS_ENV=#{rails_env} script/dj restart"
  end
end
 
before "thinking_sphinx:configure", "delayed_job:stop"  # limited resources
after  "thinking_sphinx:configure", "delayed_job:start" # limited resources
 
after "deploy:stop",    "delayed_job:stop"
after "deploy:start",   "delayed_job:start"
after "deploy:restart", "delayed_job:restart"

(Note that you can’t get by on delta indexes alone; I have a cron job that rebuilds the Sphinx index every now and then.)

Of course, you also don’t need a whole Rails environment to wake up every 5 seconds, see if there are tables that need reindexing, and fire off a request to the Sphinx server. If I want to keep this site cheap, I may need to optimize delayed_job out of the way. But for now, this is working, and it’s time to get back to making the app cool.

Yahoo! and Google both do this automatically if you search for something they recognize as an address. But Google doesn’t seem to provide an API. Yahoo! Local does, and the YM4R gem already provides a nice Ruby wrapper for it. So:

  def make_suggestions_based_on_address
    businesses = Ym4r::YahooMaps::BuildingBlock::LocalSearch::get(
      :location => address.block,
      :radius => 0.001,
      :results => 20,
      :query => '*'
      )
    businesses.each do |business|
      add_suggestion(:name, business.title)
      add_suggestion(:url, business.business_url) unless business.business_url.blank?
    end
  end

You might wonder about the radius parameter. Technically, Yahoo! doesn’t provide the service I want; I want “tell me what’s at this address”, it has “tell me what’s near this address”. With a radius of 0, it returns up to 10 results. After a little experimentation, it looks to me like 0.01 is a tight enough radius to get me all the businesses at an address - or at least the top 10 - and none of the ones next door.

This is actually one of two ways I’m subverting the API here. It’s intended for search, not simple lookup, which is why it’s a little awkward to get a simple address. It’s also why Yahoo! provides multiple URLs for each business. The BusinessUrl is the actual URL for the business’s web site. There’s also a BusinessClickUrl, which “contains extra information that helps [them] to optimize [their] search services. Yahoo! requests that you display the BusinessUrl, but link to the BusinessClickUrl, so they can track usage.

The latest code actually uses both URLs, but I’m going to stop here for now.

UI

• As the user enters data in the new venue form, if we’re able to make any guesses, we pop them up in a panel to the right of the form.
• Each suggestion is a link. If a user clicks a suggestion link, the suggestion is copied into the corresponding form field, and the field is given focus, so the user can edit it. (For example, if the “page title” suggestion has additional text around the venue name.)

There’s also an “accessible” JavaScript-free UI, but it’s bad, so I’m going to pretend it’s not there.

Client and Server

I started drawing an interaction diagram, but it’s really standard Rails stuff:

• Model, controller, and view produce the new venue form.
• When any form input’s value changes, we submit the whole form (as an AJAX GET) to the controller’s suggest action.
• The server responds with an RJS view that populates and shows the suggestions panel (if there are any suggestions) or hides it (if not).
• The suggestions are links that call a JavaScript function that copies, highlights, and focuses.
• Once the user is done editing, the create action just works as normal.

Model

In keeping with the Skinny Controller, Fat Model approach, the only interesting code is in the model. It took me a couple of tries to get a design I liked, but eventually I went with Rails-style magic:

  def make_suggestions
    @suggestions = {}
 
    methods.grep(/^make_suggestions_based_on_/).each do |method|
      attrs = method.sub('make_suggestions_based_on_', '').split('_and_')
      values = attrs.map {|attr| send(attr)}
      send(method) unless values.any?(&:blank?)
    end
 
    suggestions
  end

When asked for suggestions:

• We look through the Venue object for methods of the form make_suggestions_from_attr_and_other_attr. (We support multiple attributes so that, for example, given a venue name and vague address - city and state - we can look up the exact address.) Call these “suggesters”.
• We call each suggester only if we have non-blank values for all the named attributes (because how are you going to base a suggestion on the venue’s address if you don’t know its address?). Note that although I’m saying “attributes”, this code works for attributes, associatons, and in fact any method. This is important because a venue’s address is actually a has_one association.
• The suggesters populate @suggestions, and then we return it.

A small thing I’m pleased by: The make_suggestions method works both as “create suggestion objects” and in the idiomatic sense of “make a suggestion” as “suggest something”.