HTTParty Divorces JSON

So I guess dropping ActiveSupport just wasn’t enough for you people. You demanded more! Almost right away the windows and jruby people started whining (I kid, I kid). Oh, and then Pratik got a little bothered by the JSON gem’s interference with Rails. Let me just say that the HTTParty is not an exclusive one, so I figured I’d look into making life easier for the aforementioned party poopers.

Tonight, after watching a couple movies, I felt like hacking, so I decided to remove the JSON gem as a dependency. I remembered that ActiveSupport actually uses YAML (a core library) with some tweaks to parse JSON, so I pulled in that bit of code and HTTParty was officially free of its only remaining chaperone. Yep, now when you install HTTParty, no other gems install.

I also did some refactoring, fixed a few bugs, added a ton of specs and started to document the code base a bit. Nothing super new other than the lack of the JSON gem, but I figured I would officially invite windows, jruby and Pratik to the party.

What next?

So what is next for HTTParty? I’ve been thinking about that a lot and I have a few ideas. I’d love to see integration of HappyMapper and HTTParty. I’m thinking this would be pretty easy with a :parser option in the request methods. Maybe something like this:

HTTParty.get('http://foo.com', :parser => Foo)

Any thoughts or preferences on something like this? Then when HTTParty attempts to parse the response, it would call Foo.parse, so you could make your own parsing classes or use HappyMapper objects. Any better ideas? Also, do you have any pain points with HTTParty? Things you find monotonous when consuming web services that would be well suited for the party? Let me know.

13 Comments

Michael Siebert
Jan 31, 2009
what’s the deal with the json gem?
it’s fast and it works great? so why hack around that much?
and for jruby and windows people:

gem install json-pure

this is a pure ruby library so everbody having a ruby installed can use it.
what about something like
```
begin
  require "json"
rescue LoadError
  require "yaml-json-parser-thingy"
end
```
this way we have the best of both worlds: fast yaml if installed and no dependencies otherwise. and fast yaml even on jruby/windows/pratik
Carlo Zottmann
Jan 31, 2009

Hi, last time I’ve checked, HTTParty didn’t support retries. I’d like that… Maybe you could you add something like retryable, please? http://github.com/carlo/retryable/tree/master

I mean, I could wrap the HTTParty calls in a retryable block, but that kind of feels like doing it backwards. :)

Thanks,
C.
cheapRoc
Jan 31, 2009

I had to say, that :parser option looks insanely scrumptious!!!!
Charles Oliver Nutter
Jan 31, 2009

Great news! The JSON gem is a perfect example of why working around Ruby’s performance issues ends up limiting apps to MRI. I hope we can start to escape that with other non-MRI-specific options. Kudos for moving to a YAML solution. Is there anything that could be released as a gem to turn built-in YAML support into JSON support, as ActiveSupport does?

Michael Siebert: The problem is that unless you release separate gems, you can’t specify a dependency on a native extension without RubyGems failing your install on JRuby or Windows. If JRuby allowed specifying per-impl dependencies or “equivalent” gems, we would not have that problem. Or you would have to have no dependency on the gem and people would have to know to install it.

That said, we’d love to have a fast JSON extension for JRuby just to deal with the current unfortunate situation.
Pratik
Jan 31, 2009

Heyy ! Thanks a lot :)
John Nunemaker
Jan 31, 2009

@cheapRdoc – I know, right? :)

@Charles – It would be really easy to turn the AS JSON stuff into a gem. It is just one file in HTTParty. Also, good job of explaining the gem dependency issue to Michael.

@Pratik – You’re welcome. :) Hope it works out ok for you.

Don

Jan 31, 2009

I had a similar thought to the parser option that I was looking to add at some point. Something like:

HTTParty.add_format( "application/atom" => Proc.new { |body| blahbla }) </pre>I think it would be cool to have something like this where you could add/override the handling of a content type for every HTTParty class, but then again I hadn’t thought of the use case for sending a specific parser to individual calls. Then again you can sort of do anything you want with that patch that returns the original response body, it’s just not as fancy.

Don
Jan 31, 2009

Awww, your comment preview juked me on the greater-than sign in the hash.
Kevin
Feb 01, 2009

This seems like a question that occurs all over the place. Where do you draw the line between not repeating work and not depending on a ton of gems/libraries?

And when the problem is compatibility, when do paper over the compatibility problem by removing the dependency vs going and fixing the compatibility problem in the original library or gem? (assuming its open source). I’m struggling with that question when it comes to
Shoes and event handling.

I’d be interested in your thoughts on the more general case here.

Oh, and btw… whatever version of textile the comments are using, I found it impossible to add a direct link containing “s ~~” and “~~ s” (without spaces), as it would translate it to “s-stuff-s”. I searched for ways to escape text, but all recommended ones didn’t work, so I used tinyurl instead.
Brandon
Feb 02, 2009

Thanks John, I appreciate the awesome feedback and work you have done. I am using HTTParty in a production app at work now. I did add my own retry support in, so some sort of configurable retry support in HTTParty would be cool, but it isn’t hard to do yourself (likely everyone would have different retry requirements).
John Nunemaker
Feb 02, 2009

@Kevin – I don’t have a hard and fast rule but if there is currently people have dependency issues and I can add one file to the library to remove that problematic dependency, I go for it. In HTTParty’s case, originally I was including all of ActiveSupport for a two files of functionality. I added two files and the dependency and bulk was gone so it is a win in my book. Make sense?

The textile comment preview is all javascript so it might not be exact. In general if it is suppose to work with textile, it will work when you save the comment even if the preview isn’t quite right.

@Brandon and @Carlo – Do you guys have retry examples? I haven’t needed to do that yet so I don’t have an example to extract a pattern for. Maybe one of you would be interested in adding it to HTTParty with my help?
Michael Siebert
Feb 03, 2009
@charles: I never mentioned re-adding json as a dependency, just use it it installed. Another thing I stumbled upon just now is a bug (?) in active support’s json handling:
```
&gt;&gt; ActiveSupport::JSON.decode "ü".to_json
=&gt; "\\u00fc"
```
Which kinda sucks…. :)
Carlo Zottmann
Feb 04, 2009

Hi John,

here’s an example bit of some of my production code.

And if you don’t want any gem requirements, here’s the piece of code that got me started. :)

Hope this helps,
C.