February 20, 2011
Older: Data Modeling in Performant Systems
Newer: Hi My Name is John...
Give Yourself Constraints
Recently, I had a hernia and surgery to fix it. This knocked me out of the game and onto the couch for a couple weeks. During my recovery, I had a lot of time to think. I also had a lot of time to miss what I do every day.
This was the longest period in several years for me without creating. Once I felt good enough to get back at it, even if only for a few hours, I decided to focus all this pent up energy on something new.
What I wanted to do, was to think through a problem different than I ever have. I have been creating applications pretty much the same way for quite some time. Sure, MongoDB changed my methods a bit, but I knew I had not used it to its full potential, as I typically start all new Mongo projects with MongoMapper.
What to Build
First, I thought about what to build. I have a plethora of “someday” ideas that have never made it out of that stage. One of those ideas was to build a simple analytics program.
Sure, there are a crap ton of analytics apps out there, including Google Analytics, but not a one thus far has hit the sweet spot I am looking for.
The Old Constraint
Back in the day, I would entertain every whim I had. This is great for learning a lot of new things, but I never really focused and finished anything. What I had was a project directory full of half (or less) finished ideas.
When I actually forced myself to work on only a project or two (I chose MongoMapper and Harmony), I noticed that I actually finished things and had something to show for myself.
Important: The constraint of what I could work on made me more productive than working on whatever I was inspired to work on.
That said, rules are made to be broken, right? Plus, this new project was not entirely misguided. Harmony manages websites. Websites need analytics. There is definitely a benefit to Harmony in building an analytics system that can be deeply integrated, so I began work.
The New Constraints
Since I was bending the rules a bit, I decided to give myself different constraints this time. Rather than what I would work on, I focused on what tools I could use to do the work. The thought was that forcing myself to avoid my comfort tools would lead to thinking outside of my usual box.
The first constraint I made was that I could only use the Mongo Ruby driver. No ActiveRecord, MongoMapper or any other object mapper.
Second, no aggregate querying for reports. All of the analytics had to be built on the fly as each hit came in. I told myself I could not even store the individual hit data that was generated each time a page was tracked.
Third, no signup or authentication. I wanted to focus on the core functionality (tracking views) instead of spending my time authenticating users and all that crap.
Fourth, I cannot remember, so lets move on.
The Prototype
Within a few hours, using Sinatra and the MongoDB Ruby driver, I had a little prototype working. Each hit was a single MongoDB operation, an upsert based on the host, with year, month, day, and hour information stored in nested hashes. The nested hashes were updated in the operation using $inc. It did not do much, but it was pretty cool.
I am aware, especially now, how simple that first prototype was, but it felt great to create again. Next, I threw the app up on Heroku and MongoHQ, so I could try it out tracking this site you are reading.
Both are free to try, so it cost me nothing to get something up that I and others could react to. I showed it to the rest of the Ordered List team and every one shared the excitement.
The Result
I am amazed at what we have come up with in such a short amount of time (4 weeks using occasional evenings/weekends). In fact, the constraint of time is why Gauges is where it is today. We were really productive in the hours we snatched to work on it, because they were few and far between.
Steve came up with a great name, found a matching domain (which is a miracle these days) and Gaug.es was born.
Home Page
Dashboard
Note: Holy crap did Steve (@orderedlist on Twitter) knock the design out of the park. Dude has skills!
Conclusion
The last 4 weeks working on Gauges has made me feel, at times, like a kid again. It is almost like we were racing to the next fence post (sue me, I grew up on a farm). Sure, I broke the original constraints I set, but without them, I would never have started.
To break constraint numero uno, I used ToyStore, with my mongo adapter. Any service that involves user input needs validations and the like, and I was not in the mood to build them from scratch.
I actually stuck with constraint number two. All of the tracking code uses upserts with MongoDB modifiers to create and update reports on the fly.
Constraint number three also fell to the wayside. You cannot very well make money on a service that requires no signup (or at least it is increasingly difficult), so you do in fact have to sign up and in to use Gauges. That said, right now we are just testing things out, so we are controlling things with a code.
Whether it be the tools that you use, the projects that you work on, or the people you work with, give yourself constraints. Create something that you have always wanted. Get other people excited about it. Work cannot always be “work”.
P.S. If Gauges is something you are interested in and you would like to be an early tester, let me know. The main goals right now are hosted (no setup/servers), easy sharing with other people, how much traffic, where was it from and where was it to.
Plus, we have a few great ideas for down the road, once we get the basics ironed out. Thus far, Gauges is really scratching an itch I have had for a while and I am stoked.
12 Comments
Feb 21, 2011
Thanks for sharing the story and good luck with gaug.es! :)
Also, the constraints thing really made me think.
Feb 21, 2011
Hi John,
love the blog as usual.
Lately, I’ve had a bit of a issue deciding about tools and I thought I would share. Namely, if you aren’t using the rails orm goodness, why use rails. I’ve been playing with node.js for speed and it seems like a lot of what I need for pleasure and sugar is there. And the speed is greatly improved from rails. that plus websockets and I’m having problems understanding why I should use rails at all. thoughts?
Feb 21, 2011
@Scott Ballantyne: Enjoy the fact that we have choices and use what ever tickles your fancy. :)
Feb 21, 2011
@John Nunemaker: fair enough, to each their own
Feb 21, 2011
Looks great, John (and all of Ordered List team).
Curious. What things don’t you like about Google Analytics, and how did this project fix those things?
Feb 21, 2011
@Nate Klaiber: I like GA. I use it in addition to gauges and probably always will. That said, Gauges solves a few of the issues I have with GA.
First, GA focuses more on the past. It defaults to the last month starting at yesterday. I want to know today, right now, what is happening.
Second, it does not have a good way to see exactly what I care about in one view (all sites with views/people today).
Third, while it has some pretty parts, overall the aesthetics are pretty bland.
Fourth, GA is more of a kitchen sink. It shows you everything and allows you to slice and dice your info any way you see fit. I want something that tells me what I want to know, either in a simple interface or a daily email. This will make more sense as we work on some of the intelligence ideas we have for down the road, and it will rock.
If you just look at what we have now, all you will see (hopefully) is something far more simple and beautiful than GA. Trust me, this is just the beginning, a starting place for the real ideas we want to do. :)
Feb 21, 2011
Nice, Good deal. I know I see a lot of different analytics programs pop up, and I am curious as to how they do things differently. I agree GA is the kitchen sink. I like that it is -because it gives complete context to your traffic. I do agree that it’s not the most aesthetically pleasing. I love the UI you guys have put together.
Looks like a great start.
Feb 21, 2011
I would expect at that time, your pent up creativity would crave the freedom to retreat to your ‘comfort tools’ for something new. I love that you subjected yourself at that moment to such a narrow creative constraint. It’s completely mad and proves that there’s great art in the science of software construction… doubly so in that it was a business analytics tool. Great post and best wishes with Gauges.
Feb 21, 2011
@Scott Ballantyne: My biggest issue with node.js is the lack of documentation and tertiary functionally. Rails has a lot of stuff built in for handling request, mime types, and view helpers. I’ve tried working on a few apps with node.js and usually end up reinventing various ruby libraries.
Jul 28, 2011
How do you deal with missing data? I have often wondered what the “right” approach should be.
Given Gauges’ model, let’s say no one hits a site in a particular hour. How do you track that?
I think I’ve tried every variation and none seem to be all that efficient and most are fragile.
Jul 29, 2011
@Derek: None of the above. Say the request is for the past 10 days traffic for a site that had no views for 2 of those days. First we request the data from the db, then we loop through the past 10 days, setting each day to its appropriate value. If the day is not found in the database result, then we just return 0 or whatever would be appropriate.
The goal is to have an API that always returns days and data, not necessarily that the data is exactly like that in the database. Hope that helps.
Oct 27, 2011
YTJW7r http://KkRzxaxyKwfz3eqX.com
Thoughts? Do Tell...