Flipper: Insanely Easy Feature Flipping

Cross posted from JohnNunemaker.com as it seems relevant here too.

                                   __
                               _.-~  )
                    _..--~~~~,'   ,-/     _
                 .-'. . . .'   ,-','    ,' )
               ,'. . . _   ,--~,-'__..-'  ,'
             ,'. . .  (@)' ---~~~~      ,'
            /. . . . '~~             ,-'
           /. . . . .             ,-'
          ; . . . .  - .        ,'
         : . . . .       _     /
        . . . . .          `-.:
       . . . ./  - .          )
      .  . . |  _____..---.._/ _____
~---~~~~----~~~~             ~~

Nearly three years ago, I started work on Flipper. Even though there were other feature flipping libraries out there at the time, most notably rollout, I decided to whip up my own. Repeating others is, after all, one of the better ways to level up your game.

My main issue with rollout was that it was inflexible. You couldn’t change the ways in which a feature was enabled (ie: adding percentage of time rollout). You had to use redis. The list goes on. I poked around and couldn’t find anything like what I was looking for and I was in the mood to create, so I started flipper.

Most of the work was done off and on over the course of a few weeks. At the time, I was working on traffic graphs for GitHub and I wanted a way to turn features on/off in a flexible way.

Naming is hard

Flipper started as a simple ripoff of rollout with the primary difference being the use of adapters for storage instead of forcing redis. I struggled through awkward terminology and messy code for a while, until a great conversation with Brandon Keepers led me to the lingo flipper uses today: Actor, Feature and Gate (thanks Brandon!)

An actor is the thing trying to do something. It can be anything. On GitHub, the actor can be a user, organization or even a repository. Actors must respond to flipper_id. If you plan on using multiple types of actors, you can namespace the flipper_id with the type (ie: “User:6”, “Organization:12”, or “Repository: 2”).

A feature is something that you want to control enabled-ness for. On SpeakerDeck, I have a feature for search. With the click of a button, I can disable search if it is causing issues. On GitHub, we do thousands of feature checks per second across nearly 30 features (at the time of this writing) in different states of enabled-ness. If I told you what they were for I would have to kill you.

A gate determines if a feature is enabled for an actor. There are currently five gates — boolean, actor, group, % of actors and % of time. Amongst these you can rollout a new feature or control an existing one in whatever way you desire.

The Gates

The boolean gate allows completely enabling or disabling a feature. Think of it as a short cut to turning a feature fully on or fully off quickly. Enabling the boolean gate means the feature is on all the time for everyone. Disabling the boolean gate clears all enabled gates so the feature is completely off. Think of disable like a reset.

flipper = Flipper.new(adapter)
flipper[:search].enable # turn on
flipper[:search].disable # turn off
flipper[:search].enabled? # check

The actor gate allows enabling a feature for one or more specific actors. If you wanted to enable a new feature for one of your friends, you could use this gate.

flipper = Flipper.new(adapter)

flipper[:search].enable_actor user # turn on for actor
flipper[:search].enabled? user # true

flipper[:search].disable_actor user # turn off for actor
flipper[:search].enabled? user # false

The group gate allows enabling a feature for one or more groups. A group is a named block of code that returns true or false for a given actor. You could have a group for everyone in your company, or only engineering, or perhaps all users in the US or Europe. Anything your heart can imagine can be converted to a group and the entire group can be enabled at once.

Flipper.register(:admins) do |actor|
  actor.respond_to?(:admin?) && actor.admin?
end

flipper = Flipper.new(adapter)

flipper[:search].enable_group :admins # turn on for admins
flipper[:search].disable_group :admins # turn off for admins

person = Person.find(params[:id])
flipper[:search].enabled? person # check if enabled, returns true if person.admin? is true

The percentage of actors gate allows slowly enabling a feature for a percentage of actors. As long as you continue to increase the percentage, an actor will consistently remain enabled. This allows for careful rollouts of a feature to everyone without overwhelming the system as a whole.

flipper = Flipper.new(adapter)

# turn search on for 10 percent of users in the system
flipper[:search].enable_percentage_of_actors 10

# checks if actor's flipper_id is in the enabled percentage by hashing
# user.flipper_id.to_s to ensure enabled distribution is smooth
flipper[:search].enabled? user

# turn search off for percentage of actors, other gates could retur true still
flipper[:search].disable_percentage_of_actors # sets to 0

The percentage of time gate allows enabling a feature for a random percentage of time. This is great for dark shipping and load testing. We actually used somehing similar to this to launch traffic graphs. We wanted to be positive that we could stand up to real traffic, so we performed ajax requests behind the scenes based on a percentage of time to the new feature. This allowed us to crank up the traffic, hit a bottleneck, kill the traffic, fix the bottlneck and repeat.

flipper = Flipper.new(adapter)


# turn on logging for 5 percent of the time
# could be on during one request and off the next
# could even be on first time in request and off second time
flipper[:logging].enable_percentage_of_time 5

# turn off logging for percentage of time
flipper[:logging].disable_percentage_of_time # sets to 0

All the gates are fully documented in the flipper repo as well.

Adapters

The adapter pattern is used to store which gates gates are enabled for a given feature. This means you can store flipper’s information however you desire. At the time of this writing, several adapters already exist, such as in memory, pstore, mongo, redis, cassandra, and active record. If one of those doesn’t tickle your fancy, creating a new adapter is really easy. The API for an adapter is this:

• features – Get the set of known features.
• add(feature) – Add a feature to the set of known features.
• remove(feature) – Remove a feature from the set of known features.
• clear(feature) – Clear all gate values for a feature.
• get(feature) – Get all gate values for a feature.
• enable(feature, gate, thing) – Enable a gate for a thing.
• disable(feature, gate, thing) – Disable a gate for a thing.

At GitHub, we actually use a SQL adapter fronted by memcache for performance reasons.

Instrumentation

Flipper is wired to be instrumented out of the box, using ActiveSupport::Notifications API (though AS::Notifs are not specifically required). I even included automatic statsd instrumentation for those that are already using statsd.

require "flipper/instrumentation/statsd"
statsd = Statsd.new # or whatever your statsd instance is
Flipper::Instrumentation::StatsdSubscriber.client = statsd

If statsd doesn’t work for you, you can easily customize wherever you want to instrument to (ie: InfluxDB, New Relic, etc.).

Performance

Flipper was built based on my time working on Words with Friends and to be used at GitHub, so you can rest easy that it was built with performance in mind. The adapter API is intentionally made to allow for fetching all gate values for a feature in one network call and there is even (optional) built in memoization of adapter calls, including a Rack middleware which enables memoizing the fetching a feature for the duration of a request.

I’ve also thought about making it easy to allow for batch loading of features, though I haven’t needed this yet on any site I’ve worked on, so for now it remains a thought rather than an implementation.

Web UI

As a cherry on top, I’ve also created a rack middleware web UI for controlling flipper, which can be protected by any authentication you need. Below are a couple screenshots (at the time of this writing).

List of features

Viewing individual feature

All the gates can be manipulated to enable features however you would like through the click of a button or the clack of a keyboard.

Conclusion

Flipper is ready for the prime time. As I said earlier, we are now using it on GitHub.com for thousands of feature checks every second. The API changed a bit in 0.7, but is pretty stable now. Drop it in your next project and give it a try. If you do, please let me know (email or issue on the repo) as I love to know how people are using things I’ve worked on.