Interacting with ElasticSearch using Hubot

At OpenTable, we use a few ElasticSearch clusters. Our aim was to be able to interact with our ElasticSearch clusters via HipChat so that we could troubleshoot easily and without having to log into our VPN. We already use Hubot as part of our systems workflow, so it made sense to be able to interact with ElasticSearch with it.

Setting a cluster alias

When a pager wakes me at 3am, I really do not want to have to try and type the cluster URL into my mobile hipchat client. So the first thing that was added to the script was the ability to give a cluster an alias.

elasticsearch add alias my-test-alias


This allows us to use that alias for all commands going forward. Please note that you can remove and query aliases as well:

[Read More]

Coach don't rescue

I recently attended a fascinating and emotionally-charged talk by Samantha Soma at DareConf 2014, ‘How to stop rescuing people’. It strongly mirrored my experience of moving into a leadership role and I’d recommend anyone with a spare 30mins to watch it.

Samantha’s talk made me reflect on how I struggle to coach talented individuals; how I can identify when it’s going wrong and what steps I can take to remedy the situation.

Gold star syndrome

A new concept for me and a recurring theme throughout the sessions at DareConf, ‘Gold Star Syndrome’ is a fixation on finding validation for your work. I feel this is a result of early childhood values spinning the perception of working life away from the actual reality. As a child, especially during our school years, we discovered that when we do good things, good things happen to us. Remember how it felt to get that gold star in your spelling test or that A+ on an English essay?

That was great in school and even through to University but working life is a much more terse environment and getting positive reenforcement is much less common. As professionals, the majority of our work goes unnoticed - until there is a problem or issue to solve. Then we feel open to the stinging criticism but resentful that months of good work went unnoticed.

When we continue to search for a gold star or continually strive for perfectionism we, as individuals, become much more insular and isolated. We tend to avoid showing our work until it is 100% ready and put up a shield to protect us from any feedback, in fear of being made to look stupid or being called a fraud.

[Read More]

Hobknob v1.0: Now with authorization

We are pleased to announce the version 1.0 release of Hobknob, our open-source feature toggle management system. With it comes a few additions and several improvements.

This post will expand on some of the changes, in particular, authorisation via access control lists.
For an introduction to Hobknob, see our previous post: Introducing Hobknob: Feature toggling with etcd.

Authorisation with ACLs

A much requested feature was the ability to control who can add/update/delete toggles on an application by application basis. We achieve this via the use if an Access Control List for each application. Users that are part of the ACL for an application are known as application owners.

Hobknob Owner List

Application owners can (for an owned application):

[Read More]

PuppetConf 2014 - Part 3

Day 2

This is our summary of PuppetConf 2014. In our previous post we gave an overview of the first day of the conference. This post will provide an
overview of the final day.

There were even more inspiring keynotes and lots more talks which have given us plenty of ideas to go home and think about.

Key Notes

Animating the Puppet: Creating a Culture of Puppet Adoption - Dan Spurling (@spurling), Getty Images - Slides

Dan Spuring, VP of Tech Services at Getty came out of the gate with a strong message. His GSD t-shirt
giving you a clear understanding of who he is. His talk about creating a culture of Puppet adoption at his company was a great story of how challenging it
can be to move various business units with projects of various ages to a configuration-management (with Puppet) ethos.

[Read More]

PuppetConf 2014 - Part 2

Day 1

This is our summary of PuppetConf 2014. In our previous post we gave an overview of the contributor summit. This post will provide an overview
of the first day of PuppetConf.

As you might expect there were great keynotes with plenty of announcements and too many talks for us to attend. We have provided an outline for all the talks
we did attend and links to those we didn’t.


Nearly a Decade of Puppet: What We’ve Learned and Where We’re Going Next - Luke Kanies, PuppetLabs - Slides

The big keynote of the event to kick off the first day from the author of Puppet himself. This was obviously going to be a tweet worthy affair full of photos
and big announcements and it did not disappoint.

[Read More]

PuppetConf 2014 - Part 1

The start of PuppetConf 2014

It has been one week since our attendance at this years PuppetConf and we have just now caught up on all the great talks that were
given and the projects demonstrated over the 3 day period. Here’s our summary of the event (split into 3 parts), hopefully you will
find as much inspiration in the content as we have.

Day 0 - Contributor Summit

For the first time, this years Puppet Contributor Summit was held the day prior to the conference itself and I think this was a great idea.
Most of the Puppetlabs staff and many of the high profile community members were in town for PuppetConf anyway so it made sense. There was
roughly 60-70 people in attendance both senior contributors and people new to the community so it was a great mix that led to some
fantastic discussions.

The day itself had two tracks: a module track for forge modules and a core track for people contributing to puppet and factor.

Those of you who have seen our forge module page will understand why we chose to stay in the module track.
Although I heard there were many great discussions to be had with regards to Puppet 4 in the core track.

[Read More]

Introducing Hobknob: Feature toggling with etcd

The ability to dynamically turn features on/off in software without the need to redeploy code is extremely beneficial. Whether you are trialing a new feature or using branch by abstraction to avoid creating feature branches, the use of feature toggles can aid continuous delivery and provide a mechanism to reduce mean time to resolution when an issue occurs.

With a relatively large engineering department with multiple teams spread across the US and UK the need to manage feature toggles has evolved to the point whereby individual teams have developed their own implementations. Most of these are simple config files.

We decided to unify this effort by providing a central place to store feature toggles, provide a dashboard to be able to turn these toggles on/off and provide language specific clients to integrate into our software components.

The results of this was Hobknob.

Why etcd?

We made the decision to use etcd. Etcd is “a highly-available key value store for shared configuration” ( It provides a HTTP API to store and retrieve data. This is what makes it perfect for a feature toggling solution used by multiple components. It means that we didn’t have to write an intermediate API on top of a data store for consumers.

[Read More]

Testing Puppet with Beaker pt.3 - Testing Roles

In the first two parts of this blog series we have focusing on testing puppet modules with beaker. As an open source contributor there is always
a large test matrix so this makes absolute sense. But what about the other large use-case for beaker - what about our day-to-day internal code base?
Not all of this is modules, in fact a large portion of it is other puppet code - roles, profiles, facts, hiera data etc. All of this needs testing
as well.

In this blog post I will be showing how we have started using beaker to test our puppet roles and profiles for both Linux and Windows.


Prior to this post all our beaker testing has been master-less i.e. using using puppet agent apply. This is perfectly adequate for most use cases when
testing modules in isolation but doesn’t always work when testing an internal code base (unless you are masterless there as well then please skip to the next section).

At OpenTable we do use a central puppet master to compile our catalogs. So when testing our puppet roles we wanted to make sure that we were also testing
with a master-agent configuration. It is worth mentioning here that if (like us) you are testing windows agents then you are going to need to test with master-agent
approach due to the lack of a windows master.

Testing the master-agent configuration means configuring multi-node sets in beaker. There are not many examples of this but the principle is very much
the same as the single-node nodeset. Here is an example:

[Read More]

Testing Puppet with Beaker pt.2 - The Windows story

In part one we discussed our first steps into the world of acceptance testing our Puppet manifests.
By using Beaker we able to test managing local users on our Linux boxes. This was a positive experience for us. It allowed us to get to grips with the basics of configuring
Beaker to run tests and configuring our node sets to run those tests against. In this post, we will be discussing how we went about getting Beaker working with Windows.

As many of your reading this will be aware, OpenTable currently has quite a large Windows infrastructure and we are using Puppet extensively to maintain that environment.
We are also moving forward with releasing as many of our modules open source onto the Puppet Forge as possible (11 out of 18 of which are Windows
exclusive). What this means is that there was no way that we could ignore trying to use Beaker to test our manifests against Windows. We knew that we would have to support
many different versions and editions of Windows out there in the community as well that the ones we have to support internally.

This was going to be a challenge (configuration management with Windows usually is) but we were up for it.

The Preliminaries


The first step was looking at Serverspec. Serverspec is a Ruby gem that provides extensions to RSpec that allow you to test the actual state of your servers, either locally
or from the outside in via SSH. What we needed to know was did it support Windows? The answer was thankfully a resounding “Yes!”. All the resource types that you might want to test including file, service and user are available and supported on Windows. There are also a couple of Windows specific ones such as iis_website, Windows_feature and Windows_registry_key. We even added our own to support Windows_scheduled_task. Interestingly Serverspec also supports WinRM as an alternative to SSH when you are testing from the outside-in but we will go back into that later. As long as your using Serverspec > 1.6 you will have all the Windows support you might need.

[Read More]

What can I do?

I’ve noticed recently a lot of content in my social media network is based on the current escalating problems in the Middle East.

Whenever I see such content from a friend or follower, I’m reminded of Stephen Covey’s The 7 Habits of Highly Effective People - Circle of Influence, Circle of Concern.

Covey says that in order to remain truly effective, we should focus our time and energy on situations we can influence. Don’t worry about things we cannot control, such as the weather, the economy or indeed foreign conflicts.

Covey’s work implies that if we identify the areas that are inside our Circle of Influence we will be more productive and make more of an impact;


[Read More]