Simon Palmer’s blog

June 10, 2013

Deploying a Play 2 App to Amazon EC2 connected to RDS

Filed under: code, Uncategorized — Tags: , — simonpalmer @ 3:04 am

Some more fun and games with deployment of Play to EC2.  Just to set the scene, I have a Play 2 app which was built locally using Java 1.7.  I connect it to an Amazon RDS MySQL instance.  All this works just fine when running from localhost on my machine.

In theory, and according to the Play Documentation, I should be able to

  1. create a standalone Play distributable using play dist
  2. copy that up to my EC2 instance
  3. explode it
  4. run start

So that’s what I tried and it didn’t work.  For a start the standard Linux AMI at Amazon ships with Java 1.6 as the default, so I had to upgrade that (see here for how to do that).

Then I had to install a set of tools to give me access to my EC2 instance.  I like WinSCP which gives me a file browser and removes all the sweat of uploading and editing files on the Linux server, and does it through SFTP so I’m safe.  You can get WinSCP here.

I also needed PuTTY, or more accurately PuTTYgen to translate from the .pem file that EC2 creates for you when you make your security group in which your instances run, to a .ppk file which PuTTY and WinSCP need.  You can get the download from here but I confess I found that page very confusing.  Basically you want the one called “A Windows installer for everything except PuTTYtel“, and get the first one on the list since there seem to be two things with the same label.

Having done that using PuTTYgen is straightforward and you have a .ppk that you can use in WinSCP.  This blog has a nice write-up about that bit, in fact it’s worth a read in general.

Lots of steps to get to the basic point of being able to connect a file browser to my EC2 instance.  One thing to remember is that the DNS entry for your instance will change every time you restart it, so your saved connection in WinSCP will go stale as soon as you restart in EC2.  The way round that is to assign an Elastic IP address, but beware, that costs money.

Having connected I can very simply upload my Play distributable to EC2.  That distributable is just a zip file, so you need to unzip it when it gets there.  It doesn’t really matter where you put it on the server, but it’s simplest to put it in your ec2-user folder as you will have all the permissions you need to do that.

At this point you need a shell on your EC2 instance so you can run commands, including the start script that comes with your dist.  I tend to use the default shell that EC2 offers, but having done all that downloading you could (perhaps should) use PuTTY to do the same thing.  If you are like me (a bit lazy), then click on your instance in the EC2 console and choose “Connect” from the “Actions” menu at the top and then select “Use Java SSH client” and you’ll get a shell.

Now navigate to the folder in which you dropped your zipped dist file and unzip it.  That couldn’t be easier, just type “unzip” and it’ll explode it to the same folder you are currently in.

Now, in theory you should just be able to type ./start and your app should be running.

That didn’t work for me.  I spent many hours while the script was silently timing out in the background googling aimlessly trying to figure out what might be wrong.  It’s worth noting that a /logs folder is created with an application.log in it, which does give some clue.  In my case I was getting the error

2013-06-10 01:43:13,855 - [ERROR] - from com.jolbox.bonecp.hooks.AbstractConnectionHook in main 
Failed to obtain initial connection Sleeping for 0ms and trying again. Attempts left: 0. Exception: Connection timed out

This means “I can’t connect to your database, so I can’t get started”.  I was getting an exception thrown in the SSH client window after a very long timeout, but it actually took me quite a while to spot it.  That had a slightly less cryptic message and told me that I could not connect to my database [default].

That led me down a garden path of believing that it was not trying to connect to the right database and that the application.conf file packed into the dist was wrong or missing.  A few things on stackoverflow (this one and this one and this one and this one) helped, but they basically led me down the wrong path, although the experimentation was a useful learning curve.

So, the real problem was simply that I needed to make sure that my EC2 instance was allowed to talk to my RDS instance.  Turns out you can do this all through AWS.  I had a default security group on my RDS instance and a default security group on EC2, all I had to do was add the EC2 group to the RDS group, which you do through the RDS console by selecting Security Groups and adding an EC2 Security Group, and choosing the one you are using for your EC2 instance.

I re-ran my start script and it happily connected to the database and ran my app.


I still wasn’t done.  When I hit the page from the outside world (having assigned an elastic IP) I got a server connection error from the browser.  The last step is to make sure that the appropriate port is open to the wide world from your EC2 instance.  In the case of a Play app the default port is 9000, although you can change that, but I hadn’t, so I had to add it to the list of open ports in the EC2 security group.  You do that from the EC2 console, select Security Groups, choose the one you are using, select the “Inbound” tab and add a custom TCP rule typing 9000 as the port.

I tried to open this up on standard ports (80 and 8080) and then running the app using “./start -Dhttp.port=8080” but I couldn’t get that to work.  I suspect there are some internals of my Play app which would need to change to reflect the port change, and I was so happy to just see it all working up in the cloud I stopped there.

June 9, 2013

Upgrading to Java 7 on an Amazon Linux AMI

Filed under: code, Uncategorized — Tags: , — simonpalmer @ 2:37 am

OK, so I’m going to deploy a Play 2 app, backed by MySQL, into AWS.  I am using RDS for the MySQL instance, so I’m not expecting any trouble there, but my code does use Java 1.7, as does my Play 2 framework.

If you use the standard AWS AMI for 64 bit Linux (amzn-ami-pv-2013.03.1.x86_64-ebs (ami-05355a6c)) you will get Java 1.6 by default.  To be fair to Amazon, the AMI does include the Java 1.7 openjdk, but it doesn’t use it by default, you have to upgrade, and here’s how you do that after opening an SSH connection to your running instance:

sudo yum install java-1.7.0-openjdk 

All good so far, but then when I typed java -version it was still using 1.6.  So this is the little nugget I am recording in the hope that it might save you an hour of searching.  

To actually switch to using Java 1.7 you need to use the following:

sudo alternatives --config java

This will show both and you just choose 1.7.  Easy when you know how, impossible when you don’t.


April 19, 2012

What’s agile really for?

Filed under: recruiting — simonpalmer @ 5:49 pm

I’ve been asked by quite a number of people, in a variety of forums, what the real end goal of agile is. The context is normally prior to embarking on a transition to developing software along agile lines, and when trying to get a handle on the organisational cost/benefit trade-off.

My stock answer is that it is about quality and building the right product. It’s definitely not about productivity. It’s also about visibility into product development and putting trade-offs around prioritisation of features and fixes in the right hands in the business. It’s definitely not about productivity.

I have a longer answer where I describe some of the details of XP and what the real benefits are that we have accrued since switching.  I can also speak at length about the organisational cost, in terms of lost resources, transition time and the inevitable path along the satir change curve for the people involved.

The question of productivity always comes up, especially in the context of pair programming (surely it’s half as productive to have two people do a job as it is for one person to do it?).  Setting aside the full answer to that, which is about getting the work right, avoiding knowledge silos etc. I did secretly wonder whether I could find any measurable evidence to back up my intuitive feel that by adopting agile fully we *would* see productivity gains.

So in spite of it not being a primary rationale for the switch, my belief – and experience – was that improved productivity is a beneficial secondary corollary to agile.   I had a perfect chance in an upcoming project where I was implementing XP over the top of an existing team and asking them to go through the transition.  As we went we would have our velocity to use as a measure of our productivity, so we started to track it.

Here’s what happened…

First thing to note is our headcount dropped.  And quite dramatically.  We lost people who didn’t like the new way of working, the new working environment, and their new role in their team.  I expected some of this, but it was surprised at how many.  There were other problems in the team which I think were just as much to blame, and eventually someone’s decision about their employment is a complex and personal issue, but nonetheless, we definitely descended into a little chaos as people left when agile arrived.  The more surprising result was that, in spite of the seeming turmoil, the depletion of resources, the learning of new working practices etc. we saw the team velocity steadily increase.

It’s fair to say that velocity is a function of your ability to plan, and when you start planning in agile you learn about the meaning and value of a story point and it takes a little while to settle into a repeatable pattern.  However, the errors in planning estimates that give rise to velocity tend to be normally distributed, so it settles, and quicker than you might expect.

The end point of this transition is that there are fewer people on the team but there’s more work going on.  Obviously during transition we saw it flip around a little, but it has definitely settled into a place where the team is doing more with fewer resources. If I ignore velocity for a moment and just look at the real measure of activity, which is working product, then by that measure we are doing much better than we were with the previous team and practices.

So I’m going to change my stock answer to include the fact that, done right, agile can quite quickly result in productivity gains.

March 16, 2011

Why I will never score more than 9 on the Joel Test

Filed under: recruiting — simonpalmer @ 1:54 pm

I love the Joel Test.  Really, I do.  I wish it were stamped on the insides of the eyelids of any person with technical talent looking for a job.

BUT – and of course a BUT was coming – I will never encourage my teams to score more than 9.  And deliberately so.  For the record, and to save you having to switch between pages, here are Joel’s 12 questions:

  1. Do you use source control?
  2. Can you make a build in one step?
  3. Do you make daily builds?
  4. Do you have a bug database?
  5. Do you fix bugs before writing new code?
  6. Do you have an up-to-date schedule?
  7. Do you have a spec?
  8. Do programmers have quiet working conditions?
  9. Do you use the best tools money can buy?
  10. Do you have testers?
  11. Do new candidates write code during their interview?
  12. Do you do hallway usability testing?

Here are my answers:

1-5 Yes.  Of course.

6. DEPENDS.  If you mean an enormous GANTT chart written on wallpaper which demands a team of people whose sole purpose in life is to swim against the tide of inaccuracy, incompleteness and pointlessness, then the answer is a resounding NO.  If you will accept an iteration board, a rough release plan stretching out a couple of months, all of which is written on sticky notes and pinned to the wall and completely subject to change, then YES.  I think I score 0.5

7. I was going to say “depends” again, but my real answer is NO.  We do not have specs in any way that an ordinary person might interpret the word.  A “spec” (shot for specification) is a detailed document running to many pages which completely defines how a feature looks and behaves.  It is a contract between the person defining the requirement and the person delivering it.  We do not have these contracts.  For a start it implies that all the work of understanding and design will be complete prior to work starting.  It also presumes that we really know what the feature should be in advance and that the fungible developer will be able to interpret the words in the spec perfectly and implement both the letter and the spirit of the intent.  This never happens and with the benefit of 20-something years of software development it feels naive.  Not to mention it is in direct contradiction of the Agile Manifesto.

8. NO.  Programmers sit together round tables and are encouraged to talk, debate, call across to other people to help, gather in groups round whiteboards, converse, interact, stand up, sit down, cuddle, sing uproariously and generally be social humans engaged in a common task.  If they are silent and concentrating on work on their own, then they are not pairing and our Agile processes are breaking down and the code will suffer.  I say NO to silence.  On the rare occasions where someone has a solo task to perform (research spikes are an example) which will require dedicated concentration for a period of time I will encourage (and facilitate) them to go and find a space which suits that.

9. Yes

10. It’s another “depends”.  There are testers, and there are testers.  If you are doing TDD, then isn’t everyone a “tester”?  If you are aiming for 100% functional and integration coverage using an automated tool, then don’t you need Quality Engineers rather than testers?  Isn’t there every bit as much discipline in writing the tests as there is in writing the code?  We think there is and we religiously chant the TDD mantra, so in the same way that you won’t find “specs” in our world, your won’t find “testers” either.  Yes, we have people whose sole job is quality and we take it very seriously, but I don’t think they would call themselves testers.  I think I score a 0.5 here too.

11. Yes.  Of course.  In fact we have taken it one step further and give people coding challenges to complete between interviews and we review the code together collaboratively when they are done.  For more on this see my blog and RedCanary.

12. Yes.

So, I still love the Joel Test, but the worst, most prescribed waterfall places to work as a developer could score 12, and great creative software shops who do interesting valuable work in a collaborative and genuinely Agile way will rarely get full marks.

We’re the latter, not the former, so I stand proudly by my 9.

I love the LinkedIn Map…

Filed under: Uncategorized — simonpalmer @ 1:51 am

You can see mine here.

I’m not convinced it has any value whatever, but I do like it.

February 10, 2011

Recruiting. Challenges.

Filed under: code, recruiting — simonpalmer @ 2:45 pm

We’ve been very actively recruiting for some time and I previously posted about the framework we’re using to assess people.  At the bottom of that post I alluded to the “programming challenges” that we developed so we could get an idea of people’s technical chops.  It’s turned into much more and I wanted to write about it because I absolutely rely on it and it is the best recruiting I have ever done.  I should give an appropriate credit at this point, I say “we developed” but they are really the work of Andrew Datars who is my VP of Architecture, I can only claim to have planted the seed, Andrew did all the really hard work of bringing them into being.

Aside from the back-patting, the other reason I wanted to write about them was because their positioning is as important as their content.  For posterity, and because I think it’ll help with the explanation, here they are:
Empathica new developer challenge
Empathica NET developer challenge
Empathica NET quality engineer challenge

I should also state up-front that we use TDD in our development work and are an Agile XP shop.  These things are important to the story because we are hiring into two roles, Developer and Quality Engineer.  We also hold true to the belief that we recruit against innate skills rather than learnt ones, and therefore place a much higher value on a person’s capacity than we do on their precise experience – although experience and knowledge are obviously highly valuable once acquired.

Given our hiring philosophy we realised we needed a way to objectively assess a person from a technical perspective and within a technical context.  We also wanted to give some flavour of the work that we do to people who may have programming skills but completely outside our domain, or little programming experience at all – and it turns out that is a pretty decent size pool of people.  We are a .Net shop and therefore wrote one for that, repeated but technology agnostic, and created a third for quality engineers which is based on a publicly available code base from Microsoft.

The process goes like this…
1) have the person in for a face to face interview
2) send them away for 10 days or so with one of the challenges
3) bring them back in and review what they have done

There’s a lot we learn through this process which has little to do with the technology:

  • Do they accept the challenge and with what sort of attitude?
  • How long do they take to come back with an answer?
  • What does their code look like (style, separation of concerns, factoring etc.)?
  • How do they respond to criticism of their code?
  • How do they interact with us as developers?
  • How well did they understand the requirements?
  • How do they think through issues and debug the code

On top of which there is the code itself and the finished application.  We position the challenges not so much as a technical test but as a topic around which we can jointly work when they come in for a technical assessment.  The objective is only partly about assessing their technical skills, and almost not at all about their knowledge and experience.  Instead we want to simulate working with the person on a concrete problem in a technology we use and a context which is close to our reality.

We find this gives us an exceptionally good read on the person.  We allow enough time in the “interview” for them to overcome their nerves although that is an important consideration, especially if the person has only limited exposure to the technology.

We further request that they bring along a code base to which they have contributed significantly, ideally a hobby project to avoid NDA issues, and we spend the second half of the interview talking through and understanding their code.

This last part is important.  We found that we could be left wondering whether problems we saw in their approach or code in the challenge were to do with a fundamental lack of understanding of coding, or just unfamiliarity with the technology of the challenge we set.  Having them talk to us on their turf was a good way of finding that out.  It also gives other valuable insights such as how they are at expressing concepts to people with no domain knowledge, how motivated they are to code in their spare time, how curious they are about a problem, what sort of business sense they have, and the picture they have of where technical competence sits in the commercial world.

When it comes right down to it we end up not really caring too much about the technical aspects of the challenge, the human factors being much more relevant and harder to extract through a normal interview process.  We try and position it as being less about the technology and more about the opportunity to work together on some code, but as a candidate it is probably hard to see past it as technical test – which of course it is.

We have hired 3 people so far through this method and have a further 10 or so in our pipeline.  The results are stunning and we are in a hiring groove which is transforming our technical organisation.

If you are reading this and want to talk to me about a job please feel free to contact me by email at and make sure you mention this article and the challenges.

January 3, 2011

Mr Bob

Filed under: personal — simonpalmer @ 8:34 pm

According to ENISA, 95% of all email traffic is spam.  Up from 94% last year, but no real news there.  However I think new depths are being plumbed.  Here is an email I received this morning…

















We’ll get to why my spam filter didn’t find this at some later point, but I confess I’m intrigued by this.  I almost want to contact Mr Bob and find out why on earth he has sent this message – and obviously to correct his use of the apostrophe.

I wonder who would ever respond to an email like this.  I’m also aghast at how someone which such a poor grasp of English has such a seemingly firm grasp of spam.  If you were going to the lengths of sending out emails to strangers in the hope of them responding, wouldn’t you spend the additional 10 seconds thinking of something to say that might trigger a spark of interest?

The received wisdom is that people respond to these sorts of emails and click on the dubious links therein, otherwise the spammers would have stopped by now.  I wonder.  Maybe it is just so easy – and cheap to do it – that anyone can.  So anyone does.

Off to the IT guys to get the spam filter fixed.

December 22, 2010

A useful and objective hiring framework

Filed under: recruiting — simonpalmer @ 5:08 pm

I have the lovely, if difficult job, of hiring a bunch of new people into our organisation. It’s the thing I like most about my role (I’m CTO at Empathica) and the single most valuable legacy I can leave the business, so I take it very seriously. I also have a penchant for systems, not necessarily technical, but I like to have a framework in which I can place decisions.

Up to now I haven’t had one for choosing people and have had to fall back on my subjective, and deeply questionable, intuition. It’s impossible to get every hire right, but it is much easier to get them badly wrong, and a bad hire is not just bad for us as a business, it’s bad for the person too. I’ve been recruiting all my working life and in the final analysis I would like to think that the people who work here are all identical to me in only one respect, albeit a very important one; work should be fulfilling, rewarding and, as far as possible, pleasurable. I don’t think it is too lofty a goal for software developers to aim to be up towards the top end of Mazlow’s hierarchy.

I was triggered into some thoughts by an excellent and challenging recruitment partner with whom I have just recently started working (The Laudi Group, RedCanary). It’s rare to find someone who thinks similarly about recruiting and then holds your feet to the fire while you go through the process.  Laudi certainly do that and I appreciate it.  They are delivering us great candidates and we are hiring them.

Early on Mario, the eponymous Laudi, sent me a message with a link to a Fast Company article by Dee Hock which was an example of exactly the right information at the right moment. What I really liked in that was the cascade of values which had the learned skills at the bottom, not the top. It struck me that this was exactly how I felt about recruitment – give me raw innate qualities and I can supply the rest. If I had to trade those against knowledge I would always err on the side of the innate.

I started using this as a sounding board for myself and it has evolved into a tool for us in the recruitment team at Empathica. I have added two things to it which speak to our particular needs, namely curiosity and fit – apologies to Dee Hock who clearly has forgotten much more about this than I will ever know, and expresses himself more concisely and eloquently than I ever will. In any case, my cascade of characteristics is as follows:

  • Integrity
  • Motivation
  • Fit
  • Capacity
  • Curiosity
  • Understanding
  • Knowledge
  • Experience

Being a measurement company, and a bunch of slightly over-analytical nerds, we decided to rate people on a 5 point scale on these 8 things.  Once we had done it a few times we realised there was a bar of acceptability given our current recruitment needs and this provided us a very good way of formalising, and sometimes justifying, our recruiting decisions.  There is the intriguing possibility too of taking it one step further, namely to look at our current people.  I’m not quite ready to do that because it is a very sharp knife, but it is tempting.

So our process is to meet with someone once, generally a couple of people at our end, then rate them on our scale.  From the initial rating we get two things, first an obvious “No” if one exists, and second some direction on what we would want to do with them in a subsequent interview. We draw it on my whiteboard and collaborate over the scores, which are sometimes a bit gray, and always somewhat subjective – but guess what, all hiring is.

Here’s what it looks like:

5 point scale

Over time it was clear that we would not compromise on the innate elements of integrity, motivation, fit, curiosity and capacity and we would make quite significant compromises on the other three which we consider learned.  So the solid black line is where we set our bar.  The green line is a candidate we recently saw and although we liked the person we realised that they scored well on the learned end of the scale, and were only OK on the innate end.  We did not hire them.  The red line is a person who we lost because we weren’t quick enough with an offer – a tale for another time.

In addition to this, and to complement it, we have developed some coding challenges for technical hires that we have them complete and then we review with them live on their machines.  This drags out everything you’ll ever need to know about a coder and we only give it to people who have got above our notional bar and is the best technical interview tool I have ever used.  Kudos goes to Andrew Datars, my VP of Architecture, for dreaming up a great set of coding and QA tasks.

We’ll continue to use this framework and it has already added a great deal of value and allowed us to have exactly the right conversations between us, with the candidate and with our recruiter.

December 13, 2010

Stackoverflow flair!

Filed under: stackoverflow — simonpalmer @ 8:12 pm

Stack Overflow profile for Simon at Stack Overflow, Q&A for professional and enthusiast programmers

October 12, 2010

Toronto – WAKE UP!

Filed under: Uncategorized — simonpalmer @ 5:45 pm

Toronto is a city dominated and ruined in equal measure by cars.

Cars, and their insane drivers, prevent Toronto from being the world-class lakeside city it should by all rights be.  Instead it is a crucible of collisions with a cowering population stuck in the middle on an ugly traffic island.

Here are some thoughts for Torontonians:
1) The average stopping distance for a vehicle traveling 100kph is 80m.  You don’t need to leave 80m between vehicles traveling at 100kph, half that will do.  However, bear in mind that judgment and reaction time alone means you cannot stop within 40m from 100kph.  Nobody leaves 40m.  If you do, at least one car, and probably two, will pull into it.

2) If you travel at 120 while I am traveling at 110kph, and we both have roughly half an hour journey, you will arrive less than 3 minutes before me.  On the other hand the risk you introduce of an accident goes up as the square of the difference between our speed.  That means, if we both travel at 110, we both arrive at the same time and there is minimal risk.  Your extra 10kph buys you a couple of minutes – at the most – at the cost of dramatically increased risk.  In reality you’ll never recover those additional minutes, I’ll pass you again later as the traffic speeds vary during the journey, or we’ll both be held at the same light as we get off the highway 50m apart.  IT’S INSANE.

3) While you may accept the additional risk for 10kph, I do not.  Very few accidents at speed on a highway involve a single car.  You are making a choice for me.

4) More roads = more traffic, not less.  Countless studies prove the conclusions from the original British study about trunk roads (Google Trunk Roads SACTRA).  “An average road improvement, for which traffic growth due to all other factors is forecast correctly, will see an additional [i.e. induced] 10% of base traffic in the short term and 20% in the long term”.  The answer is public transport.  If Toronto is going to be a world class city and not drown under the weight of its own traffic as it is currently doing, you have to improve public transport.  Someone tell the new Mayor when he gets into office.

5) You will never have a usable lakefront until you get rid of the Gardiner Expressway.  It is an ugly monstrosity born out of an outdated and disproven philosophy which prevents Toronto from being a lakefront city.  Think Boston.  Think Big Dig.  Start now.

I want to shake Toronto by the shoulders.  WAKE UP!  Stop driving like morons.  Stop crashing into each other every day on your journey to work.  Stop killing thousands of people on the roads every year.  STOP DRIVING!  Press your public officials to build public transit.  Reclaim your wonderful, beautiful, majestic lakefront.  Be less Birmingham and more Sydney.  Less Shanghai, more Boston.

oh, and before I forget…

6) If you drive a Pontiac Grand Am or Grand Prix, please go to the nearest Police station, surrender your license to an officer and hold your hands out ready to be cuffed.  You are not in a sports car and you do not own the road.

Older Posts »

Blog at