I'm on a Mac OSX machine using a version of libx264 from MacPorts. It turns out that the default install does not enable any CPU optimisations. So I just needed to rebuild it with asm variant to enable it's assembly code based optimisations:
sudo port -u install x264 +asmI reran ffmpeg and libx264 logged that it was using a whole string of it's CPU specific optimisations and running a whole lot faster!
If you're on Linux/Windows/Raspberry-Pi you're probably running the same problem. This is because the people who ship software need to compile it in a way that runs on most machines - which means it's not going be tailored to run best on your particular machine. So the helpful folks at ffmpeg have put together a compilation guide on how to do it yourself.