To fix the problem you need to install an up to date version of libdvdcss - either by downloading a slightly old version and copying it into the /usr/loca/lib e.g. if the downloaded library file is in Downloads:
sudo cp ~/Downloads/libdvdcss.2.dylib /usr/local/libOr you can build/install the latest version using Macports, homebrew, or from source. If you use Macports then you may need to link the macports installed version to /usr/local/lib (so that Handbrake can find it):
sudo ln -s /opt/local/lib/libdvdcss.2.dylib /usr/local/libNow start Handbrake and you should be good to go...