It's a second time I stumbled upon this issue. It's funny that the first time was so long ago that I forgot about it completely and had to troubleshoot the issue from scratch again. This is a good reason to finally write it down.
After you get a new version of raspbian image copied to your SD card and boot up the RPi, you'll be greeted by the raspi-config utility. It'll offer you to do a lot of things and, naturally, you'll follow the menu options from top to bottom. So, among the first things you'll do will likely change the password. Then, a few line down, if you're not in UK, you'll change the keyboard layout from the UK (default) to whatever (US in my case.) For the most cases it'll go smooth; however, there are occasions when it won't. What if your password contains a symbol character, say # (Shift-3 in US layout)? While still in the UK keyboard layout, this would be a British pound (currency symbol.) After you change your layout to say US, typing Shift-3 won't produce British pound sign anymore, rather #. So, the password is incorrect and you unreasonably keep calling yourself stupid for forgetting your password (as quite a few people did on RPi forums.)
So, now your only option is to power off RPi, pull out the SD card from RPi, insert it into another computer, open /boot/cmdline.txt, and append init=/bin/sh. Then unmount SD, insert it into RPi and power on. Upon boot up, the screen will look like RPi froze in the middle. Don't worry, it loaded the shell but for some reason did not show up the shell prompt. Just press Enter, and the shell prompt will show up. Now, type
passwd pi
and Raspbian will ask you to change the password. Since you have your desired keyboard layout now, you can go ahead and enter any non-alphanumeric characters you want for your password. Unless you decide to change your keyboard layout again, you should be safe. Finally, you need to undo the change you made to /boot/cmdline.txt so that your RPi boots normally.
P.S.: what's the raspi-config design bug you ask? Well, IMHO, the sequence of steps should first offer you to change the keyboard layout, and then the password, not the other way around.
