Recovering Bash and login to proxmox LXC

I have an issue with not knowing when my brain gets tired. The only way to know is when I start making the silly mistakes. One such mistake is I lost the bash shell and got locked out of my main web host container. This post is on recovering bash and login back into the container.

Situation – Background

I am a sucker for fancy, but usable UI. Zsh has been my main shell on my development mac for quite long. Along with Oh-My-Zsh, I love my Iterm2. But since I spend most of the time in remote Linux containers over SSH, I was missing my fancy Zsh terminal interface. There started my problem. All went well when I installed Zsh. Went on to change my shell to zsh through the following commands.
apt install zsh

followed by

chsh -s $(which zsh)

Then went on to install oh-my-zsh. Started getting errors on missing icons and the screen looked hell. Got crazy. and removed zsh, without switching to default shell. – Mistake number 1

Exited out of the ssh session, expecting Bash shell to take effect. – Mistake number 2

Now when I tried login back into my container. Boom, there comes the following error.

cant find /usr/bin/zsh

and also cant login with the following error

Password: PAM authentication failed

Locked the hell out. Result Panic!!!

Recovery – First backup

  1. First things first. Take a quick back up of my web application. I use updraftplus for the same.
  2. Created another container and restored the websites.
  3. Changed my Shorewall setting to reflect website pointing to the right container.
  4. verified the website is all working well

Recovery – Actual Bash recovery and login

Now its time for the actual bash recovery. My setup is proxmox-zfs, as I had mentioned in one of the earlier posts. So it was a bit of easy job to browse through the file system from the host.

  1. Login to the proxmox host system
  2. Browse to
/rpool/data/subdiskvolXXXX/

Actual sub disk name can be found from the proxmox panel in the Resources of the guest.

  1. Create a temporary fix by Soft linking bash to the old location of zsh using the
ln -s rpool /data/{diskname}/bin/bash rpool/data/{diskname}/usr/bin/zsh
  1. Try login into the guest. Magic happens you have the shell!! – But this is not a fully functional shell. Most shell commands won’t work well, including apt-get.
  2. Try changing the default shell to bash using 
chsh -s /bin/bash

If it works great. But most probably it will come back with the following message.

Now its time for the actual bash recovery. My setup is proxmox-zfs, as I had mentioned in one of the earlier posts. So it was a bit of easy job to browse through the file system from the host.
PAM: Authentication failure
  1. if PAM error occurs, we need to do a bit of trick by editing
/etc/pam.d/chsh

changing “required” to “sufficient”

auth required pam_shells.so
auth sufficient pam_shells.so

– save and exit

  1. Try again the changing to default shell using the command in step 5. Things should work perfectly.
  2. Logout and login to verify you have the proper shell. Try which bash. You should get the proper path. Try running a couple of shell scripts and you would not have any issues.
  3. Go back the chsh, change “sufficient” to “required” like the step 6. Always better keep things tidy.

That’s all folks.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.