Recovering Bash and login to proxmox LXC

R

I have an issue with not knowing when my brain gets tired. The only way to know is when I start making the silly mistakes. One such mistake is I lost the bash shell and got locked out of my main web host container. This post is on recovering bash and login back into the container.

Situation – Background

I am a sucker for fancy, but usable UI. Zsh has been my main shell on my development mac for quite long. Along with Oh-My-Zsh, I love my Iterm2. But since I spend most of the time in remote Linux containers over SSH, I was missing my fancy Zsh terminal interface. There started my problem. All went well when I installed Zsh. Went on to change my shell to zsh through the following commands.
 apt install zsh

followed by

 chsh -s $(which zsh) 

.

Then went on to install oh-my-zsh. Started getting errors on missing icons and the screen looked hell. 
Got crazy. and removed zsh, without switching to default shell. – Mistake number 1
Exited out of the ssh session, expecting Bash shell to take effect. – Mistake number 2
Now when I tried login back into my container. Boom, there comes the following error.
cant find /usr/bin/zsh 

and also cant login with the following error

Password: PAM authentication failed 
Locked the hell out. Result Panic!!!

Recovery – First backup

  1. First things first. Take a quick back up of my web application. I use updraftplus for the same.
  2. Created another container and restored the websites.
  3. Changed my Shorewall setting to reflect website pointing to the right container.
  4. verified the website is all working well

Recovery – Actual Bash recovery and login

Now its time for the actual bash recovery. My setup is proxmox-zfs, as I had mentioned in one of the earlier posts. So it was a bit of easy job to browse through the file system from the host.
  1. Login to the proxmox host system
  2. Browse to
     /rpool/data/subdiskvolXXXX/ 

    . Actual sub disk name can be found from the proxmox panel in the Resources of the guest.

  3. Create a temporary fix by Soft linking bash to the old location of zsh using the
     ln -s rpool /data/{diskname}/bin/bash rpool/data/{diskname}/usr/bin/zsh 
  4. Try login into the guest. Magic happens you have the shell!! – But this is not a fully functional shell. Most shell commands won’t work well, including apt-get.
  5. Try changing the default shell to bash using
     chsh -s /bin/bash 

    . If it works great. But most probably it will come back with the following message.

    PAM: Authentication failure 
  6. if PAM error occurs, we need to do a bit of trick by editing by
    /etc/pam.d/chsh 

    changing “required” to “sufficient”

    auth required pam_shells.so 
    auth sufficient pam_shells.so 

    – save and exit

  7. Try again the changing to default shell using the command in step 5. Things should work perfectly.
  8. Logout and login to verify you have the proper shell. Try which bash. You should get the proper path. Try running a couple of shell scripts and you would not have any issues.
  9. Go back the chsh, change “sufficient” to “required” like the step 6. Always better keep things tidy.

That’s all folks.

About the author

nagu

Business leader, Technology Consultant and a Polyglot programmer. Infectious passion for Innovations and new technologies. Respects good UX. Current interests are in Microservices, ethical hacking and lockpicking!.

Add comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

By nagu

Nagu Gopalakrishnan

Business leader, Technology Consultant and a Polyglot programmer. Infectious passion for Innovations and new technologies. Respects good UX. Current interests are in Microservices, ethical hacking and lockpicking!.

About Me