Have you upgraded any of your Debian systems to Jessie yet?
No? Then read on so that you can avoid some of the possible pitfalls that come with this upgrade.
The Story of my mistakes during the upgrade of my Proxmox guests to Debian 8.0 Jessie and what you can learn from them
Debian Jessie has been released a couple of days ago and naturally I wanted to check it out. Especially since some of my machines could really benefit from some of the newer software for Jessie. So I upgraded a few test machines and the first upgrade, a KVM Machine went pretty much as expected. There were only a few minor issues with configurations that are no longer supported.
After that I upgraded one of my OpenVZ test machines. The upgrade itself went as expected again, but after a reboot most services on the machine (including the network) would not start anymore.
After a bit of searching in the log files (which did not actually log anything after the update), I started looking to the most obvious cause: The change to systemd.
I was under the impression that the Proxmox OpenVZ Kernel supported systemd in OpenVZ Containers but it made the most sense for the symptoms. I spun up a new container, upgraded it to Jessie and installed sysvinit-core immediately after the update.
And everything worked as expected. Now I just had to deal with the Machine I upgraded earlier. I wanted to know if I could still get it to a working state. After a bit of messing around with it I managed to get it upgraded as well, but restoring a Backup and starting the upgrade again would have been quite a bit faster.
Now that everything was working I checked my Proxmox again since systemd should have been supported for my containers. It turns out, that while I did have the current Kernel 2.6.32-37 installed, I didn’t reboot the hypervisor in quite a while. Thus the Kernel that I had running was still 2.6.32-33. I could have saved myself a couple hours of troubleshooting if I had just restarted my Hypervisor before running the upgrade to Jessie on my OpenVZ container.
So if you plan on Upgrading your Debian Containers to Jessie, do yourself a favor and ssh to your Proxmox host and check what kernel version it is running:
uname -r #the Output should be a current kernel like this one 2.6.32-37-pve
If the output of this commands returns an older kernel than this, you should consider upgrading Proxmox to its current Version and then restarting. If that is not an option for you, then you should scroll down to my guide to Upgrade OpenVZ and install sysvinit-core.
If you already bricked a Machine you have no backup for have a look at the end of this article.
Follow these 6 steps to Upgrade your Debian Wheezy to Jessie
- Update your current wheezy install to the current state:
apt-get update apt-get upgrade
- Check if there are any issues with your current install, that you need to take care of:
dpkg --audit
You want the output of this command to be empty, if not fix the issues listed there.
- Change your sources list towards the Jessie repositories:
nano /etc/apt/sources.list
The file might look like this:
deb http://ftp.us.debian.org/debian/ wheezy main contrib non-free deb http://security.debian.org/ wheezy/updates main contrib non-free
You should replace Wheezy with Jessie in the lines of the official Debian repositories:
deb http://ftp.us.debian.org/debian/ jessie main contrib non-free deb http://security.debian.org/ jessie/updates main contrib non-free
If you have third party repositories, check if they are compatible with Jessie and what their path to the Jessie repository is.
- Now it is time to upgrade to Jessie:
apt-get update apt-get upgrade apt-get dist-upgrade
The update will ask if you want to disallow the use of the password for root logins over ssh. If you don’t use root for ssh or if you already authenticate with your private ssh key, then I suggest you choose yes. If not choose no for now and look into generating your own ssh key.
If you are Updating an OpenVZ machine with an old kernel, continue here to exchange systemd for sysvinit - After the upgrade make sure that all services are still working as intended and verify that everything installed correctly. Now reboot to finish the upgrade.
dpkg --audit reboot
- Once the reboot is done, make sure that all services started properly again. I had a few issues with mount cifs for example.
I will address any software issues that I find in the next part of this article.
Quick fixes for software issues I encountered after the upgrade
Mount cifs does not mount my smb shares anymore
This issue was a simple compatibility issue with my old configuration in /etc/fstab
It took me a while to figure out though, since there were 2 parts to the issue and many of the fixes suggested on the net did not work.
This is a line out of my old configuration:
\\smbserver.example.com\share3 /mnt/share3 cifs password=pwd,uid=1006,gid=119,username=shareuser 0 0
This is the line for the same share now:
//smbserver/share3 /mnt/share3 cifs password=pwd,uid=1006,gid=119,username=shareuser,sec=ntlm 0 0
What changed?
- the new version of mount cifs does not consider ntlm the default authentication algorithm anymore. My Solaris file server did not like that.
- I can’t specify the path to the share as unc path anymore.
Proftpd not starting anymore
In proftpd I was using a module that does not seem to be in Jessie anymore. The module is mod_vroot_c
Just remove it from the proftpd configuration at:
nano /etc/proftpd/modules.conf
And proftpd will start again.
Did you encounter any issues while upgrading to Jessie? If so I would like to hear about it in the comments.
Squid3 not starting anymore
After the Upgrade of my proxy box Squid 3 refused to start anymore. I had two issues with my old configuration file:
- the log_access directive was obsoleted and I still had one line with it in my configuration. I simply removed it since this particular statement was not doing much. If you need to replace it, use: access_log
- The name names of the authentication helpers in “/usr/lib/squid3/” have changed. The ones that I was using, that caused issue were:
- kerb_auth this changed to: negotiate_kerberos_auth
- wbinfo_group.pl this changed to: ext_wbinfo_group_acl
Spam assassin fails to compile
After the Upgrade spamassassin failed to compile with following error:
Running sa-compile (may take a long time) May 16 01:51:19.956 [10909] info: config: failed to parse line, skipping, in "/etc/spamassassin/local.cf": use_dcc 1 sa-compile: not compiling; 'spamassassin --lint' check failed! dpkg: error processing package sa-compile (--configure): subprocess installed post-installation script returned error exit status 2 Errors were encountered while processing: sa-compile E: Sub-process /usr/bin/dpkg returned an error code (1)
I simply commented out the use dcc line in the spamassassin config and told apt-get to fix the issue:
nano /etc/spamassassin/local.cf #in the file look for this line and delete it or comment it out: use dcc 1 apt-get -f install
You don’t like the newfangeled stuff? Get rid of it!
If you can’t Upgrade the Kernel of your Proxmox host for some reason, you can follow these steps to install sysvinit-core instead of systemd.
This also works for KVM hosts, if you happen to dislike systemd.
- If you completed step 4, your Jessie OpenVZ guest will be running just fine. But if you reboot it, it will not come up cleanly due to issues with systemd and old OpenVZ kernels. One of the things that will not start properly is the networking. Which is why you should change to sysvinit before rebooting. This is how you exchange systemd for sysvinit:
apt-get install sysvinit-core reboot
- Next remove the remnant of systemd:
apt-get remove systemd libsystemd0 Enter: Yes, do as I say! apt-get autoremove reboot
- After the reboot continue with step 5 of the normal update process.
So you bricked your OpenVZ guest
If you followed the guide, you won’t need this section. If you did reboot before installing sysvinit you can still get that machine back to a working state. But you will need access to the Proxmox host and its file system. It it is usually faster to restore from a backup. If you can’t do that for some reason, follow these steps:
- Connect to your Proxmox host and enter the OpenVZ machine.
vzctl enter machineid
- Try removing systemd-sysv. This won’t work, but it will give you the download links for the packages you are missing:
apt-get remove systemd-sysv Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: cgmanager libcgmanager0 libnih-dbus1 libnih1 systemd-shim sysvinit-core Suggested packages: pm-utils The following packages will be REMOVED: systemd-sysv The following NEW packages will be installed: cgmanager libcgmanager0 libnih-dbus1 libnih1 systemd-shim sysvinit-core 0 upgraded, 6 newly installed, 1 to remove and 1 not upgraded. 1 not fully installed or removed. Need to get 493 kB of archives. After this operation, 1074 kB of additional disk space will be used. Do you want to continue? [Y/n] Y Err http://ftp.debian.org/debian/ jessie/main libcgmanager0 i386 0.33-2+deb8u2 Could not resolve 'ftp.debian.org' Err http://ftp.debian.org/debian/ jessie/main libnih1 i386 1.0.3-4.3 Could not resolve 'ftp.debian.org' Err http://ftp.debian.org/debian/ jessie/main libnih-dbus1 i386 1.0.3-4.3 Could not resolve 'ftp.debian.org' Err http://ftp.debian.org/debian/ jessie/main cgmanager i386 0.33-2+deb8u2 Could not resolve 'ftp.debian.org' Err http://ftp.debian.org/debian/ jessie/main systemd-shim i386 9-1 Could not resolve 'ftp.debian.org' Err http://ftp.debian.org/debian/ jessie/main sysvinit-core i386 2.88dsf-59 Could not resolve 'ftp.debian.org' E: Failed to fetch http://ftp.debian.org/debian/pool/main/c/cgmanager/libcgmanager0_0.33-2+deb8u2_i386.deb Could not resolve 'ftp.debian.org' E: Failed to fetch http://ftp.debian.org/debian/pool/main/libn/libnih/libnih1_1.0.3-4.3_i386.deb Could not resolve 'ftp.debian.org' E: Failed to fetch http://ftp.debian.org/debian/pool/main/libn/libnih/libnih-dbus1_1.0.3-4.3_i386.deb Could not resolve 'ftp.debian.org' E: Failed to fetch http://ftp.debian.org/debian/pool/main/c/cgmanager/cgmanager_0.33-2+deb8u2_i386.deb Could not resolve 'ftp.debian.org' E: Failed to fetch http://ftp.debian.org/debian/pool/main/s/systemd-shim/systemd-shim_9-1_i386.deb Could not resolve 'ftp.debian.org' E: Failed to fetch http://ftp.debian.org/debian/pool/main/s/sysvinit/sysvinit-core_2.88dsf-59_i386.deb Could not resolve 'ftp.debian.org'
Download every single one of the packages that is listed. In the output of apt-get
- Transfer the packages to your OpenVZ container. The exact way to do this depends on the storage backend you are using.
If you are using the default Proxmox Virtual Machine directory, you can sftp to your Proxmox host and drop the files to this path:
/var/lib/vz/private/machineid/tmp
This will put the files in the /tmp folder of the container.
- Change into the container again and install the packages by hand:
dpkg -i /tmp/libcgmanager0_0.33-2+deb8u2_i386.deb dpkg -i /tmp/libnih1_1.0.3-4.3_i386.deb dpkg -i /tmp/libnih-dbus1_1.0.3-4.3_i386.deb dpkg -i /tmp/cgmanager_0.33-2+deb8u2_i386.deb dpkg -i /tmp/systemd-shim_9-1_i386.deb dpkg -i --ignore-depends=systemd-sysv /tmp/sysvinit-core_2.88dsf-59_i386.deb
The cgmanager and systemd-shim packages will fail configuration. That is ok for now.
- Restart the machine. The reboot command won’t work, so do this instead:
exit vzctl stop machineid vzctl start machineid vzctl enter machineid
- Fix the packages that failed configuration earlier:
dpkg --configure --pending reboot
- Continue with Step 5 of the upgrade guide
Thank you! I was worried about this upgrade, because of systemd, but your guide was helpful.
Thanks for this wonderful description, very well written!
I’m running now Debian 8.0 with sysv-init in an OpenVZ container. Unfortunately dmesg shows nothing. Dmesg under Debian Wheezy works. I did some research about dmesg, /proc/kmsg, “key ring buffer”… no success.
If someone has an idea, let me know.
I never really needed to get output from dmesg in one of my OpenVZ machines so I do not know if this was possible for me before.
But I do know that you can configure the Host in a way that prevents OpenVZ clients from Outputting any information using dmesg,
I quickly spun up a Wheezy VM to verify this, but for those I also have a blank output for the dmesg command.
So maybe the Proxmox Kernel just changed the behavior in this regard an it only kicked in now because you did not reboot the host in a while?
Thanks for your help.
I moved the guest system to another proxmox system and dmesg works. Now I’m not sure if it’s a kernel issue or system reboot helps (cannot reboot proxmox host system). Anyway, here are the kernels:
– proxmox host 1: 2.6.32-26-pve -> dmesg is empty
– proxmox host 2: 2.6.32-37-pve -> dmesg works!
No Problem, this blog is to help out fellow nerds and sysadmins.
Thank for letting me know.
Hi.
Im new to Debian Jessie. I have installed the container, and I logged in as root. But what ever I do, I am unable to update using apt-get update and when i see nano etc/sources.list I see nothing written there, I think the container is not installed properly ? or what is going on inside the container ? I would be really glad if could help me understand
What Version of Proxmox are you using, and which template did you use to create the container?