Upgrading Debian Guests on Proxmox to Jessie

Have you upgraded any of your Debian systems to Jessie yet?
No? Then read on so that you can avoid some of the possible pitfalls that come with this upgrade.

The Story of my mistakes during the upgrade of my  Proxmox guests to Debian 8.0 Jessie and what you can learn from them

Debian Jessie has been released a couple of days ago and naturally I wanted to check it out. Especially since some of my machines could really benefit from some of the newer software for Jessie. So I upgraded a few test machines and the first upgrade, a KVM Machine went pretty much as expected. There were only a few minor issues with configurations that are no longer supported.
After that I upgraded one of my OpenVZ test machines. The upgrade itself went as expected again, but after a reboot most services on the machine (including the network) would not start anymore.
After a bit of searching in the log files (which did not actually log anything after the update), I started looking to the most obvious cause: The change to systemd.

I was under the impression that the Proxmox OpenVZ Kernel supported systemd in OpenVZ Containers but it made the most sense for the symptoms. I spun up a new container, upgraded it to Jessie and installed sysvinit-core immediately after the update.
And everything worked as expected. Now I just had to deal with the Machine I upgraded earlier. I wanted to know if I could still get it to a working state. After a bit of messing around with it I managed to get it upgraded as well, but restoring a Backup and starting the upgrade again would have been quite a bit faster.

Now that everything was working I checked my Proxmox again since systemd should have been supported for my containers. It turns out, that while I did have the current Kernel 2.6.32-37 installed, I didn’t reboot the hypervisor in quite a while. Thus the Kernel that I had running was still 2.6.32-33.  I could have saved myself a couple hours of troubleshooting if I had just restarted my Hypervisor before running the upgrade to Jessie on my OpenVZ container.

So if you plan on Upgrading your Debian Containers to Jessie, do yourself a favor and ssh to your Proxmox host and check what kernel version it is running:

uname -r
#the Output should be a current kernel like this one
2.6.32-37-pve

If the output of this commands returns an older kernel than this, you should consider upgrading Proxmox to its current Version and then restarting. If that is not an option for you, then you should scroll down to my guide to Upgrade OpenVZ and install sysvinit-core.
If you already bricked a Machine you have no backup for have a look at the end of this article.

Follow these 6 steps to Upgrade your Debian Wheezy to Jessie

  1. Update your current wheezy install to the current state:
    apt-get update
    apt-get upgrade
    
    
  2. Check if there are any issues with your current install, that you need to take care of:
    dpkg --audit
    

    You want the output of this command to be empty, if not fix the issues listed there.

  3. Change your sources list towards the Jessie repositories:
    nano /etc/apt/sources.list
    

    The file might look like this:

    deb http://ftp.us.debian.org/debian/ wheezy main contrib non-free
    deb http://security.debian.org/ wheezy/updates main contrib non-free
    

    You should replace Wheezy with Jessie in the lines of the official Debian repositories:

    deb http://ftp.us.debian.org/debian/ jessie main contrib non-free
    deb http://security.debian.org/ jessie/updates main contrib non-free
    

    If you have third party repositories, check if they are compatible with Jessie and what their path to the Jessie repository is.

  4. Now it is time to upgrade to Jessie:
    apt-get update
    apt-get upgrade
    apt-get dist-upgrade
    

    The update will ask if you want to disallow the use of the password for root logins over ssh. If you don’t use root for ssh or if you already authenticate with your private ssh key, then I suggest you choose yes. If not choose no for now and look into generating your own ssh key.
    If you are Updating an OpenVZ machine with an old kernel, continue here to exchange systemd for sysvinit

  5. After the upgrade make sure that all services are still working as intended and verify that everything installed correctly. Now reboot to finish the upgrade.
    dpkg --audit
    reboot
    
  6. Once the reboot is done, make sure that all services started properly again. I had a few issues with mount cifs for example.
    I will address any software issues that I find in the next part of this article.

Quick fixes for software issues I encountered after the upgrade

Mount cifs does not mount my smb shares anymore

This issue was a simple compatibility issue with my old configuration in /etc/fstab

It took me a while to figure out though, since there were 2 parts to the issue and many of the fixes suggested on the net did not work.

This is a line out of my old configuration:

\\smbserver.example.com\share3   /mnt/share3  cifs    password=pwd,uid=1006,gid=119,username=shareuser      0       0

This is the line for the same share now:

//smbserver/share3   /mnt/share3  cifs    password=pwd,uid=1006,gid=119,username=shareuser,sec=ntlm        0       0

What changed?

  • the new version of mount cifs does not consider ntlm the default authentication algorithm anymore. My Solaris file server did not like that.
  • I can’t specify the path to the share as unc path anymore.

Proftpd not starting anymore

In proftpd I was using a module that does not seem to be in Jessie anymore. The module is mod_vroot_c

Just remove it from the proftpd configuration at:

nano /etc/proftpd/modules.conf

And proftpd will start again.

Did you encounter any issues while upgrading to Jessie? If so I would like to hear about it in the comments.

Squid3 not starting anymore

After the Upgrade of my proxy box Squid 3 refused to start anymore. I had two issues with my old configuration file:

  • the log_access directive was obsoleted and I still had one line with it in my configuration. I simply removed it since this particular statement was not doing much. If you need to replace it, use: access_log
  • The name names of the authentication helpers in “/usr/lib/squid3/” have changed. The ones that I was using, that caused issue were:
    1. kerb_auth this changed to: negotiate_kerberos_auth
    2. wbinfo_group.pl this changed to: ext_wbinfo_group_acl

Spam assassin fails to compile

After the Upgrade spamassassin failed to compile with following error:

Running sa-compile (may take a long time)
May 16 01:51:19.956 [10909] info: config: failed to parse line, skipping, in "/etc/spamassassin/local.cf": use_dcc 1
sa-compile: not compiling; 'spamassassin --lint' check failed!
dpkg: error processing package sa-compile (--configure):
subprocess installed post-installation script returned error exit status 2
Errors were encountered while processing:
sa-compile
E: Sub-process /usr/bin/dpkg returned an error code (1)

I simply commented out the use dcc line in the spamassassin config and told apt-get to fix the issue:

nano /etc/spamassassin/local.cf
#in the file look for this line and delete it or comment it out:
use dcc 1
apt-get -f install

You don’t like the newfangeled stuff? Get rid of it!

If you can’t Upgrade the Kernel of your Proxmox host for some reason, you can follow these steps to install sysvinit-core instead of systemd.
This also works for KVM hosts, if you happen to dislike systemd.

  1. If you completed step 4, your Jessie OpenVZ guest will be running just fine. But if you reboot it, it will not come up cleanly due to issues with systemd and old OpenVZ kernels. One of the things that will not start properly is the networking. Which is why you should change to sysvinit before rebooting.  This is how you exchange systemd for sysvinit:
    apt-get install sysvinit-core
    reboot
    
  2. Next remove the remnant of systemd:
    apt-get remove systemd libsystemd0
    Enter: Yes, do as I say!
    apt-get autoremove
    reboot
    
  3. After the reboot continue with step 5 of the normal update process.

So you bricked your OpenVZ guest

If you followed the guide, you won’t need this section.  If you did reboot before installing sysvinit you can still get that machine back to a working state. But you will need access to the Proxmox host and its file system. It it is usually faster to restore from a backup. If you can’t do that for some reason, follow these steps:

  1. Connect to your Proxmox host and enter the OpenVZ machine.
    vzctl enter machineid
    
  2. Try removing systemd-sysv. This won’t work, but it will give you the download links for the packages you are missing:
    apt-get remove systemd-sysv
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following extra packages will be installed:
      cgmanager libcgmanager0 libnih-dbus1 libnih1 systemd-shim sysvinit-core
    Suggested packages:
      pm-utils
    The following packages will be REMOVED:
      systemd-sysv
    The following NEW packages will be installed:
      cgmanager libcgmanager0 libnih-dbus1 libnih1 systemd-shim sysvinit-core
    0 upgraded, 6 newly installed, 1 to remove and 1 not upgraded.
    1 not fully installed or removed.
    Need to get 493 kB of archives.
    After this operation, 1074 kB of additional disk space will be used.
    Do you want to continue? [Y/n] Y
    Err http://ftp.debian.org/debian/ jessie/main libcgmanager0 i386 0.33-2+deb8u2
      Could not resolve 'ftp.debian.org'
    Err http://ftp.debian.org/debian/ jessie/main libnih1 i386 1.0.3-4.3
      Could not resolve 'ftp.debian.org'
    Err http://ftp.debian.org/debian/ jessie/main libnih-dbus1 i386 1.0.3-4.3
      Could not resolve 'ftp.debian.org'
    Err http://ftp.debian.org/debian/ jessie/main cgmanager i386 0.33-2+deb8u2
      Could not resolve 'ftp.debian.org'
    Err http://ftp.debian.org/debian/ jessie/main systemd-shim i386 9-1
      Could not resolve 'ftp.debian.org'
    Err http://ftp.debian.org/debian/ jessie/main sysvinit-core i386 2.88dsf-59
      Could not resolve 'ftp.debian.org'
    E: Failed to fetch http://ftp.debian.org/debian/pool/main/c/cgmanager/libcgmanager0_0.33-2+deb8u2_i386.deb  Could not resolve 'ftp.debian.org'
    
    E: Failed to fetch http://ftp.debian.org/debian/pool/main/libn/libnih/libnih1_1.0.3-4.3_i386.deb  Could not resolve 'ftp.debian.org'
    
    E: Failed to fetch http://ftp.debian.org/debian/pool/main/libn/libnih/libnih-dbus1_1.0.3-4.3_i386.deb  Could not resolve 'ftp.debian.org'
    
    E: Failed to fetch http://ftp.debian.org/debian/pool/main/c/cgmanager/cgmanager_0.33-2+deb8u2_i386.deb  Could not resolve 'ftp.debian.org'
    
    E: Failed to fetch http://ftp.debian.org/debian/pool/main/s/systemd-shim/systemd-shim_9-1_i386.deb  Could not resolve 'ftp.debian.org'
    
    E: Failed to fetch http://ftp.debian.org/debian/pool/main/s/sysvinit/sysvinit-core_2.88dsf-59_i386.deb  Could not resolve 'ftp.debian.org'
    
    

    Download every single one of the packages that is listed. In the output of apt-get

  3. Transfer the packages to your OpenVZ container. The exact way to do this depends on the storage backend you are using.
    If you are using the default Proxmox Virtual Machine directory, you can sftp to your Proxmox host and drop the files to this path:
/var/lib/vz/private/machineid/tmp

This will put the files in the /tmp folder of the container.

  • Change into the container again and install the packages by hand:
    dpkg -i /tmp/libcgmanager0_0.33-2+deb8u2_i386.deb
    dpkg -i /tmp/libnih1_1.0.3-4.3_i386.deb
    dpkg -i /tmp/libnih-dbus1_1.0.3-4.3_i386.deb
    dpkg -i /tmp/cgmanager_0.33-2+deb8u2_i386.deb
    dpkg -i /tmp/systemd-shim_9-1_i386.deb
    dpkg -i --ignore-depends=systemd-sysv /tmp/sysvinit-core_2.88dsf-59_i386.deb
    

    The cgmanager and systemd-shim packages will fail configuration. That is ok for now.

  • Restart the machine. The reboot command won’t work, so do this instead:
     exit
     vzctl stop machineid
     vzctl start machineid
     vzctl enter machineid
    
  • Fix the packages that failed configuration earlier:
    dpkg --configure --pending
    reboot
    
  • Continue with Step 5 of the upgrade guide

 

8 thoughts on “Upgrading Debian Guests on Proxmox to Jessie”

  1. Thanks for this wonderful description, very well written!

    I’m running now Debian 8.0 with sysv-init in an OpenVZ container. Unfortunately dmesg shows nothing. Dmesg under Debian Wheezy works. I did some research about dmesg, /proc/kmsg, “key ring buffer”… no success.

    If someone has an idea, let me know.

    1. I never really needed to get output from dmesg in one of my OpenVZ machines so I do not know if this was possible for me before.
      But I do know that you can configure the Host in a way that prevents OpenVZ clients from Outputting any information using dmesg,
      I quickly spun up a Wheezy VM to verify this, but for those I also have a blank output for the dmesg command.
      So maybe the Proxmox Kernel just changed the behavior in this regard an it only kicked in now because you did not reboot the host in a while?

      1. Thanks for your help.

        I moved the guest system to another proxmox system and dmesg works. Now I’m not sure if it’s a kernel issue or system reboot helps (cannot reboot proxmox host system). Anyway, here are the kernels:

        – proxmox host 1: 2.6.32-26-pve -> dmesg is empty
        – proxmox host 2: 2.6.32-37-pve -> dmesg works!

  2. Hi.

    Im new to Debian Jessie. I have installed the container, and I logged in as root. But what ever I do, I am unable to update using apt-get update and when i see nano etc/sources.list I see nothing written there, I think the container is not installed properly ? or what is going on inside the container ? I would be really glad if could help me understand

Leave a Reply

Your email address will not be published. Required fields are marked *