Wiki Navigation

How To Fix Debian Systems with broken OpenSSL

Debian 4.0 (etch) systems have been found to contain a significant vulnerability in their packaging of the OpenSSL libraries, as described on the [Debian Security site]. These notes document the steps to patch and update the security credentials on the affected systems.

Applying Patches

The standard Debian update steps are

apt-get update
apt-get upgrade

These steps install more than the 5 affected packages documented in the security bulletin on our systems since some previous updates had not yet been applied.

The upgrade scripts prompt for sshd to be restarted, which is OK. Additional services that are upgraded will be restarted as needed. We'll be rebooting the systems after all updates are in place to ensure pristine run state.

Replacing the Keys

All keys generated on the system using broken openssl library need to be replaced.

Updating Host SSH Keys

The host SSH keys were generated during the system install, are stored in /etc/ssh and need replacement.

cd /etc/ssh
for file in ssh_host_*
do
  mv $file $file.weak
done
ssh-keygen -f /etc/ssh/ssh_host_rsa_key -t rsa -N ''
ssh-keygen -f /etc/ssh/ssh_host_dsa_key -t dsa -N ''
/etc/init.d/ssh restart

Restaring sshd won't effect existing connections, so you will need to log out and reconnect to ensure the properly seeded keys are used during the secure connection setup. The fingerprints for the new keys will be reported during the key regeneration, or use the following commands

ssh-keygen -l -f ./ssh_host_rsa_key
ssh-keygen -l -f ./ssh_host_dsa_key

Report the new finger prints in the section below for each of the hosts updated. By default, the RSA key should be 2048 bits and the DSA key 1024.

Regenerating Authorized Keys

The initial exploits are focusing on key based login via SSH where weak keys generated on the affected systems are used to log into remote systems. Currently, the time to exploit this is about 20 minutes.

A system can be searched for accounts which have this password-less SSH login configured with the following command:

find / -name id_[rd]sa.pub -o -name .ssh -o -name authorized_keys

This will help find accounts that need closer inspection.

For the Debian VM systems on UABgrid there are no files found matching this search on these systems. We have not been using this form of login to access these systems. These systems also have a very small set (<5 at most) of user accounts defined and there is no knowledge of them having ever been used to create keys for password-less login to other systems. The login scenario is normally from non-Debian based systems to these systems, but again, no authorized_keys files exist.

A broader problem exists though is that normal users systems have had this type of login configured and that some of the origin systems may be from affected Debian systems. The above search should help identify those accounts for further inspection.

Key Fingerprints

ca.uabgrid.uab.edu SSH host keys for:

RSA: a3:66:77:87:fa:bb:e8:de:89:dc:75:08:d9:18:f6:34
DSA: 53:36:8d:4a:0b:80:22:c5:f7:b2:1d:3d:d1:88:4a:e1

apps.uabgrid.uab.edu SSH host keys for:

RSA: 21:56:20:e5:d7:39:b8:3c:b7:5c:8c:80:78:4a:6b:d0
DSA: 42:03:1f:c3:d1:ca:63:c6:e6:a8:a4:31:51:03:42:ef

Note: this host is also known by blogs.uabgrid and docs.uabgrid, so SSH to those hosts will use the same keys.

projects.uabgrid.uab.edu SSH host keys for:

RSA: cd:85:ff:11:93:7d:de:9d:d6:95:4d:88:a4:60:57:c5
DSA: 5a:7c:66:47:8e:3d:64:3e:28:75:f8:53:e2:d6:29:ad

Note: this host is also known by dev.uabgrid so this host will use the same keys.

System Notes

ca.uabgrid

The patches were installed for this system but the following packages were held back:

# apt-get upgrade
The following packages have been kept back:
  linux-image-2.6-686 openssh-client openssh-server
The following packages will be upgraded:
  apache2-doc apache2-mpm-prefork apache2-utils apache2.2-common bsdutils cpio
  debconf debconf-i18n e2fslibs e2fsprogs findutils klibc-utils
  libapache2-mod-php5 libblkid1 libc6 libc6-i686 libcomerr2 libfreetype6
  libicu36 libklibc libkrb53 libmagic1 libmysqlclient15off libpam-modules
  libpam-runtime libpam0g libpcre3 libperl5.8 libpq4 libss2 libssl0.9.8
  libuuid1 libxml2 locales mount mysql-common perl perl-base perl-modules
  php5-common python2.4 python2.4-minimal tar tzdata util-linux vim-common
  vim-tiny
47 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

Note the openssl package is not in the list because it was independently upgraded with apt-get install openssl before following the bulk update above.

apps.uabgrid

The patch summary:

# apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
The following packages have been kept back:
  openssh-client openssh-server
The following packages will be upgraded:
  apache2-doc apache2-mpm-prefork apache2-utils apache2.2-common bsdutils cpio
  debconf debconf-i18n debian-archive-keyring e2fslibs e2fsprogs file
  findutils gs-gpl initramfs-tools klibc-utils libapache2-mod-php5 libblkid1
  libc6 libc6-i686 libcomerr2 libcurl3 libdbi-perl libfreetype6 libicu36
  libklibc libkrb53 libmagic1 libmysqlclient15off libpam-modules
  libpam-runtime libpam0g libpcre3 libperl5.8 libpoppler0c2 libpq4 libss2
  libssl0.9.8 libt1-5 libuuid1 libxml2 linux-image-2.6.18-4-686 locales
  lsb-base lvm2 mount mysql-client-5.0 mysql-common mysql-server
  mysql-server-5.0 nano openssl perl perl-base perl-modules php5 php5-cli
  php5-common php5-mysql python2.4 python2.4-minimal tar tetex-base tzdata
  unzip util-linux vim-common vim-tiny wordpress x11-common
70 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

This system did apply the kernel patch. (It was not held back.) There was a warning that the same kernel version as the running system was being installed and that a reboot is necessary to fix the modules.dep file. Don't know if this is a standard warning or if the kernel really didn't need updating.

This system includes vendor supplied MediaWiki for docs.uabgrid and Wordpres for blogs.uabgrid. There was only a patch for wordpress. !Mediawiki has been customized locally for external authn (shib), so we need to keep an eye on any updates. Wordpress supports external authn out of the box and so ShibSP configuration for Wordpress is not affected by the Wordpress patch. To be sure, I tested the login on the jpr blog and it still works after the patch.

projects.uabgrid