wiki:CephUpgrade
Last modified 22 months ago Last modified on 10/28/15 16:37:46

Notes on upgrading the Ceph software on our Dell PoC Crowbar/Chef? managed fabric.

The Chef recipes for ceph will auto update software after an apt-get update. The summary of our approach will be to successively stage the next release of ceph packages in in the ceph barclamp package database, apt-get update on a node, manually run the chef-client to allow the ceph default.rb recipe to update the software, restart the monitor processes, then repeat the same for the osd nodes and restart the osds.

We will run chef-client by hand to control install dependencies and the order of execution. Chef is primarily intended to serve as our upgrade trigger so we can leverage the PoC built-in barclamp repo structure. This allows us to upgrade ceph without removing the cluster from the PoC.

The PoC nodes are currently at Ubuntu 12.04.1 and Ceph at Bobtail 0.56. The goal is to update Ceph all the way to Hammer 0.94.

Some open questions

  • how far forward can we take the Ceph binaries without upgrading any other parts of the system, ie. can we stay at 12.04.1?
  • are there dependencies on in libvirt used by Nova nodes for the rbd client libraries working with newer ceph code

Setting up a Test Env

In order test this effectively, we need to work with a set of test nodes. It's convenient to use our OpenStack fabric for the dev nodes. We will focus on testing the Chef interaction and Ceph cookbook. We will avoid using the Crowbar auto provision for the dev nodes and instead provision them semi-automatically, using nova instance management tools.

Some preliminary steps:

  • Create a poc-dev project in openstack
  • Add myself as a user
  • Create security group chef with port 4000 and 4040 access

Add Ubuntu 12.04.1 to glance

wget https://cloud-images.ubuntu.com/releases/precise/12.04.1/ubuntu-12.04-server-cloudimg-amd64-disk1.img
glance image-create --name "precise - 12.04.1 ubuntu-12.04-server-cloudimg-amd64-disk1" --container-format bare --disk-format qcow2 --is-public True --file ubuntu-12.04-server-cloudimg-amd64-disk1.img

Create an admin instance to host Chef

Provision the 12.04.1 into a small instance. Assign public IP and open ssh port.

Connect apt to the PoC repo

Open an port forward to the admin node

ssh -L 8091:admin:8091 jpr@172.22.1.9

Replace the default repo from the cloud image with those of the PoC

Get the collection of barclamp repos from one of the nodes in the PoC cluster

tar -cf apt-sources.tar /etc/apt/sources.list.d/

Bring put the repos in place on our cloud node

scp jpr@172.22.1.9:apt-sources.tar .
sudo tar -xf apt-sources.tar -C /
sudo mv /etc/apt/sources.list /etc/apt/sources.list-orig

Update the config to point at the localhost proxy

for file in *.list
do
 sed -e 's/172.16.128.101/127.0.0.1/' $file > $file.tmp
 mv $file.tmp $file
done

Update the apt cache

sudo apt-get update

Install a test chef server

sudo apt-get install chef-server-webui

This configures the chef server and prompts for a hostname and password.

Once complete can access the server at http://admin.poc-dev:4040