For the past couple of weeks I’ve been getting more and more frustrated with Ubuntu’s 10.04 LTS release. The latest round of patches broke LOADS of things my company uses on a daily basis :

• A samba/winbind update which didn’t trigger a proper service restart meant none of the Active Directory users could log on anymore. Net result : a few dozen workstations which needed admin intervention.

• The Adobe Acrobat package was split into acroread and acroread-common ; by itself no mayor problem. However when updating it asks again if you wish to accept the license. This used to be part of the acroread package but is now part of acroread-common. So no setting in debconf present. With unattended updates things just hang since you don’t see the question as a user. With preseed deploys ditto… The fix is easy but a lot of wasted time.

• Several machines which use the proprietry nVidia driver developed ‘issues’ and the easiest fix so far was to boot the previous kernel again. Yet more manual admin intervention and quite a bit of grief

The biggest issue so far is the fact that with Ubuntu, unless you buy into Canonical’s Landscape / Ubuntu One you have basically no control over updates and no management tools. That allows for 3 options : no patching (not good, machines can go online and some updates fix bugs or add functionality which is needed by users) , manual patching (undoable with over 4 dozen workstations) or by automating it as much as possible. (Unfortunately Spacewalk support for Ubuntu is still not useable yet, hopefully a next release will address most of the outstanding Ubuntu issues)

So we’ve set up a simple proxy and a custom repository for our inhouse software. To have a (somewhat) consistant setup across the company we’ve got a CRON job every week to do an apt-get update ; apt-get upgrade

This should work just fine, if Ubuntu would remain consistant at least within a release. However issues such as above indicate you can not trust any Canonical supplied update to apply clean and smooth.

Exactly that is one of the qualities I expect from any enterprise distribution: being consistant, reliable and predictable.

The more I work with Ubuntu the more I start longing for a RedHat workstation (or one of it’s proper derivatives, such as Scientific Linux. Definately no longer CentOS which got lost in it’s own politics somewhere along the way causing things to lag by months which to me is also unacceptable) In the almost 3 years that I used RedHat while working for a big hosting company I’ve only once seen a RedHat update go seriously wrong. Through a RedHat Satellite server I was totally comfortable in updating hundreds of machines , including new kernels, and rebooting them without the fear of machines not coming back up as they should.

I never really had that level of confidence in Ubuntu but recent events really made me loose the last bit of faith in this distribution which is more like a pool of quicksand, just covered up really nice. But I prefer stability and reliability over looks any day…

The latest Ubuntu 11.10 release appears to contain even more bloat, still no large deployment management tools in sight and going by the bug reports and forums stability is even worse than before. So once we need to renew our distro again I’ll be seriously considering alternatives. If anybody else has recommendations I’m open to suggestions!

** EDIT 19 dec 2011 EDIT **

They’ve gone from bad to worse! Oracle has not renewed the distribution license for it’s Sun Java packages so Ubuntu can no longer ship sun-java6 from the partner repository. That I can understand…

However they’ve decided to ‘pro-actively’ remove it on all systems and replace it with openjdk without asking for user input, all under the security mantra. That is TOTALLY ABSURD. openjdk != sun-java6 , it’s still lacking in many ways (the java plugin doesn’t work properly with Lights-Out modules of several vendors for instance) And I can only speculate on the horrors of Tomcat/JBoss servers running Ubuntu…

Here’s my workaround for this latest fuckup :

sudo aptitude hold sun-java6-bin sun-java6-fonts sun-java6-jdk sun-java6-jre sun-java6-plugin

That makes these packages ‘immutable’ ; but to be safe I also fetched a copy of the latest version + sources, so I won’t be stuck whatever way things proceed to go.

(For those that wonder why I still put up with Ubuntu; it’s the OS of choice at work and I want to keep ‘current’. My personal preference is Red Hat Enterprise Linux or it’s derative Scientific Linux

Tags: linkedin
Categories: Linux
Comments Off

After having Spacewalk 1.5 up and running for about a month here are some observations, useful hints and such…

First of all, support for Debian/Ubuntu systems is still too much in development to be usable; there are no packages for the latest Ubuntu LTS release 10.04 (Lucid Lynx) which I am using both at home and at work. There are some debian packages, but it’s unsure as to which version they’re meant for (I’m guessing 5.0, dubbed ‘Lenny’, since it has to be newer than Ubuntu 10.04 given some of the version dependencies) So I’ll hold out until version 1.6 to come around, as that hopefully contains some more fixes that allow the distribution of .deb files through spacewalk (import of .deb files is currently broken)

After having tested all the options, including updating / installing packages, I removed all the yum repositories on the client machines to make sure all files now come through spacewalk. Some people might find this overkill, but it’s a failsafe way to make sure everything goes through Spacewalk.

In my installation I use a dedicated PostgreSQL server (I plan on using it for other PostgreSQL databases as well in future) I’ve been reading up on PostgreSQL tuning and right now the database seems to be running quite stable and fast enough on just 1 core and 1 GB of RAM. This was done largely with the following settings :

shared_buffers = 32MB
temp_buffers = 8MB
work_mem = 32MB
maintenance_work_mem = 128MB

The shared_buffers option might seem low, but according to most documents PostgreSQL already benefits largely from the OS doing the (disk) caching. So I set it conservative. The relatively large value of maintenance_work_mem was set to speed up the autovacuum process by reducing disk IO and that is working out very well; I only see short large write bursts now instead of large sustained periods of both reads and writes, causing my system load to blow up.

In my previous post I showed the registration script I use, to get around the use of external repositories on all clients. After that post, I decided to just use my custom child channel for every release to include all the needed RPM’s and just update that child channel regularly. I can leave the RPM’s on the webserver as is and immediately after registration update as needed. This makes maintenance a lot easier. In future I might even rewrite the script to use the packages from Spacewalk itself, which is an even better way. Another improvement I’ve made since is to run, as a last step, rhn-profile-sync to make sure satellite has the right info, both soft- and hardware-wise. I had a few machine who appeared to be ‘off’ between actual state and what Spacewalk had in it’s database…

Tags: linkedin
Categories: Linux
Comments Off

In a previous job I worked for a large hosting company where I was responsible for over 700 Linux servers and the patch management that goes with them. To handle such a large amount of machines you need a proper tool and luckily I had such a tool available to me : we used a Red Hat Satellite server.

Unfortunately such a license is rather expensive (think in the 10K range and up) so for many smaller companies it’s not feasible. Enter Spacewalk!
Spacewalk is the playground of the Satellite developers. And even better: Spacewalk unlike Satellite doesn’t require an Oracle database!

I am very familiar with MySQL and know my way around a PostgreSQL database server but an Oracle database server to me is still a black box… up until version 1.5 Spacewalk ran on PostgreSQL but functionality was very limited. With version 1.5 everything but the monitoring part works like a charm. And best of all : this makes the setup even easier than the full commercial product!

The requirements aren’t bad either; running it on a VM with 1 core and 4 GB RAM right now with no complaints; only thing to watch out for is that you’ll need around 7-8 GB of diskspace per repository PLUS another 2-3 GB for the SQL (probably per repository …)

In less than 2 hours from starting with a clean Scientific Linux VM I was already registering new servers to our Satellite server, thanks to the excellent step-by-step setup guides. The only thing I didn’t like was the registration procedure: it depends upon several external repositories. I never like to just add external repo’s since they might contain different versions of ‘base software’ so I made a note of only the absolute required RPM’s, pulled those to the satellite server and wrote this little register script


#!/bin/bash
export http_proxy=
yum install -y PyXML libselinux-python > /dev/null
cd /usr/share/rhn/
wget -o /dev/null http://spacewalk_server_FQDN/pub/RHN-ORG-TRUSTED-SSL-CERT > /dev/null

echo "== Dependencies installed =="

rpm -Uhv http://spacewalk_server_FQDN/sl6register/jabberpy-0.5-0.20.el6.noarch.rpm http://spacewalk_server_FQDN/sl6register/rhncfg-actions-5.10.9-1.el6.noarch.rpm http://spacewalk_server_FQDN/sl6register/rhnsd-4.9.12-1.el6.x86_64.rpm http://spacewalk_server_FQDN/sl6register/osad-5.10.18.2-1.el6.noarch.rpm http://spacewalk_server_FQDN/sl6register/rhncfg-client-5.10.9-1.el6.noarch.rpm http://spacewalk_server_FQDN/sl6register/rhn-setup-1.5.16-1.el6.noarch.rpm http://spacewalk_server_FQDN/sl6register/python-hwdata-1.2-1.el6.noarch.rpm http://spacewalk_server_FQDN/sl6register/rhn-check-1.5.16-1.el6.noarch.rpm http://spacewalk_server_FQDN/sl6register/spacewalk-backend-libs-1.5.45-1.el6.noarch.rpm http://spacewalk_server_FQDN/sl6register/rhn-client-tools-1.5.16-1.el6.noarch.rpm http://spacewalk_server_FQDN/sl6register/yum-rhn-plugin-1.5.11-1.el6.noarch.rpm http://spacewalk_server_FQDN/sl6register/rhncfg-5.10.9-1.el6.noarch.rpm http://spacewalk_server_FQDN/sl6register/rhnlib-2.5.41-1.el6.noarch.rpm > /dev/null 2>&1
rm -f /etc/sysconfig/rhn/osad.conf > /dev/null
cd /etc/sysconfig/rhn/
wget -o /dev/null http://spacewalk_server_FQDN/sl6register/osad.conf > /dev/null

echo "== Satellite client installed =="

# Register action
/usr/sbin/rhnreg_ks --serverUrl=http://spacewalk_server_FQDN/XMLRPC --activationkey=1-my_activation_key > /dev/null

echo "== Registration completed =="

/usr/bin/rhn-actions-control --enable-run > /dev/null
/etc/init.d/osad restart > /dev/null 2>&1

echo "== OSAD initialized =="

With this setup you also get the two-way communication where updates get picked up directly and where remote script execution (one of my favorite features of Satellite / Spacewalk!) is enabled. I chose this method since it doesn’t require ANY configuration on a new VM or physical machine apart from a working yum config just with the base RedHat/CentOS/Scientific Linux packages, no extra repo’s needed. I pulled the RPM’s (listed on the one big ‘rpm -Uhv’ line) from the spacewalk-client and EPEL repositories (choose the right version and architecture!) Only thing to watch out for is that if they get updated I will need to manually fetch them and probably put them in a local repository (Red Hat does with the the RHN Tools repository) and update my script.

Now all that’s missing is the errata functionality. Unfortunately that still appears to be missing; there’s a script to fetch CentOS errata but as we’re not using CentOS it doesn’t appear to be very useful (it’s hardcoded to work with the CentOS mailing list archives) If anyone has found a good solution I’d love to hear it!

All in all hats off to the Spacewalk developers for an excellent product. Now to check out how much is possible with the recently added apt-bridge so that I can also use it to manage our 50 or so Ubuntu machines…

Tags: linkedin
Categories: IT, Linux
2 Comments »