Faking Production – database access

One of our services has been around for a while, a realy long time.  It used to get developed in production, there is an awful lot of work involved in making the app self-contained, to where it could be brought up in a VM and run without access to production or some kinds of fake supporting environment.  There’s lots of stuff hard coded in the app (like database server names/ip etc), and indeed, and there’s a lot of code designed to handle inaccessible database servers in some kind of graceful manor.

We’ve been taking bite sized chunks of all of this over the last few years, we’re on the home straight.

One of the handy tricks we used to get this application to be better self-contained was avoid changing all of the database access layer (hint, there isn’t one) and just use iptables to redirect requests to production database servers to either local empty database schema on the VM, or shared database servers with realistic amounts of data.

We manage our database pools (master-dbs.example.com, slave-dbs.example.com, other-dataset.example.com etc) using DNS (PowerDNS with MySQL back end), in production, if you make a DNS request for master-dbs.example.com, you will get 3+ IPs back, one of which will be in your datacentre, the others will be other datacentres, the app has logic for selecting the local DB first, and using an offsite DB if there is some kind of connection issue.  We also mark databases as offline by prepending the relevant record in MySQL with OUTOF, so that a request for master-dbs.example.com will return only 2 IPs, and a DNS request for OUTOFmaster-dbs.example.com will return any DB servers marked out of service.

Why am I telling you all of this?  Well, it’s just not very straight forward for us to update a single config file and have the entire app start using a different database server. Fear not, our production databases aren’t actually accessible from the dev environments.

But what we can do is easily identify the IP:PORT combinations that an application server will try and connect to.  And once we know that it’s pretty trivial to generate a set of iptables statements that will quietly divert that traffic elsewhere.

Here’s a little ruby that generates some iptables statements to divert access to remote, production, databases to local ports, where you can either use ssh port-forwarding to forward on to a shared set of development databases, or to several local empty-schema MySQL instances:

require “rubygems”
require ‘socket’

# map FQDNs to local ports
fqdn_port = Hash.new
fqdn_port[“master-dbs.example.com”] = 3311
fqdn_port[“slave-dbs.example.com”] = 3312
fqdn_port[“other-dataset.example.com”] = 3314

fqdn_port.each do |fqdn, port|
puts “#”
puts “# #{fqdn}”
# addressess for this FQDN
fqdn_addr = Array.new

# get the addresses for the FQDN
addr = TCPSocket.gethostbyname(fqdn)
addr[3, addr.length].each { |ip| fqdn_addr << ip }

addr = TCPSocket.gethostbyname(‘OUTOF’ + fqdn)
addr[3, addr.length].each { |ip| fqdn_addr << ip }

fqdn_addr.each do |ip|
puts “iptables -t nat -A OUTPUT -p tcp -d #{ip} –dport 3306 -j DNAT –to{fqdn_port[fqdn]}”

And yes, this only generates the statements, just pipe the output into bash if you want the commands actually run.  Want to see what it’s going to do?  Just run it.  Simples.

The New Toolbox

In days gone by, any computer guy worth his salt had a collection of boot floppies, 5.25″ & 3.5″, containing a mix of MS-DOS, DR-DOS, Toms Root Boot & Norton tools. These days passed and the next set of essentials was boot cd-r, containing BartPE, RIPLinux, Knoppix etc. People quickly switched to carrying these tools USB sticks, smaller, easier to change, great when the dodgy PC you were trying to breathe life into supported USB booting.

I think there’s a better way, based on the last 3 days of hell spent setting up what should have been identical touchscreen machines (no cd, slow USB interfaces)

Your new toolkit is a cheap laptop, with a big hard disk, running the following:

  1. Your favourite Linux distro (I’ve used Ubuntu for this laptop)
  2. tftpd, dhcpd & dnsmasq setup for PXE booting other machines from this laptop (FOG uses dhcpd for all it’s automatic DHCP magic, use dnsmasq for simple local DNS, required for Unattended)
  3. FOG Cloning System
  4. Unattended Windows 2000/XP/2003 Network Install System
  5. CloneZilla PXE Image (for good measure)
  6. RIPLinux PXE Image

Why?  USB booting stills seems troublesome, installing Windows from flash seems very slow.  Nearly everything supports PXE these days, if it has a built in ethernet port, it’s pretty much guaranteed to support PXE booting.  There is nothing like the feeling of being able to image a machine into FOG over a 1Gb crossover cable in a matter of minutes.  Got everything working? image it and walk away, safe in the knowledge that if somebody comes along and breaks things, you can image it back in minutes, instead of having to do another clean install and build all your updates & software back on top.

There’s a little bit of plain in getting all of separate packages to run from the one /tftpboot/pxelinux.cfg/default, but it’s just a matter of careful copy & paste from the canned configs.


There’s a flaw in ssh-vulnkey, it doesn’t always show you the name of the file with an offending blacklisted key in it. Here’s a couple of ways round this:

For a small machine, inspect the files by hand:

strace ssh-vulnkey -a 2>&1 | grep ^stat64| grep -v NOENT| cut -d” -f 2| sort | uniq | xargs vi

Or, a little longer, using ssh-vulnkey to find all relevant keys & reprocess them displaying the filename & then the result of the ssh-vulnkey for the individual file:

strace ssh-vulnkey -a 2>&1 | grep ^stat64| grep -v NOENT| cut -d” -f 2| sort | uniq | xargs -i bash -c “echo ; echo {} ; ssh-vulnkey {};”

This really is a dirty hack, using strace to extract the files ssh-vulnkey and then reprocess them individually, there are a million ways this could be done better, but not on a single bash line 🙂

Why would you do that?

I indulged myself recently and obtained a new MacBook, which I’ve spent the last few days loading up with the usual suspects (Firefox) as well as getting acquainted with some new tools (Mail.app).

I use Foxmarks to keep my bookmarks in order, with a neat toolbar of my regular sites, I noticed that in FF on OSX, all my toolbar favicon images were missing (several of my bookmarks on the Personal Toolbar have no text, so just the Icon shows up on my other machines (XP & Ubuntu/Linux))

A bit of research and playing found this: http://blog.rbenson.info/tech/firefox-hide-bookmark-toolbar-icons.html

Now this describes exactly how to recreate my problem (Hide the Icons!) I wanted to undo this (guessing that this was just a daft default in the OSX FF build), a bit more poking & research suggested that I wanted to make my userChrome.css to look like this:

/* Hide bookmark icons in the Personal Toolbar */
toolbarbutton.bookmark-item > .toolbarbutton-icon {
display: inline !important; }

And he presto, things are back to normal, I have a neat toolbar with icons for the sites.

But why would you set the default to hide them?

Using GPRS Easy Connect 3.0.1 with Ubuntu 7.04 (Edgy Eft)

I’ve been playing around with some cobbled together pppd & chat scripts to get my Novatel Wireless U630 working under Ubuntu, with a Vodafone UK Data SIM (I think most of the problmes come from VF UK’s DNS Servers being broken in some odd way)

One of the many resources for APN settings etc is from the GPRS Easy Connect web site, http://gprsec.hu, (other great resource are filesaveas and Taniwha). After I got my stuff working by hand, I thought I would have a play with their tool that wraps all there settings together. I hit some problems installing from the source package (the supply rpms, but no .deb files), here’s how they were resolved:

wget http://www.gprsec.hu/downloads/GPRS_Easy_Connect_301.tar.bz2
tar xvjf GPRS_Easy_Connect_301.tar.bz2
cd GPRS_Easy_Connect_301
sudo ./INSTALL.sh
./INSTALL: error while loading shared libraries: libgtk-1.2.so.0: cannot open shared object file: No such file or directory

This easy resolved via the libgtk1.2 package:

sudo apt-get install libgtk1.2

Take #2:

sudo ./INSTALL.sh
Gtk2:TrayIcon perl module [Error]

Better, but still some problems, this one is easy resolved, using the libgtk2-trayicon-perl package:

sudo apt-get install libgtk2-trayicon-perl

BTW, it’s fairly easy to identify these packages required by playing about with apt-cache search, in this case:

apt-cache search trayicon | grep perl
libgtk2-trayicon-perl – Perl interface to fill the system tray

After that, the source package was able to complete it’s installation.

PS, here are the cobbled together scripts are working for me, YMMV.
/etc/ppp/ip-up.d/2override-dns (prepends the PPP IPCP supplied DNS servers with your own DNS Server of choice)

I just kick this off with a “pppd debug vodafone-U630” in a root terminal window.

PPS The Windows based Novatel tool is able to tell me all sorts of into about signal strength etc, as is the 3rd party tool MWConn, I wonder are there any specs around that would let us get similar info in a non-Windows environment?

Unison, Cygwin & syncing stuff

I have a Linux VPS account which I use for hosting this and some private photo collections [i.e. personal, not porn!] etc, I used to use Unison to sync my photo directories between my Linux VPS and my Linux Laptop. I’m currently running Windows XP on my laptop, so I was looking for a way of syncing stuff with my off site storage again, after some playing around with different versions of Unison, I settled on running unison from withing Cygwin. I have a bash script that fixes the permissions on the remote filesystem and some other stuff using SSH. Getting ssh-agent to work cleanly in a Cygwin environment is tricky, I found the following tips, and liked this one. It needed some tweaking to work, probably just the knock on effect of having spaces in my $HOME.

This fragment can be used to setup the ssh-agent so that the rest of the commands in the script that use SSH don’t prompt for passphrases (including running unison using ssh)
# Setup ssh-agent if it’s not already there
# This is where the ssh-agent environment settings are stored for scripts
# using this fragment
function start_agent {
echo “Initialising new SSH agent…”
/usr/bin/ssh-agent | sed ‘s/^echo/#echo/’ > “${SSH_ENV}”
echo succeeded
chmod 600 “${SSH_ENV}”
. “${SSH_ENV}” > /dev/null

# Source SSH settings, if they are there
if [ -f “${SSH_ENV}” ]; then
. “${SSH_ENV}” > /dev/null
# If ssh-agent is dead or gone, start from scratch
ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent > /dev/null || {
# start from scratch

After this script, I just call ssh a couple of times to tidy up the remote destination, and then call unison using a profile that also uses ssh, slicker than a greased monkey falling of a pole.

OpenBox & fbpanel

I have an old laptop (PII-400Mhz, 196Mb RAM, it’s small & light, which is why I still keep it around), it used to run Debian Woody just fine, I wiped it recently and dropped Debian Sarge on to it, it just seemed sluggish, I reckoned it couldn’t cope with all the new graphical schmuck in the new GNOME release.

One of the things I liked about Sawfish/Sawmill was the pack window feature, I can be quite keyboard orientated and liked to be able to bounce windows round without have to jump for the mouse.

Well, I tried Ion, FluxBox, BlackBox etc, but discovered that OpenBox has a MoveToEdgeEast (& West, South & North!), which I quickly mapped to Ctrl-Alt-Right etc, and hey presto, it was like being at home again…

All I then needed was some kind od task bar, fbpanel seems to fit the job so far!

Both are in Debian, so no hacking around building from source 🙂

suspend2.net rocks!

My last major requirement was decent hibernate & resume, the standard suspend to disk in 2.6.8/10 didn’t cut it, so I started on patching the kernel with stuff from suspend2.net, but it wouldn’t patch a debian’ised kernel source, or even any kernel source I tried to pull from kernel.org, then I remembered a post on debian-laptop about a some debian’ised suspend2 patches, some hunting through my inbox and I found it:

Hello, world.

I have just finished the first public (yet EXPERIMENTAL!) version of
the Debian software suspend 2 patch (www.suspend2.net). The patch
integrates with Debian’s kernel-package and modifies the
kernel-image in such a way as to automatically modify any initial
ramdisk and reinitialise swap partitions to preven data loss.

The patch currently supports Debian kernels 2.6.8 and 2.6.10. For
best results, use the latest kernel. I did most of my testing with
2.6.10 on i386. Patches for 2.4.2[789] and 2.6.11 are also included
but not tested. 2.4.27 does not currently apply to the Debian
kernels, but should be usable with vanilla ones.

Help in porting the patch to other Debian kernels would be greatly
appreciated. I am especially looking for an elegant way to support
Debian and vanilla kernels from the same source.

I am not going to upload the patch to unstable yet, but it has been
submitted to the Debian experimental archive. It will go into
unstable when I have received enough positive feedback and no
serious bugs exist.

At this point, I want to thank Nigel and everyone else who
contributed for the slick patch, which works very nicely. Also
thanks to Nigel for backporting the 2.1 version to the
kernel, which is going to be the standard 2.6 kernel for Debian

If you want to give the patch a whirl, you can obtain it from the
following APT repository:

deb http://debian.madduck.net ~madduck/packages/stage/kernel-patch-suspend2/
deb-src http://debian.madduck.net ~madduck/packages/stage/kernel-patch-suspend2/

then install it:

apt-get update
apt-get install kernel-patch-suspend2

and then compile the kernel:

tar xjf /usr/src/kernel-source-2.6.10.tar.bz2
cd kernel-source-2.6.10
make-kpkg –added-patches suspend2 … binary

Then, install the kernel-image and reboot.

Note that I did my testing using an initial ramdisk (basically the
plain Debian kernel with SUSPEND2 enabled). If you can use an
initial ramdisk, pass the –initrd option to make-kpkg and be done.
Without an initial ramdisk, you have to make sure to pass the right
‘resume2’ value to the kernel via Grub/Lilo, or whichever is your
bootloader. I plan to add a warning about this when I have more

I suggest also pulling in the hibernate image (kernel images with
this patch will Recommend it). With UseSwsusp2 set to yes in
/etc/hibernate/hibernate.conf, just running ‘hibernate’ should
suspend the machine, and it should come back up without any more of
your doing when you start it again.

Even though I did all testing on my productive machine(s) and lost
no data (knock on wood), it’s probably best if you try it on
a less-important machine. Nevertheless, you can turn off
$CHECK_SWAP_PARTITIONS_ON_BOOT in /etc/default/suspend2 to prevent
the swap partition check. Make sure to read
http://suspend2.net/HOWTO-4.html#ss4.4 about this issue.

Have fun, feedback welcome.

— Please do not send copies of list mail to me; I read the list! .”`. martin f. krafft : :’ : proud Debian developer, admin, user, and author `. `’` `- Debian – when you have better things to do than fixing a system Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver!

It works a treat!
After you’ve built your patched kerel, grab the hibernate package from suspend2.net:

wget http://cp.yi.org/apt/hibernate/hibernate_1.05-1_all.deb

dpkg -i hibernate_1.05-1_all.deb

BTW, I rebuild my kernel’s slightly differently, I don’t do initrd or anything, so all I did was:

cd /usr/src/linux
make-kpkg –added-patches suspend2 –append-to-version -sm7-suspend2 kernel_image

I do this because I sm7 indicates that this is the 7th interation of tweaking settings for this laptop, and that the suspend2 patch has been applied, it makes the kernel namesa bit long, but easier to spot what I’ve done 🙂

Reboot using your new kernel, et voila! You’re ready to hibernate, get all your usual apps up and running (Thunderbird, Firefox, Word using CrossOver), fire up a shell, su to root and type hibernate, it rocks…