MySQL DNS woes

Today I was trying to bring up a mysql instance to begin setting up a mysql cluster and faced a strange problem. I was unable to connect to mysql/mysqladmin giving the host option both on the box where mysql is installed and from another box with just the client.

root@lucid-myndbmgr# mysqladmin version
mysqladmin  Ver 8.42 Distrib 5.1.39-ndb-7.0.9, for debian-linux-gnu on i486
Copyright 2000-2008 MySQL AB, 2008 Sun Microsystems, Inc.
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under GPL license

Server version      5.1.39-ndb-7.0.9-1ubuntu7
Protocol version    10
Connection     Localhost via UNIX socket
UNIX socket     /var/run/mysqld/mysqld.sock
Uptime      24 min 57 sec
Thread: 1  Questions: 7  Slow queries: 0  Opens: 15  Flush tables: 1  Open tables: 8  Queries per second avg: 0.4   
root@lucid-myndbmgr# ifconfig eth0
eth0       Link encap:Ethernet  HWaddr 08:00:27:1d:ed:e8
       inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
       inet6 addr: fe80::a00:27ff:fe1d:ede8/54 Scope:Link
       UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
       RX packets: 22 errors:0 dropped:0 overruns:0 frame:0
       TX packets: 21 errors:0 dropped:0 overruns:0 carrier:0
       ncollisions:0 txqueuelen:1000
       RX bytes:3294 (3.2 KB)  TX bytes:2136 (2.1 KB)
       Interrupt:10 Base address:0xd020

root@lucid-myndbmgr# mysqladmin ping
mysqld is alive
root@lucid-myndbmgr# hostname
lucid-myndbmgr
root@lucid-myndbmgr# grep 'bind-address' /etc/mysql/my.cnf
bind-address       = 10.0.0.2
root@lucid-myndbmgr# netstat -an | grep 3306
tcp       0       0   10.0.0.2:3306  0.0.0.0:* LISTEN
root@lucid-myndbmgr# mysqladmin -h 10.0.0.2 ping
mysqladmin: connect to server at '10.0.0.2' failed
error: 'Can't get hostname for your address'

After some furious googling and some wisdom from #mysql I figured out that mysql requires a functional DNS to work (I still haven't figured out why my DNS config is getting ignored :(..). Initial suggestions were to open up port 3306, remove bind-address and do not use skip-networking. They din't help

root@lucid-myndbmgr# iptables -A INPUT -p tcp --dport 3306 -s 0/0 -j ACCEPT
root@lucid-myndbmgr# iptables -A OUTPUT -p tcp --sport 3306 -d 0/0 -j ACCEPT
root@lucid-myndbmgr# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT    tcp --  anywhere   anywhere  tcp dpt:mysql

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT    tcp --  anywhere   anywhere tcp spt:mysql
root@lucid-myndbmgr# grep 'bind-address\|skip-networking' /etc/mysql/my.cnf
# Instead of skip-networking the default is now to listen only on
#bind-address       = 10.0.0.2
root@lucid-myndbmgr# mysqladmin -h 10.0.0.2 ping
mysqladmin: connect to server at '10.0.0.2' failed
error: 'Can't get hostname for your address'

So, as a workaround found skip-name-resolve parameter which conveniently skips this.

root@lucid-myndbmgr# grep 'skip-name-resolve' /etc/mysql/my.cnf
skip-name-resolve
root@lucid-myndbmgr# mysqladmin -h 10.0.0.2 ping
mysqld is alive
root@lucid-myndbmgr# 

Prevent vi from opening files for which you do not have read permission

Found an annoying thing in one of the new servers I was playing around. It generated some log files as root and permissions of 600 and others as the designated user. Now the problem is that when I open log files with 600 permission in vi it happily opens up but, shows a blank buffer with 'Permission Denied' error. After a couple of times I got really annoyed and decided to fix it.

Attempt 1: ~/bin is ahead in the path to /usr/bin/vi so, wrote a simple script to check for file permissions.

#!/bin/sh
if [ -e $1 -a ! -r $1 ]; then
  echo No read permission for $1
  exit 1
fi
/usr/bin/vi $1

But, sadly that did not cut it
Attempt 2: Stuffed the above in an alias. I'll leave as an exercise for you to figure out what happens
Attempt 3: Then converted the same logic to a shell function called vi and w00t! it worked :)

vi() {
  if [ -e $1 -a ! -r $1 ]; then
    echo No read permission for $1
  else
    /usr/bin/vi $1
  fi
}

Mounting VM images with multiple partitions

Of late, I have been playing around with the eucalyptus cloud project. I have a pretty basic setup with a single cloud-controller/cluster-controller and 2 nodes (using xen on centos 5.4). Usually the simplest way to mount a vm image file is as follows (find a more detailed description here)

$ # associate image file with a loop device
$ losetup /dev/loop0 jaunty.img
$ # create a mount point
$ mkdir mnt
$ # mount it :)
$ mount -t ext3 /dev/loop0 mnt
$ # chroot to navigate easily
$ chroot mnt

Then I found the jboss CirrAS project that gives a vm image with a clustered jboss setup (both for ec2 and xen/kvm). So, out of curiosity I downloaded the images and since my eucalyptus cloud is operating in SYSTEM, I have to setup the vm instance such that it pings the CLC a few times so that the IP address gets updated. When I tried the tried and tested method above...boom nothing happened I kept getting

$ mount -t ext3 /dev/loop0 mnt
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

So, after some googling and some very helpful hints from #stormgrind turns out the CirrAS images have 2 partitions and if there are multiple partitions the loop device must be given an offset from which the actual image starts and the partition table ends :|. Thankfully mgoldmann in the irc channel suggested some sites that worked like a charm :)

$ # associate image file with a loop device with offset
$ losetup -o 32256 /dev/loop0 back-end-sda.raw
$ # create a mount point
$ mkdir mnt
$ # mount it :)
$ mount -t ext3 /dev/loop0 mnt
$ # w00t

How did I arrive at 32256? Simple. For images created by bximage you must use the value 32256 words of wisdom from here References http://bochs.sourceforge.net/doc/docbook/user/loop-device-usage.html http://varghese85-cs.blogspot.com/2008/11/mouting-partitions-with-losetup.html

iptables-fu

I've had a love hate relationship with iptables. I love the control it offers but, am quite confounded by all the low-level networking concepts at play. Today, I had to do some simple stuff, like blocking pings and block ssh from a particular IP. Here is my humble attempt at both

# reject all ping requests
iptables -A INPUT -p icmp -j REJECT
# drop all ssh (tcp:22) requests from 10.0.0.37
# DROP means that whoever is trying to connect from 37 will not get a connection refused...devious >(
iptables -A INPUT -p tcp --sport 22 -s 10.0.0.37 -j DROP