Testing Out Oracle’s Unbreakable Enterprise Kernel 2

As announced a few days ago, Oracle’s core database product is now supported on Oracle Linux 6. Coming a full 13 months after Oracle Linux 6’s launch, and 16 months after Red Hat Enterprise Linux 6, it’s a much anticipated announcement.

Update 28-Mar-12: The official certification information has come out on My Oracle Support. So far, it’s only certified for Oracle Linux 6 Unbreakable Enterprise Kernel version 1 and only for version 11.2.0.3 under Linux x86-64. It also means that a certified combination should be possible using the Oracle-supplied OEL 6 EC2 AMI, though it’s frozen at the original 6.2 release kernel. Unfortunately, Oracle 11.2.0.3 is not currently available on OTN but rather requires an active license to download from My Oracle Support.

Oracle’s UEK2 (and Red Hat Enterprise Linux 6) adds some very interesting features like:

• btrfs, a big break in filesystem organization from traditional ext2/3, with the promise of lightweight copy-on-write snapshot support,
• transparent hugepages, dynamically allocating hugepages as needed and performing background memory defragmentation to attempt to free up contiguous space, and
• transmit packet steering, allowing multiple CPUs to load-balance outgoing network traffic.

Although the press release states that it’s available “immediately”, I couldn’t find any notes on the My Oracle Support support portal relating to the product support; in fact, it still lists Oracle Enterprise 6 as being uncertified as of this writing. So I’m not sure how it will pass the pre-installation operating system checks.

No matter; I’m going to test this out. An obvious way to do this would be to use Amazon EC2, providing high-capacity instances on demand.

After some blind allies getting the Oracle Linux UEK2 kernel working with Amazon EC2 and Oracle VM, I found that I could make it work without Oracle VM, with Amazon’s default Xen hypervisor. Here are the steps I used:

– Sign up for an Amazon EC2 account and set up the EC2 API tools on your client machine. There are lots of tutorials on how to do this online.

– Create a new EC2 instance using a “builder” AMI; I chose a 64-bit CentOS 6 image “CentOS 6 PVGrub EBS AMI”, AMI ID ami-1f66b276:

[[email protected] tmp]$ ec2-run-instances -k marc-aws -n 1 -t m1.medium -z us-east-1d ami-1f66b276
RESERVATION     r-d18f28b2      462281317311    default
INSTANCE        i-22d8f846      ami-1f66b276                    pending marc-aws        0             m1.medium        2012-03-24T21:25:11+0000        us-east-1d      aki-427d952b                    monitoring-disabled                                    ebs                                     paravirtual   xen              sg-5fc61437     default

– Assign a name to the instance.

[[email protected] tmp]$ ec2-create-tags i-22d8f846  --tag 'Name=Instance Builder'
TAG     instance        i-22d8f846      Name    Instance Builder

– Authorize the incoming SSH.

[[email protected] ~]$ ec2-authorize default -p 22 -s $(ip addr list dev eth0 | awk '/inet / {print $2}')

– Create a new 20G EBS volume; this will be the “golden image” root disk. Attach it to the builder instance.

[[email protected] tmp]$ ec2-create-volume -s 20 -z us-east-1d
VOLUME  vol-d7340cbb    20              us-east-1d      creating        2012-03-24T21:31:39+0000
[[email protected] tmp]$ ec2-attach-volume -i i-22d8f846 -d /dev/sdd vol-d7340cbb
ATTACHMENT      vol-d7340cbb    i-22d8f846      /dev/sdd        attaching       2012-03-24T21:33:26+0000

– Get the IP address to connect to (substituting the correct image ID and hostname):

[[email protected] tmp]$ ec2-describe-instances  i-22d8f846
RESERVATION     r-d18f28b2      462281317311    default
INSTANCE        i-22d8f846      ami-1f66b276    ec2-50-19-45-24.compute-1.amazonaws.com ip-10-116-237-78.ec2.internal  running marc-aws        0               m1.medium       2012-03-24T21:25:11+0000      us-east-1d       aki-427d952b                    monitoring-disabled     50.19.45.24     10.116.237.78 ebs                                      paravirtual     xen             sg-5fc61437     default
BLOCKDEVICE     /dev/sda1       vol-39310955    2012-03-24T21:25:28.000Z        true
TAG     instance        i-22d8f846      Name    Instance Builder
[[email protected] tmp]$ ssh -i marc-aws.pem [email protected]

– Find the volume inside our AMI, which just got hotplugged:

[[email protected] ~]# dmesg | tail -2
blkfront: xvdh: barriers disabled
 xvdh: unknown partition table

– Create a filesystem and mount it. Note: I’m not creating a partition table. It’s a raw filesystem. This will make things much easier if the volume ever needs to be re-sized.

[[email protected] ~]# mke2fs -j -L / /dev/xvdh
mke2fs 1.41.12 (17-May-2010)
Filesystem label=/
OS type: Linux
...
This filesystem will be automatically checked every 35 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[[email protected] ~]# mkdir /mnt/ec2-fs
[[email protected] ~]# mount /dev/xvdh /mnt/ec2-fs

– Create the base directories, as per Jonathan Hui’s excellent blog post.

mkdir /mnt/ec2-fs/dev
/sbin/MAKEDEV -d /mnt/ec2-fs/dev/ -x console
/sbin/MAKEDEV -d /mnt/ec2-fs/dev/ -x null
/sbin/MAKEDEV -d /mnt/ec2-fs/dev/ -x zero
mkdir /mnt/ec2-fs/etc
mkdir /mnt/ec2-fs/proc

– Create /etc/fstab:

cat > /mnt/ec2-fs/etc/fstab <<EOF
LABEL=/ /       ext3    defaults 1 1
none    /proc   proc    defaults 0 0
none    /sys    sysfs   defaults 0 0
EOF

– Mount /proc:

mount -t proc none /mnt/ec2-fs/proc

– Grab the OEL 6 yum config file:

[[email protected] ~]# cd /root
[[email protected] ~]# wget https://public-yum.oracle.com/public-yum-ol6.repo
--2012-03-24 22:42:54--  https://public-yum.oracle.com/public-yum-ol6.repo
Resolving public-yum.oracle.com... 141.146.44.34
Connecting to public-yum.oracle.com|141.146.44.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1461 (1.4K) [text/plain]
Saving to: âpublic-yum-ol6.repoâ
100%[======================================>] 1,461       --.-K/s   in 0s
2012-03-24 22:42:55 (106 MB/s) - âpublic-yum-ol6.repoâ
cat <<-EOF >> public-yum-ol6.repo
[main]
cachedir=/var/cache/yum
debuglevel=2
logfile=/var/log/yum.log
exclude=*-debuginfo
gpgcheck=0
obsoletes=1
pkgpolicy=newest
distroverpkg=redhat-release
tolerant=1
exactarch=1
reposdir=/dev/null
metadata_expire=1800
EOF

– Install the base OS:

[[email protected] ~]# yum -c /root/public-yum-ol6.repo --installroot=/mnt/ec2-fs -y groupinstall Core
ol6_latest                                               | 1.1 kB     00:00
...

– Install the latest UEK2 kernel:

yum -c /root/public-yum-ol6.repo --enablerepo=ol6_UEK_latest --installroot=/mnt/ec2-fs -y install kernel-uek

(not small: 200m+ for the kernel alone)

– Set up base networking scripts:

cat > /mnt/ec2-fs/etc/sysconfig/network-scripts/ifcfg-eth0 <<EOF
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
PEERDNS=yes
IPV6INIT=no
EOF
echo "NETWORKING=yes" > /mnt/ec2-fs/etc/sysconfig/network
echo "nameserver 172.16.0.23" > /mnt/ec2-fs/etc/resolv.conf
echo "UseDNS  no" >> /mnt/ec2-fs/etc/ssh/sshd_config
echo "PermitRootLogin without-password" >> /mnt/ec2-fs/etc/ssh/sshd_config
echo "hwcap 0 nosegneg" > /mnt/ec2-fs/etc/ld.so.conf.d/libc6-xen.conf

– Script download of SSH private key on startup in case it’s missing (though with EBS-backed storage this shouldn’t be necessary).

cat > /mnt/ec2-fs/usr/local/sbin/get-sshkey.sh <<EOF
#!/bin/sh
if [ ! -d /root/.ssh ] ;
then mkdir -p /root/.ssh
chmod 700 /root/.ssh
fi
# Fetch public key using HTTP
/usr/bin/curl -f https://169.254.169.254/latest/meta-data/public-keys/0/openssh-key > /tmp/my-key
if [ $? -eq 0 ] ;
then
cat /tmp/my-key >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
rm -f /tmp/my-key
fi
EOF
chmod u+x /mnt/ec2-fs/usr/local/sbin/get-sshkey.sh
echo "/usr/local/sbin/get-sshkey.sh" >> /mnt/ec2-fs/etc/rc.d/rc.local

– Clean up temporary files from the installs (mostly the RPM packages).

yum -c /root/public-yum-ol6.repo --installroot=/mnt/ec2-fs clean all

– Set up GRUB boot files.

chroot /mnt/ec2-fs
cd /boot/grub
cat > grub.conf <<EOF
default=0
fallback=1
timeout=1
title Oracle Linux UEK
        root (hd0)
        kernel $(echo /boot/vmlinuz*uek.x86_64) ro root=LABEL=/ ro console=hvc0 crashkernel=auto LANG=en_US.UTF8 selinux=0
        initrd $(echo /boot/initramfs*uek.x86_64.img)
title Oracle Linux Compatibility Kernel
        root (hd0)
        kernel $(echo /boot/vmlinuz*el6.x86_64) ro root=LABEL=/ ro console=hvc0 crashkernel=auto LANG=en_US.UTF8 selinux=0
        initrd $(echo /boot/initramfs*el6.x86_64.img)
EOF
ln -s grub.conf menu.lst
exit

– Set up swap. (There’s no need to put this on expensive EBS storage when ephemeral storage will do.)

cat > /mnt/ec2-fs/usr/local/sbin/add-swap.sh <<EOF
#!/bin/bash
VOL=\${1-/dev/xvdb}
SIZE=\${1-2097152}
dd if=/dev/zero of=\$VOL bs=\${SIZE}k count=1 && mkswap \$VOL \$SIZE && swapon \$VOL
EOF
chmod +x /mnt/ec2-fs//usr/local/sbin/add-swap.sh
echo "/usr/local/sbin/add-swap.sh /dev/xvdb 2097152" >> /mnt/ec2-fs//etc/rc.d/rc.local

– Unmount our root disk.

umount /mnt/ec2-fs/proc
umount /mnt/ec2-fs

– Log out of the builder instance; our work there is done.

exit

– Create a snapshot of the root volume. Use the volume ID originally used to create the volume.

[[email protected] tmp]$ ec2-create-snapshot -d "UEK2 AMI creation point" vol-d7340cbb
SNAPSHOT        snap-b94519c3   vol-d7340cbb    pending 2012-03-25T02:05:43+0000                462281317311   20      UEK2 AMI creation point

– Check when it’s completed.

[[email protected] tmp]$ ec2-describe-snapshots snap-b94519c3
SNAPSHOT        snap-b94519c3   vol-d7340cbb    completed       2012-03-25T02:05:43+0000        100%  462281317311     20      UEK2 AMI creation point

– Register the snapshot, effectively creating an AMI image. This is a long command and cannot be changed once created, so it deserves some explanation:

-n: user-provided unique name
-a: architecture, which must match the 64-bit kernel
-d: description, a text description
–root-device-name: This maps to the “root” parameter given to the PVGRUB bootloader.
-b: block mapping. There are two here: one pointing to the root volume snapshot we just created, and one on non-permanent storage we’ll use for swap.
–kernel: This kernel is actually a stub kernel running PVGRUB, a bootloader that loads the UEK2 kernel from the root drive. This particular kernel is for a 64-bit unpartitioned image in the us-east region.

The kernel ID is a generic 64-bit AMazon PVGRUB kernel for the US-East region

[[email protected] tmp]$ ec2-register -n UEK2-AMI -a x86_64 -d "AMI using the Oracle Linux UEK2 kernel" --root-device-name /dev/sda -b /dev/sda=snap-b94519c3 -b /dev/sdb=ephemeral0 --kernel aki-427d952b
IMAGE   ami-c39f41aa

– Now for the moment of truth: launch a VM based on the newly-created AMI.

[[email protected] tmp]$ ec2-run-instances -k marc-aws -n 1 -t m1.medium  ami-c39f41aa
RESERVATION     r-19b0167a      462281317311    default
INSTANCE        i-5688ab32      ami-c39f41aa                    pending marc-aws        0             m1.medium        2012-03-25T00:08:10+0000        us-east-1d      aki-427d952b                    monitoring-disabled                                    ebs                                     paravirtual   xen              sg-5fc61437     default
[[email protected] tmp]$ ec2-describe-instances i-5688ab32
RESERVATION     r-19b0167a      462281317311    default
INSTANCE        i-5688ab32      ami-c39f41aa    ec2-23-20-123-219.compute-1.amazonaws.com       ip-10-62-98-125.ec2.internal   running marc-aws        0               m1.medium       2012-03-25T02:08:10+0000       us-east-1d      aki-427d952b                    monitoring-disabled     23.20.123.219   10.62.98.125                   ebs                                     paravirtual     xen             sg-5fc61437    default
BLOCKDEVICE     /dev/sda        vol-d59aa2b9    2012-03-25T00:08:28.000Z        true
[[email protected] tmp]$ ec2-create-tags --tag="Name=UEK2 Test Instance" i-5688ab32
TAG     instance        i-5688ab32      Name    UEK2 Test Instance

-Ssh’ing into the machine, we can confirm it’s running the UEK:

[[email protected] ~]# uname -a
Linux ip-10-62-98-125 2.6.39-100.5.1.el6uek.x86_64 #1 SMP Tue Mar 6 20:26:00 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

Adding swap

Oracle’s pre-installation steps require swap space. Since Amazon EBS charges for storage by the GB, it makes little sense to pay for persistent storage for swap. The alternative is to use transient storage for this. Since we can’t be guaranteed of a state at boot time, it’s safest to zero it out and create swap at that point. We set aside some space on /dev/sdb (which maps to /dev/xvdb since the Oracle UEK kernel doesn’t do the drive mapping that the Amazon kernel does).

We’ll create a startup script to run in rc.local, the last point in the startup. It will take a while to run, but since sshd and other system services will already be running, it shouldn’t slow down the startup of any other processes.

cat > /usr/local/sbin/add-swap.sh <<EOF
#!/bin/bash
VOL=\${1-/dev/xvdb}
SIZE=\${1-2097152}
dd if=/dev/zero of=\$VOL bs=\${SIZE}k count=1 && mkswap \$VOL \$SIZE && swapon \$VOL
EOF
chmod +x /usr/local/sbin/add-swap.sh
echo "/usr/local/sbin/add-swap.sh /dev/xvdb 2097152" >> /etc/rc.d/rc.local

Troubleshooting

There isn’t a whole lot of troubleshooting tools in EC2, especially compared to something like Oracle databases. There is one invaluable tool to debug AMI builds though: the console output. It usually takes several minutes to appear, but it can help determine what went wrong when an instance is inaccessible.

[[email protected] tmp]$ ec2-get-console-output i-76634012

It was particularly frustrating to get GRUB to find the root devices; when it can’t find them, it just displays a grubdom> prompt without an error message. The official documentation recommends a –rootdevice of /dev/sda1 (rather than /dev/sda) and hd00 kernel aki-4e7d9527, but I couldn’t get these to work. It might be because there is no partition table on the root disk, but without access to the interactive grub interface or more diagnostic output I can’t know for sure.

References

Amazon docs for use of specified kernels
Jonathan Hui’s blog post about creating CentOS images from scratch
Amazon docs on how to launch images from snapshots
Wim Coakearts’ blog post on using public-yum.oracle.com

Coming up: the actual Oracle database software install