I’ve recently moved to Ubuntu Linux, and this post describes my attempt to play around with Oracle ASM on Ubuntu. For this demonstration, I used Oracle 11.1.0.6 on Gutsy Gibbon. I hope it will be useful to somebody out there.
Important Notice: What I describe below is among the worst thing you can ever do with ASM. You can use it to play around but never use it with anything other than test data. If you lose something because of me, you’ll be the only one to blame !
Question #1: How do you simulate a disk from a file?
If you have a free partition or disk to be used as an ASM disk, just skip this step. If you don’t, you can create a file with the dd command and create a device that actually loops to the file with the losetup command.
Let’s assume you’ve created a directory named /asmdisks (and you have write access to it). Run the command below to create a file named disk1 that is 3GB in size:
$ dd if=/dev/zero of=/asmdisks/disk1 bs=1024k count=3072 3072+0 records in 3072+0 records out 3221225472 bytes (3.2 GB) copied, 80.9113 seconds, 39.8 MB/s
Once you’ve created the file, map it to a device named loopN in /dev. You can list the used loop devices with the losetup -a command.
Once you’ve made sure the one you plan to used is free, e.g. /dev/loop1, you can map the device to the file with the following commands (you have to be root) :
# losetup /dev/loop1 /asmdisks/disk1 # losetup -a /dev/loop1: [0802]:7438407 (/asmdisks/disk1)
Question #2: How do you create an interface to the disk that is usable by ASM?
Actually, the preferred way to access a disk from ASM on Linux is ASMLib. To be fair, I must say I didn’t even give it a try. Of course, I’d be more than interested if anybody could make it work on Ubuntu. You’ll have to recompile the source code and I doubt I’ll be able to do it myself.
So you may think, why not use Raw Devices? Because you don’t need raw devices. You can just map the ASM disks to your disk/partition or loop devices.
So the only thing you need to do to start is change the ownership of the device so that ASM can access it in read/write mode as oracle:
# chown oracle:dba /dev/loop1 # ls -l /dev/loop1 brw-rw---- 1 oracle dba 7, 1 2008-02-06 23:32 /dev/loop1
If you have a real disk partition (e.g. /dev/sdb1):
# chown oracle:dba /dev/sdb1 # ls -l /dev/sdb1
Note 1:
I couldn’t manage to demonstrate 11g’s very cool offline ASM without ASMLib or raw devices. Nonetheless, I’ll describe how to set up raw devices on Gutsy later in this post.
Note 2:
If you want losetup run and ownership set up automatically at boot time, you’ll have to define the correct rules in /etc/udev/rules.d.
Question #3: How do you start up the Cluster Synchronization Service Daemon?
You may not have paid attention to it before, but ASM relies on the CSS Daemon. Who cares? I do, because CSSD relies on /etc/inittab to startup and there is no such a file in Ubuntu. Nevermind, run the setup script as root :
# /u01/app/oracle/product/11.1.0/db_1/bin/localconfig reset Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Configuration for local CSS has been initialized Adding to inittab Startup will be queued to init within 30 seconds. Checking the status of new Oracle init process... Expecting the CRS daemons to be up within 600 seconds.
Once you get the message about the 600 seconds, hit CRTL+C to stop the script, and then run the command below to actually run the CSS Daemon:
# nohup /etc/init.d/init.cssd run >/dev/null 2>&1 </dev/null &
Note 3: If you want the CSSD daemon to startup automatically, you’ll have to create a service in Ubuntu.
Question #4: How do you create the ASM instance, add a diskgroup, etc?
This is not really the subject of this post, but assuming you’ve already installed Oracle 11g accordingly to Augusto’s Installing Oracle 11g on Ubuntu Linux 7.10 (Gutsy Gibbon) and you’ve setup the PATH, ORACLE_HOME and ORACLE_BASE variables, you can do everything in one command:
$ dbca -silent -configureASM \
-asmSysPassword change_on_install \
-diskString "/dev/loop*" \
-diskList /dev/loop1 \
-diskGroupName DG1 \
-redundancy EXTERNAL
Once that is done, you can connect to the ASM instance with SQL*Plus:
$ source oraenv
ORACLE_SID = [oracle] ?+ASM
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 is /u01/app/oracle
$ sqlplus / as sysdba
SQL> select NAME, TOTAL_MB
from v$asm_diskgroup;
NAME TOTAL_MB
---- --------
DG1 3072
Or you can use ASMCMD as below:
$ source oraenv ORACLE_SID = [oracle] ?+ASM The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 is /u01/app/oracle $ asmcmd ASMCMD> ls DG1/
Question #5: How do you create a database that uses ASM?
I cannot resist providing the syntax for 11g (totalMemory will have to be replaced by memoryPercentage in 10g):
$ dbca -silent -createDatabase \
-templateName General_Purpose.dbc \
-gdbName BLUJ \
-sysPassword change_on_install \
-systemPassword manager \
-emConfiguration NONE \
-storageType ASM \
-asmSysPassword change_on_install \
-diskGroupName DG1 \
-characterSet WE8ISO8859P15 \
-totalMemory 250
ulimit: 1: Illegal option -u
ulimit: 1: Illegal option -u
ulimit: 1: Illegal option -u
ulimit: 1: Illegal option -u
Copying database files
1% complete
3% complete
37% complete
Creating and starting Oracle instance
40% complete
45% complete
50% complete
55% complete
56% complete
60% complete
62% complete
Completing Database Creation
66% complete
70% complete
73% complete
77% complete
88% complete
100% complete
Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/BLUJ/BLUJ.log" for further details.
Question #2 (Revisited): How do you create an interface to the disk that is usable by ASM ?
For 11g ASM offline tests, raw devices can actually be useful. And, despite the missing pieces, creating a raw device that maps the loop device (or real disk/partition) is straightforward. The first step consists in filling the holes in Ubuntu, i.e. in creating some character files that are not created by default on Gutsy Gibbon (rawctl, raw0 . . . raw1).
Run the set of commands below as root:
# mknod /dev/rawctl c 162 0 # mknod /dev/raw/raw0 c 162 1 # mknod /dev/raw/raw1 c 162 2 # ln -s /dev/rawctl /dev/raw/rawctl
Once the missing pieces rebuilt, mapping a raw devices to your disk or loop interface is as easy as:
# raw /dev/raw/raw1 /dev/loop1
Or if you use a partition named /dev/sdb1:
# raw /dev/raw/raw1 /dev/sdb1
You can display the result of your settings with the command below.
# raw -qa /dev/raw/raw1: bound to major 7, minor 0
The raw devices won’t change anything but the path to the device when you’ll go from Question #3 to Question #5.
Note 4: If you want the raw devices to be setup automatically at boot time, you’ll have to define the correct rules in /etc/udev/rules.d.
Conclusion
What I really like about Oracle is that once you’ve done something, you have ten different ways to go. I hope this post will enable you to explore some 11g’s new features on your favorite operating system.
15 Comments. Leave new
[…] Pythian’s Grégory Guillou contributes a howto on setting up Oracle ASM on Ubuntu Gutsy Gibbon. […]
Note: In “Question 2 (revisited)” about raw devices, you may get the following error when running the command shown “raw /dev/raw/raw1 /dev/sdb1”:’cannot open master raw ‘/dev/rawctl’ (no such device)’. The fix for this is to first enter the command (as root): “modprobe raw”. Then “raw” will work.
Hello Gregory,
Thank you is hardly adequate to express my profound gratitude for your pioneering effort in documenting setup of ASM on ubuntu Gutsy Gibbon. Here is my contribution and extension of your great work: It would seem that I have developed a way to get a clean and “normal” start of the Cluster Synchronization Service (CSS) Daemon on our Gutsy Gibbon. This was developed by coordinating many diverse reference materials on various aspects of teh problem from the web at many many diverse sources and a bit of luck was involved too. Give this recipe a try and let me know if it works for you also.
Gil Standen
Procedure to Reliably spawn ocssd.bin during bootup:
I am running Ubuntu 7.10 Gutsy Gibbon on vmware 1.0.3 on a Dell Latitude D820 Laptop. I run it with between 512-788Mb which is sufficent to support my linux and my ASM instance. Here are the steps which hopefully folks will try out to get a textbook startup of ASM CSS on Gutsy Gibbon so that I can get some feedback if this is working generally for folks. Also, this can probably be simplified and improved I would imagine. Here it is:
Step 1.
Go to /etc and edit the “inittab” file there and change the part
“h1:35:respawn…” to
“h1:23:respawn…”
This change is needed because active runlevels are 23 (the Debian/Ubuntu defaults) instead of 35 (the Red Hat and Suse defaults). For more information on this change, see these references:
https://help.ubuntu.com/community/Oracle10gBreezy
https://translate.google.com/translate?u=http%3A%2F%2Fblog.chinaunix.net%2Fu%2F7667%2Fshowart_146276.html&langpair=zh%7Cen&hl=en&ie=UTF8
[ Note 1, I actually first got hip to this at the second link (from a China source) so I’d like to say thanks to “blue_stone” who says of himself at his website: “I blue_stone, a drift of the oracle engineers in Beijing”. Thanks, blue_stone.]
[Note 2: *A useful side note which I humbly suggest at this point, for people struggling with remedying the issues with Oracle on Ubuntu: Debian was the distribution of Linux from which Ubuntu originally was forked. When looking for solutions to Ubuntu problems, also google on “Debian” because many of the old Debian fixes for Oracle will work for Ubuntu also.]
[Note 3: There is also a copy of the inittab in the directory
$ ORACLE_HOME/css/admin/inittab_local where you *may* or *may not* want to make this change as well (changing “h1:35:respawn…” to “h1:23:respawn…”] but I think that the change has to be made here as well because I think that this may be used as a template when oracle creates /etc/inittab so it has to have the correction to “”23” as well so that the templating will be correct if “localconfig reset” is ever used (which I *guess* overwrites /etc/inittab).
Step 2
When the Oracle installer runs on Ubuntu, it creates the symlinks in the /etc/rcX.d directories; however, it omits the symlink for init.cssd in rc2.d (again, the runlevels issue due to Debian/Ubuntu vs. RedHat/Suse runlevels). Since the default runlevel for Ubuntu is level 2, we need that symlink in /etc/rc2.d. Create it as follows:
ln -s /etc/init.d/init.cssd /etc/rc2.d/S96init.cssd
[note, you should also create the following link at this time if it does not exist already:
ln -s /etc/init.d/init.cssd /etc/rc2.d/K96init.cssd
so that css will exit cleanly at reboot or shutdown time].
This information about this was also found at the link:
https://help.ubuntu.com/community/Oracle10gBreezy
Note however, that when I attempted to implement the changes described there, I found that this command did not work as described:
update-rc.d init.cssd defaults 96
Basically when I did what was described there, (i.e. the “rm” commands for the symlinks in /etc/rc[2,3,5].d worked great, but the update-rc.d didn’t create anything to replace them. However, I believe that only the link described above in /etc/rc2.d is needed. But my advice would be to skip the rm’s that are described at that reference, and just create the needed link in /etc/rc2.d as described above.
[ Note: You can also get a listing of what Oracle puts out in the /etc/rcX.d directories by default when installing from this link:
https://translate.google.com/translate?u=http%3A%2F%2Fwww.sql.ru%2Fforum%2Factualthread.aspx&langpair=ru%7Cen&hl=en&ie=UTF8
it looks to me like Oracle does install the / etc/rc3.d/S01init.cssd -> .. /init.d/init.cssd link and that it only forgets the startup link described above.]
Step 3
As noted in the original article Gregory pointed out that Ubuntu by default in Gutsy Gibbon does not use inittab anymore. This has been replaced by something called “Upstart” which is described at the following webpage as “…an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot…”.
Now my install of Ubuntu 7.10 apparently ended up with upstart configured and running when I installed Ubuntu. I installed my Ubuntu according to the other Pythian article “Installing Oracle 11g on Ubuntu Linux 7.10 (Gutsy Gibbon) November 6th, 2007 – by Augusto Bott” so I’m guessing if you follow Augusto’s article for your Oracle install and then move on to Gregory’s article for your ASM configuration on Gutsy, you will probably have “Upstart” running as well and so should not have to take any action to make use of it.
You can test to see if Upstart is running ok by creating this file /etc/event.d/event.d.echo and then rebooting your Ubuntu:
#—————————————
# this is a test of upstart
description “Upstart Echo”
start on startup
exec echo “Special Startups for Oracle/ASM/Cluster Support…”
console output
#———————————-
Then reboot your Ubuntu 7.10 and see if you see that echo’d message appear during startup. If it appears, and if you created the “event.d.echo” file correctly, that phase will jump out at you during bootup.
If upstart is working, then we can move on to the final piece of this method to “startup ASM cleanly on Gutsy” procedure. If it isn’t then go to this link to find out how to get upstart installed and configured:
https://upstart.ubuntu.com/getting-started.html
Once upstart is working, we put in the last piece of the puzzle. Create a file /etc/event.d/event.d.cssd and put this in the file:
#————————————-
# This service maintains the CSS Daemon for ASM
# from the point the system is started until it is shutdown.
description “CSS Daemon”
start on startup
stop on shutdown
script
sleep 45
nohup /etc/init.d/init.cssd run >/dev/null 2>&1 </dev/null &
end script
#————————————————-
[ Note1: I don’t know if the sleep 45 is needed or not or if the timing matters. I haven’t had a chance to play around with the setup to see how changes affect the startup of ocssd.bin at boot time. This setup is working perfectly for me as is. ]
[ Note2: You’ll notice this command comes right out of Gregory’s procedure here for starting ASM from Gregory’s “Question #3” (see original article) which is where I got this command from to plug in to this upstart script ]
[ Note3: General note about init.cssd vs. ocssd.bin from Metalink: In a normal environment, init spawns init.cssd, which in turn spawns OCSSD as a child. Also, it’s worth noting the following additional info from Metalink: In a non-RAC environment, the ocssd.bin process is used for communication between a database instance and the ASM (Automatic Storage Management) instance. Even if you are not using ASM this process will be run out of the inittab. These two nuggets of background info come from Metalink docs 311647.1 and one other doc which I can’t recall at the moment, but it’s just general background info which helped me understand the difference between init.cssd and ocssd.bin ]
So now, if your Gutsy setup digests these changes the same way mine did, you end up with a Gutsy that will reliably spawn ocssd.bin first time every time at bootup. I rebooted 5 or 6 times to test to be sure it was working always, and it was. I hope I didn’t leave out any steps. The script in event.d.cssd is needed.
Also, I’d like to say hi to Babette Turner-Underwood who I met while attending AUSOUG 2007 at Perth and Melbourne, at which conference we both had the honor of giving presentations. I’d also like to say that Pythian just keeps coming up with one hit after another pushing the envelope of what is possible with Oracle!! I am so grateful to you that words cannot express for this really superb work you have done showing how to use Ubuntu Gutsy with Oracle!! THANKS!
PS I’ve got some more goodies I’ve worked out with a starting point from Gregory’s article so if anyone finds this post useful, let me know and I’ll cough up some more discoveries of ASM on Ubuntu, including how to automate setup of your raw devices and how to use dbca to create your ASM instance, and in particular how to fix the problem which pops up during dbca creation of ASM regarding “MEMORY_TARGET” error ORA-00845 MEMORY_TARGET not supported on this system. There is a fix for that as well!
Gil Standen
[email protected]
Gil,
Awesome comment. Actually up to now I was re-running the following command after rebooting :
/u01/app/oracle/product/11.1.0/db_1/bin/localconfig reset
I’ll set that up asap.
Regarding the MEMORY_TARGET parameter, actually it works fine on Ubuntu. The problem I guess comes from the fact it check the available memory before using it. The message says the feature is not available but actually, it will be… If you free your memory.
Thank you very much.
Gregory, thank YOU very much. Without your great article I’d have been completely lost!
[…] tested the whole process with Oracle 11.1.0.6 on Ubuntu Gutsy Gibbon, based on a previous post I made on this blog. In order to copy the ASM disks, I’ve used the dd command, but it should not make any […]
Ok sportsfans, I think we have a first. I’m pleased to announce I’ve got a running Oracle 11g 11.1.0.6.0 2-node RAC cluster running on Ubuntu 7.10 Linux. Installed 100% clean, runs beautifully. Remember, you heard it first here! My starting point was the work by Augusto Bott and Gregory Guillou and then I just kept at it until the secret was found. Can a stable 11g RAC cluster be run on Ubuntu 7.10? Yes, it can!
Can a stable 11g RAC cluster be run on Ubuntu 7.10? Yes, it can!
It depends on the definition of stable.
Anyway, any sensible DBA will *not* run Oracle RAC on Ubuntu in *production*. For test and development — maybe. On the other hand, why would one want to make your test and development environments to look different?
IMHO, it’s only usable for some personal tests and research. I personally installed it earlier this year only because Ubuntu consumed less resources so that I could run more virtual machine and have 3 nodes RAC demo.
Kudos! I’m impressed and I’m looking forward to learning how you’ve done that.
The main trick (although there is of course alot more to it than this) is to install Ubuntu 7.10, then go over to a CentOS 5 (or similar Red Hat) and create a cpio archive of the CentOS/RedHat kernel, then install that kernel on the Ubuntu system and then run the Ubuntu under the CentOS/RedHat kernel – this will cause all kinds of key markers (such as /proc/version) to display CentOS/RedHat markers/versions, which goes a long long way to make the Oracle Installer feel more “comfortable” with your Ubuntu distro – effectively allows Ubuntu to “masquerade” as RedHat. Then you do the install (you have to manually install some packages – which involves converting them from RPM to .deb packages with alien) and once you have all the packages it wants, you get an “almost” clean “pre-install” (and also runcluvfy.sh will now work -with one little tweak to a configurationfile). The install then runs very cleanly. Once the install is done, you take away the RedHat kernel, and go back to booting the 2.6.22-14 Ubuntu kernel. It’s not quite that easy because you have to put some rawdevice startups in /etc/event.d and (in the rudimentary configuration I am running) you have to run some things from rc.local – I’m looking to improve and make this more elegant. But the sum total is that at the end of it all it appears to be a stable system. crsctl check crs comes back clean, ASM comes up normally, etc. I’m letting it run for a few days to see if it still has any problems. One problem that Ubuntu seems to have (at least in vm) is that the linux clocking isn’t as compatible with the host as is CentOS (which keeps very good inter-node time synch) wheras the ubuntu node clocks drift apart quite quickly – even with all the tweaks and even with ntp running. I haven’t found a good fix for this yet. I think this clock problem can eventually bring the RAC nodes down but I’m not sure how tolerant the RAC is of widely out-of-synch linux system clocks.
I got the idea for the kernel bait-and-switch trick from Ingvar Hagelund. See: https://users.linpro.no/ingvar/rpm.html
This successful RAC 11g install on Ubuntu could not have been accomplished without the help of this webpage, and I emailed Ingvar and he helped step me through some problems I ran into embedding the Red Hat kernel in the Ubuntu system.
I may have to retract the comment about the time synch on Ubuntu – it seems to be ok now. The nodes have been up several hours and they are within about 1 second of each other – within the time it takes to actually check them. But I’d still like to learn more inter-node RAC clock synch tips and tricks since clocking has been one of the nagging problems I’ve run into with RAC installs and deployments.
Greg/Gil,
you guys are amazing.
[…] https://www.pythian.com/news/810/howto-set-up-oracle-asm-on-ubuntu-gutsy-gibbon/ May 27, 2010 11:36 am David Waters This question may be better suited for https://serverfault.com/ which is a site dedicated to systems administration May 27, 2010 12:48 pm Gary Generally Oracle supports their software on Red Hat (and it’s clones). May 27, 2010 11:36 am […]
This is pretty cool and worked on Ubuntu 10.10. However, for Question #3, to startup CSS, when I ran localconfig add/reset, it failed since rc.* directories didn’t exist in /etc. I created all 7 of them /etc/rc.d/rc.[0-6] and ran localconfig and it created local OCR and installed init.cssd. Then I manually restarted CSS using the command given in Question #3. Thanks for your post!
Septemeber 2014: To install Oracle ASM on Ubuntu, use Linux Containers (LXC). LXC has made deploying Oracle database products a snap. I wrote up my steps here: https://sites.google.com/site/nandydandyoracle/technologies/lxc/oracle-lxc-vlc