The first videocast was quite popular — it has more than 300 views in couple weeks and considering no Hollywood stars were starring in that video and there were no nude scenes, I think it’s fair to say that this format was very much welcomed by the audience.
Today, I’m posting the follow up session on VIP’s (Virtual IP’s) with Oracle RAC. What I demonstrate today is a typical configuration mistake for a RAC databases created with Database Configuration Assistant (DBCA) and what’s the result of such omission.
It’s actually logged as a bug 4338578 on Metalink (thanks to Marcin Przepiorowski for reference). However, I believe it’s not a database issue but (1) misunderstanding how remote listener registration works, (2) documentation bug and (3) DBCA bug.
The init.ora parameter remote_listener defines the list of remote listeners that an instance should register with. The parameter local_listener sets the address for the local listener registration and if not specified, by default it’s hostname and the default port 1521. However, it turns out that the local_listener parameter is used not only for local registration but for something else as well…
Whenever an instance registers with a remote listener, it provides a connection descriptor that is used by the listener to redirect the incoming connection requests from client programs (this process is known as server-side connection load balancing). This connection descriptor for the redirection is created based on the local listener configuration for that instance — using the address or alias provided in the local_listener parameter. So what’s many DBA’s (and obviously developers of DBCA) are missing is that local_listener controls how (to what address and port) connections from the remote listeners are redirected.
As the video demonstrates, an out-of-the box DBCA-created database causes some connections to use host’s main IP’s (use netstat to confirm that on the client or on the servers). As a result, some of the connections (and it might be the majority of connections if you use services distributed across many nodes of a cluster) don’t use virtual IP’s.
The video explains it all — demonstrates how to detect this issue and shows how to fix this misconfiguration. I had to use a smaller embedded size to fit it on our blog and it’s still a bit wider than our blog format here. However, the video is in HD format now (noticed the improvement from my first attempt?) uploaded as 720p. To see it in full size — go directly to YouTube but you won’t see anybody sexy there :) and because my glasses broke yesterday, I don’t even look intelligent anymore.
I have added some annotation to it and if you notice something — you can annotate it yourself as well. I don’t know how it works when guests annotating the videos on YouTube so try it and let me know — let’s see how cool is that.
The next videocast will be on troubleshooting ORA-12545 in RAC environment and this is the natural continuation of today’s topic. This will probably be the last videocast about virtual IP’s for Oracle RAC.
As usual, all feedback is welcome here — this is what will keep these videocasts going.
22 Comments. Leave new
In term of audiance, I don’t think it is such a good idea to put content of blog on video. The reason is that most of us read blog Oracle at job and most societies use firewall. I can never access any post in this format.
@Bernard: OK. I see the problem. Let’s try to address it and not just discard that format. I’ve had few ideas… stay tuned…
Great topic, Alex. I’ve seen this behavior on must about every RAC environment I’ve looked at. (and fixed, accordingly) Another gotcha I had at my last job was that even though we had the database set up correctly, the app servers were behind a firewall and different DNS servers. the VIP addresses weren’t added to those DNS servers. Consequently, app servers would get sporadic TNS errors when they were re-directed to another node.
@Bradd:That’s exactly what the topic of the final session will be. ;-)
@Alex get out of my head :)
Alex,
Looking forward to the next session. My situation is similar to Brad’s but not quite the same. My app servers are able to access the database just fine. My problem is when I try to access the database over a different network than the one on which the VIP have been set up. When the server side load balancing kicks in I run into issues because the VIP address is not accessible on this other network.
Pradeep
Hi
I have RAC with 2 instances, I need set local_listener with the virtual ip.
Is necesary restart the instances after set local_listener?
Regard
Nery
Hi Gorvachev,
I saw your video about “Oracle RAC – VIP Configuration Mistakes”, is excellent.
I have RAC with 2 instances, I need set local_listener with the virtual ip, my questions are:
1.- Is necesary restart the instances after set local_listener?
2.- It is necessary to register the virtual ip in the DNS or only the name_server and ip local?
eg
Bus-a-vip 172.22.4.54
Bus-b-vip 172.22.4.55
@ Nery:
1. When you chance local_listener, you don’t need to restart anything. You might want to do ALETER SYSTEM REGISTER that triggers PMON to register with the listener or you can just wait a minute or so and PMON will do it anyway.
To verify that it worked, do “lsnrctl services” and see that REMOTE SERVER are registered with VIP’s.
2. You want to make sure that VIP’s are resolvable from *all* your clients. It can be resolved using local name resolutions (hosts file) or with DNS. The reason Oracle strongly suggests to define VIP’s in DNS is that any new application node or workstation can resolve VIP automatically without configuring local hosts file.
Another approach would be to use only real IP’s for remote/local listener registration and in client’s TNS descriptors. This would help to avoid the name resolution problem all together.
Well, it’s really the time for the last videocast now…
Thank you, Alex ;)
this’s a great job and helpful.
[…] 1 — Pythian Video: Oracle RAC — VIP Configuration Mistakes 2 — Pythian Video: Oracle RAC — Why VIPs? […]
What about dispatcher registration?
I have problem with it.
…
Service “iadbu” has 8 instance(s).
Instance “iadbu1”, status READY, has 3 handler(s) for this service…
Handler(s):
“DEDICATED” established:0 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=srv1p0-vip)(PORT=1521))
“DEDICATED” established:0 refused:0 state:ready
LOCAL SERVER
“D000” established:0 refused:0 current:3 max:1022 state:ready
DISPATCHER
(ADDRESS=(PROTOCOL=tcp)(HOST=srv1p0)(PORT=60556))
Instance “iadbu2”, status READY, has 2 handler(s) for this service…
Handler(s):
“D000” established:0 refused:0 current:2 max:1022 state:ready
DISPATCHER
(ADDRESS=(PROTOCOL=tcp)(HOST=srv1p1)(PORT=15729))
“DEDICATED” established:0 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=srv1p1-vip)(PORT=1521))
…
Oh, sorry, it is not a problem already. I have misconfigured “dispatchers” parameter.
@Yuri: Thanks for update. Good point about dispatchers indeed. Can you post here you dispatcher configuration?
Ok, but I have confused with dispatchers parameter configuration.
If I set it onto dispatchers='(protocol=TCP)’ and hosts file on client-side not contain resolving for public IP-adresses
then I receive ORA=12545 when attempt to connect with database.
If I set dispatchers='(protocol=TCP)(listener=)’ then connection is successful, but server-side load balancing is absent.
And if I set back dispatchers='(protocol=TCP)’ and add public IP-adresses into client-side hosts file then connection and server-side load balansing work fine.
Does last manner is a right way for configuration of connect through dispatcher?
Sorry, second choice is dispatchers='(protocol=TCP)(listener=LOCAL_LISTENER)’
Specify HOST parameter for DISPATCHERS
and DO NOT specify LISTENER.
See
https://odenysenko.wordpress.com/2009/09/24/rac-correct-dispatchers-configuration-no-more-ora-12545/
Oleksandr
I would think Oleksandr is correct but I would like to research what happens when VIP fails temporary and is moved to another node and back. In the early version of CRS, there was dependency of a database instance on the node’s VIP but it was removed in later releases. So if VIP is relocated to “non-native” node then instance is still up. Would dispatchers recover then VIP is moved back?
“Another approach would be to use only real IP’s for remote/local listener registration and in client’s TNS descriptors. This would help to avoid the name resolution problem all together”
Wouldn’t that defeat the purpose of using VIP’s and all the advantage which using VIP’s give?
Thanks a lot for clear description of the problem & ways to fix it. I am quite novice in Oracle, but was able to do all steps succesfully. Later, already knowing what to search, I was able to find direct reference to manual configuring the local_listener parameter in documentation (https://download.oracle.com/docs/cd/B28359_01/install.111/b28264/undrstnd.htm#RILIN006) but without your article it seems to be impossible. Thanks again.
Alex,
>>Would dispatchers recover then VIP is moved back?
I have tested and there is nothing unusual:
1) VIP failback – manual or auto, if configured
2) LISTENER will start
3) new DISPATCHER(D001) started
4) after some time PMON will register services with LISTENER,
including local DISPATCHERs
so infrastructure will function as before VIP relocation
I have tested it with 10.2.0.4,
but the same have to be in later releases.
Oleksandr
I really like this style of pythian. videos are much easy to learn.