Does every Database Machine runs RAC databases ? No … ! Lots of consolidated databases run in single instance configuration.
Are we able to use the same tools , as for preferred by Oracle RAC configuration ?
Yes…
Recently I was faced to run exachk tool on Database Machine with lots of single instance databases. The effect was, that I got lots of warnings …
It all was, because exachk was expecting clustered databases, so lots of checks returned negative results.
For those of you, who didn’t use exachk, it is a tool , that bases on best practices developed by Oracle specialists about configuring your Exadata and your databases.
Please follow document
Understanding, Navigating and Troubleshooting the Exachk Report (Doc ID 1947100.1)
To understand exachk reports and become familiar what it can do for you.
You can download the newest version from
Oracle Exadata Database Machine exachk or HealthCheck (Doc ID 1070954.1)
So .. comming back to the point, when you look on help which provides exachk -h command you will find
-localonly Run exachk only on local node.
Yep ..! This is it ! 🙂 Now checks will be run on local node only, and you will stop getting negative results.
Now it is time to emphasize one of most important checks that was in Warning status
Health Check 11109: Database node IB MTU size should be set to 65520
Virtually, we also use ZFS storage, so that is why it was so important for us.
What does it mean ?
Exachk checks whether Maximum Transmition Unit is set optimally for Infiniband network. Because database nodes on Exadata machine, use TCP/IP over Infiniband protocol it is very important to set it larger then it was by default. It does not appleal to storage servers, which use RDS protocol. That all is because of fragmentation.
Another worth to mention fact is that infiniband HBAs in database nodes, should use connected mode. So how to check MTU size and connected mode ? It should be set on bonded bondib0 interface
[root@x2nodeadm02 ~]# ifconfig | grep -v "bondib0:1" | grep -A 3 bondib | grep MTU UP BROADCAST RUNNING MASTER MULTICAST MTU:31000 Metric:1
Where to change it ?
[root@x2nodeadm02 ~]# vi /etc/sysconfig/network-scripts/ifcfg-bondib0 Here we have MTU size #### DO NOT REMOVE THESE LINES #### #### %GENERATED BY CELL% #### DEVICE=bondib0 USERCTL=no BOOTPROTO=none ONBOOT=yes IPADDR=10.220.199.11 NETMASK=255.255.255.0 NETWORK=10.220.199.0 BROADCAST=10.220.199.255 BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100" IPV6INIT=no #MTU=31000 MTU=65520
To verify that the database server InfiniBand network is in “connnected” mode, use the following command on all database servers:
grep [a-z] /sys/class/net/ib*/mode
Output should be like
/sys/class/net/ib0/mode:connected /sys/class/net/ib1/mode:connected
If it is not, than follow instructions below
vi /etc/ofed/openib.con
find line containing SET_IPOIB_CM and change it to “yes”
# Enable IPoIB Connected Mode SET_IPOIB_CM=yes
Unfortunately, it demands to reboot your db nodes. But it is another story..:)
