Running exachk when no RAC databases installed, Health Check 11109: Database node IB MTU size should be set to 65520

Does every Database Machine runs RAC databases ? No … ! Lots of consolidated databases run in single instance configuration.

Are we able to use the same tools , as for preferred by Oracle RAC configuration ?

Yes…

Recently I was faced to run exachk tool on Database Machine with lots of single instance databases. The effect was, that I got lots of warnings …

It all was, because exachk was expecting clustered databases, so lots of checks returned negative results.

For those of you, who didn’t use exachk, it is a tool , that bases on best practices developed by Oracle specialists about configuring your Exadata and your databases.

Please follow document

Understanding, Navigating and Troubleshooting the Exachk Report (Doc ID 1947100.1)

To understand exachk reports and become familiar what it can do for you.

You can download the newest version from

Oracle Exadata Database Machine exachk or HealthCheck (Doc ID 1070954.1)

So .. comming back to the point, when you look on help which provides exachk -h command you will find

 -localonly
                Run exachk only on local node.

Yep ..! This is it ! 🙂 Now checks will be run on local node only, and you will stop getting negative results.

Now it is time to emphasize one of most important checks that was in Warning status

Health Check 11109: Database node IB MTU size should be set to 65520

Virtually, we also use ZFS storage, so that is why it was so important for us.

What does it mean ?

Exachk checks whether Maximum Transmition Unit is set optimally for Infiniband network. Because database nodes on Exadata machine, use TCP/IP over Infiniband protocol it is very important to set it larger then it was by default. It does not appleal to storage servers, which use RDS protocol. That all is because of fragmentation.

Another worth to mention fact is that infiniband HBAs in database nodes, should use connected mode. So how to check MTU size and connected mode ? It should be set on bonded bondib0 interface

[root@x2nodeadm02 ~]# ifconfig | grep -v "bondib0:1" | grep -A 3 bondib | grep MTU
UP BROADCAST RUNNING MASTER MULTICAST  MTU:31000  Metric:1

Where to change it ?

[root@x2nodeadm02 ~]# vi /etc/sysconfig/network-scripts/ifcfg-bondib0

Here we have MTU size

#### DO NOT REMOVE THESE LINES ####
#### %GENERATED BY CELL% ####
DEVICE=bondib0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
IPADDR=10.220.199.11
NETMASK=255.255.255.0
NETWORK=10.220.199.0
BROADCAST=10.220.199.255
BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100"
IPV6INIT=no
#MTU=31000
MTU=65520

To verify that the database server InfiniBand network is in “connnected” mode, use the following command on all database servers:

grep [a-z] /sys/class/net/ib*/mode

Output should be like

/sys/class/net/ib0/mode:connected
/sys/class/net/ib1/mode:connected

If it is not, than follow instructions below

vi /etc/ofed/openib.con

find line containing SET_IPOIB_CM and change it to “yes”

# Enable IPoIB Connected Mode 

SET_IPOIB_CM=yes

Unfortunately, it demands to reboot your db nodes. But it is another story..:)

This entry was posted in Nieokreślona, Oracle DBA. Bookmark the permalink.

Leave a Reply