News
Perth, Australia
+ (61) 417713124
prosolutions@gavinsoorma.com

Installing and configuring the Exadata Database Machine plug-in for EM Cloud Control

  • Posted by Gavin Soorma
  • On August 25, 2017
  • 0 Comments

The Oracle Exadata plug-in for OEM enables us to monitor as well as manage Exadata targets through Enterprise Manager Cloud Control 12c.

The features include:

  • Monitoring of the Exadata Database Machine as an Enterprise Manager target.
  • Monitoring of the all the components of the Exadata Database Machine target, including the Exadata Storage Cells
  • Execute CELLCLI commands and also configure and manage I/O Resource Management (IORM) feature.
  • Support SNMP notification for Exadata cells.
  • Support dashboard report creation from Enterprise Manager Cloud Control for Exadata Database Machine
  • Enhanced InfiniBand network fault detection and InfiniBand schematic port state reporting.

This note describes the procedure used to deploy the plug-in on an Exadata X5-2 one eighth rack.

The version of the Oracle Enterprise Manager agent is 12.1.0.5 and the Exadata plug-in version is 12.1.0.6.

At a high level, the Exadata plug-in installation and deployment consists of the following steps:

  • Create a database server ILOM Service Processor user
  • Verify ILOM Ipmitool version
  • Verify Infiniband Switch firmware
  • Obtain the credentials needed for Agent installation and Target Discovery
  • Install Enterprise Manager Agents on compute nodes
  • Setup SSH equivalence between the cellmonitor user and the agent (oraagent) user
  • Discover the Exadata Database Machine
  • Post-Discovery Configuration and Verification
  • Create Exadata Database Machine Dashboard

 

Create a Database Server ILOM Service Processor User

 

  • Log in to the database server ILOM Service Processor (exdb1dbadm01-ilom) as root

login as: root

Using keyboard-interactive authentication.

Password:

Oracle(R) Integrated Lights Out Manager

Version 3.2.4.52 r101649

Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved.

Hostname: exdb1dbadm01-ilom

 

  • Change to the users directory

-> cd /SP/users

/SP/users

  • Create the oemuser user and password

 -> create oemuser

Creating user…

Enter new password: ********

Enter new password again: ********

Created /SP/users/oemuser

 

  • Change to the new user’s directory and set the role

 -> cd oemuser

/SP/users/oemuser

-> set role=’cro’

Set ‘role’ to ‘cro’

 

Note: Repeat all the same steps for exdb1dbadm02-ilom

 

  • Test the ILOM user ID oemuser created above, by listing the last 10 system events:

 

Run the command below on both compute nodes:

ipmitool -I lanplus -H exdb1dbadm01-ilom -U oemuser -P welcome1 -L USER sel list last 10

 

[root@exdb1db01 ~]# ipmitool -I lanplus -H exdb1dbadm01-ilom -U oemuser -P welcome1 -L USER sel list last 10

 

35a | 01/18/2016 | 05:19:22 | System Firmware Progress | Hard-disk initialization | Asserted

35b | 01/18/2016 | 05:19:22 | System Firmware Progress | Option ROM initialization | Asserted

35c | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted

35d | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted

35e | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted

35f | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted

360 | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted

361 | 01/18/2016 | 05:20:14 | System Firmware Progress | Option ROM initialization | Asserted

362 | 01/18/2016 | 05:20:22 | System Firmware Progress | System boot initiated | Asserted

363 | 01/18/2016 | 05:20:22 | System Firmware Progress | System boot initiated | Asserted

 

 

[root@exdb1db02 ~]# ipmitool -I lanplus -H exdb1dbadm01-ilom -U oemuser -P welcome1 -L USER sel list last 10     

 

35a | 01/18/2016 | 05:19:22 | System Firmware Progress | Hard-disk initialization | Asserted

35b | 01/18/2016 | 05:19:22 | System Firmware Progress | Option ROM initializat ion | Asserted

35c | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted

35d | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted

35e | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted

35f | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted

360 | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted

361 | 01/18/2016 | 05:20:14 | System Firmware Progress | Option ROM initialization | Asserted

362 | 01/18/2016 | 05:20:22 | System Firmware Progress | System boot initiated| Asserted

  • 01/18/2016 | 05:20:22 | System Firmware Progress | System boot initiated| Asserted

 

Verify ILOM Ipmitool version

 

 Enterprise Manager requires a minimal Ipmitool software version of 1.8.10.3 for Oracle Linux.

Using DCLI, we can check the version on both compute nodes by running the ipmitool –V command.

[root@exdb1db01 ~]# dcli -g ~/dbs_group -l root ipmitool -V

exdb1db01: ipmitool version 1.8.12.3

exdb1db02: ipmitool version 1.8.12.3

 

The dbs_groupfile contains the list of compute node hostnames, one on each line.

Create the dbs_groupfile if it does not exist.

[root@exdb1db01 ~]# cat dbs_group

exdb1db01

exdb1db02

 

Verify Infiniband Switch firmware

 

Log on to the management interface for the InfiniBandSwitch (exdb1sw-iba01) as root.

login as: root

root@exda1sw-iba01’s password:

 

Run the command nm2version.

The minimum version should be 1.1.3-2

 

[root@exda1sw-iba01 ~]# nm2version

(Note: The command ‘nm2version’ is deprecated and will be removed from future

releases. Please use ‘version’ instead)

 

SUN DCS 36p version: 2.1.5-1

Build time: Oct  6 2014 10:35:15

SP board info:

Manufacturing Date: 2015.02.17

Serial Number: “NCDIV0061”

Hardware Revision: 0x0200

Firmware Revision: 0x0000

BIOS version: SUN0R100

BIOS date: 06/22/2010

 

Note: Perform the same check on the second Infiniband Switch.

 

 

Obtain the credentials needed for Agent installation and Target Discovery

List of credentials needed for Discovery

  • Agent host credentials: the username and password of the user running the agents on the compute nodes

Note – Agent owner should be in the same OS group as the CRS and Database software owners

  • Monitoring credentials:
  • Storage Cell Root password
  • nm2user password of each InfiniBand switch
  • ILOM username and password
  • InfiniBand Switch root credentials: Username and Password

 

 

Install Enterprise Manager Agents on compute nodes

 

From the Setup menu select Add Target followed by  Add Targets Manually

 

 

Select option “Add Host Targets

Click Add Host ….

 

 

Click Add

 

 

Enter the names of both the compute nodes as well as the directory path location on the compute nodes for installing the 12.1.0.5 Agent software.

The agent software base directory is /opt/oraagent

Note – In this example we are adding one compute node at a time, but we can do the agent installation on both the compute nodes in one step.

 

 

Provide the credentials of the OS user we are going to use for the agent deployment (oraagent)

Review and click on Deploy Agent

 

 

 

 

 

 

Note: After the agent deployment is complete, execute the root.sh script on each compute node

After the agent has been installed and deployed on both compute nodes, we should be able to see both agents up and running as shown below:

 

 

Setup SSH equivalence between the cellmonitor user and the agent (oraagent) user

 

 

On each of the compute nodes we will run the command ssh-keygen –t dsa as the agent user oraagent  to generate the agent’s SSH public key.

Then copy that file to each of the Storage Cells and add the contents of  the file id_dsa.pub to the authorized_keys file belonging to the cellmonitor user.

Basically the oraagent user needs to be able to connect to each of the Storage Cells as the cellmonitor user and execute the command “cellcli –e list cell attributes msstatus”.

Otherwise after the Exadata Database Machine discovery is completed, in the schematic diagram we will see the Storage Cells displaying with  a red colour down status when actually all the CELLSRV processes are running.

 

 

 

 


Verify if the cellmonitor user can SSH to one of the Storage Cells and no password is being prompted

 

 

Discover the Exadata Database Machine

  Select Add Targets Manually  from the Setup and Add Target menu

 

 

Select the option “Add Targets Using Guided Process” and in the Target Types drop-down list select “Oracle Exadata Database Machine

Click on the button Add Using Guided Process

 

 

Select the option to Discover a new Database Machine and its hardware components and click on Discover Targets

 

 

In the Discovery Inputs screen specify the agent that you would want to use for discovery. You can choose any one of the compute nodes but it is recommended to use the first node of the cluster.

Provide the credentials for the agent.

 


 

Click on Set Credential and enter the details of the agent user account on the compute node

 

 

Select the schematic file as databasemachine.xml and check the File Location corresponds to the /opt/oracle.SupportTools/onecommand directory

 

 

For the Infiniband network discovery, we create a new credential. The credentials of the nm2user are used here.

 

 

 

A prerequisite check is now performed to verify that the databasemachine.xml file is present and is readable by the OEM agent.

 

 

Verify the names and IP addresses of the Database Machine components which have been discovered.

 

 

For each of the Database Machine components, a monitoring agent is assigned as well as a backup agent. We can either use the in-built random assignment or manually assign primary monitoring and backup agents for each component.

 

 

 

We now provide the credentials of the OEM agents running on each of the compute nodes. Test the connection as well.

 

 

In the Monitoring Credentials screen, we need to provide the root passwords for the Storage Servers (Cells), nm2user for the Infiniband Switches, and the oemuser for the ILOMs.

 

 

 

 

 

 

This screen is for the configuration of SNMP subscriptions for the OEM agents to receive SNMP traps.

For the SNMP Community String provide the value ‘Public’ and provide the root passwords for both the Infiniband Switches.

 

 

 

 

Review and then click on Submit.

Then review the summary of the Target Creation .

Ensure each component has a successful Discovery Status.

 

 

After the Database Machine discovery is complete, we will see a new menu option called “Exadata” added to the Targets menu in OEM 12c Cloud Control

Launch DB Machine Home

Click on the Exadata target and that will take us to the Oracle Exadata Schematic page

 

 

 

Post-Discovery Configuration and Verification

 

  • Set Up SNMP Trap Forwarding on Compute Node

 

We need to set up  SNMP trap forwarding on the compute node where the Enterprise Manager Agent is running so that the snmptrapd daemon can receive traps using port 162 and forward the trap to the port the OEM agent is listening on.

In this case we have verified from the emd.properties (property EMD_URL) file located in the $AGENT_INST/sysman/config directory that the value of the port is 3872

Confirm that the agent receivelet is listening on this port over UDP

Then stop the SNMP trap daemon (in this case it is already stoped – otherwise issue command service snmptrapd stop)

 

 

Add the following entry to the /etc/snmp/snmptrapd.conf configuration file:

 

authcommunity log,execute,net <community string>

forward default udp:localhost:3872

 

 

Note: Do this on both compute nodes

 

 

 

Edit the /etc/hosts.allow file to add the following line:

snmptrapd : ALL

 

 

 

Start the SNMP trap daemon and ensure via chkconfig command that the daemon will start on a node reboot

 

 

 

 

 

  • Verify SNMP Configuration for Storage Cells

 

For Exadata plug-in Release 12.1.0.2.0 and later, SNMP trap setup for Exadata Storage Cells is done automatically during the discovery process

 

Verify

 

[root@exdb1celadm01 ~]# cellcli -e list cell attributes snmpSubscriber

((host=exdb1dbadm02.rac.com.au,port=1830,community=public),(host=exdb1dbadm01.rac.com.au,port=1830,community=public),(host=exdb1db01.rac.com.au,port=3872,community=public),(host=exdb1db02.rac.com.au,port=3872,community=public),(host=61.88.37.93,port=162,community=public,type=ASR))

[root@exdb1celadm01 ~]# cellcli -e list cell attributes notificationMethod

mail,snmp

[root@exdb1celadm01 ~]# cellcli -e list cell attributes notificationPolicy

critical,warning,clear

 

  • Configure and Verify SNMP for InfiniBand Switch Targets

 

For Exadata plug-in Release 12.1.0.2.0 and later during the discovery process, we provide the necessary root credentials to set up SNMP trap for the InfiniBand Switch.

SNMP trap setup for Infiniband Switches is done automatically during the discovery process.

We can verify this by connecting via ILOM web interface to the Infiniband Switch.

 

 

 

 

Click Configuration, then System Management Access, and finally SNMP


Ensure the following values are set:

 

State=Enabled

Port=161

Protocols=v1,v2c,v3

 

 

Verify the InfiniBand Switch SNMP configuration for Enterprise Manager monitoring

From one of the compute nodes execute the command:

snmpget -v 1 -c public exdb1sw-iba01 1.3.6.1.4.1.42.2.70.101.1.1.9.1.1.5

 

[root@exdb1db01 onecommand]# snmpget -v 1 -c public exdb1sw-iba01 1.3.6.1.4.1.42.2.70.101.1.1.9.1.1.5

SNMPv2-SMI::enterprises.42.2.70.101.1.1.9.1.1.5 = INTEGER: 1

 

Note: We should not get a Timeout message. If so, it means that the SNMP configuration for OEM monitoring is not complete.

 

Create Exadata Database Machine Dashboard

 

We create the Database Machine dashboard to give us a single pane-of-glass view of the performance and usage metrics of the Exadata Database Machine system as well as its components.

 

Navigate to “Enterprise” -> “Job” -> “Activity

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

0 Comments

Leave Reply

Your email address will not be published. Required fields are marked *