Upgrading 11gR2 RAC Grid Infrastructure to 11.2.0.3

 

Here are some notes I have prepared when I did an upgrade of a 11GR2 Two-Node RAC Grid Infrastructure from 11.2.0.2 to 11.2.0.3 on a Linux 64 bit test environment.

 

The 11.2.0.3 is an out of place upgrade. So we need to install the 11.2.0.3 software in seperate location from the existing 11.2.0.x software.

 

The 11.2.0.3 software is not directly available from OTN. We need to download 11g Release 2 (11.2.0.3) Patch Set 2 (Patch 10404530) from the Metalink MOS site.

The patch 10404530 comes with 7 seperate zip files. We do not need all the zip files and can just download a sub-set.

For Grid Infrastructure 11.2.0.3, we need to use p10404530_112030_platform_3of7.zip

For Database we need to download p10404530_112030_platform_1of7.zip & p10404530_112030_platform_2of7.zip

 

You need to use the most latest version of  OPatch. We need OPatch utility version 11.2.0.1.5 or later to apply this patch. Download latest version of patch 6880880 appropriate to your platform.

I installed OPatch version 11.2.0.3 on BOTH nodes of the cluster.

We need to install a prerequisite patch 12539000 and follow the README.txt file specifically.

 

To run the 12539000 patch, we need to take quite a few points into consideration.

 

We have to run the emocmrsp script to create the OCM (Oracle Configuration Manager) response file. This file is located under the Grid Oracle Home in the following location:

<GRID ORACLE_HOME>/OPatch/ocm/bin/emocmrsp

When we run the emocmrsp file, we do not need to enter our email address details and can just specify Y when asked if we wish to remain uninformed of  any patch updates related to security issues.

We have to specify the location of the OCM (Oracle Configuration Manager) response file. We need to specify the FULL PATH of the ocm.rsp file or esle the OPatch will fail.

For example – /u01/app/11.2.0/grid/OPatch/ocm.rsp

The directory where we unzip the patch 12539000 needs to EMPTY. It should not contain any other files or else OPatch will fail with some real misleading errors.

To run the patch 12539000, we need to do the following on BOTH nodes of the RAC cluster assuming the patch has been unzipped in the directory – /u01/stage/GI_11203_PATCH

cd $GRID_HOME/OPatch

./opatch auto /u01/stage/GI_11203_PATCH

Note – this should patch the Database Oracle Homes as well.

After the patch installation , run the opatch lsinvemtory command to confirm that the patch 12539000 has been applied on both database as well as Grid Infrastructure Oracle Homes.

 

For example:

[grid@kens-racnode1 OPatch]$ ./opatch lsinventory

Oracle Interim Patch Installer version 11.2.0.3.0

Copyright (c) 2012, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/11.2.0/grid

Central Inventory : /u01/app/oraInventory

from           : /u01/app/11.2.0/grid/oraInst.loc

OPatch version    : 11.2.0.3.0

OUI version       : 11.2.0.2.0

Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2012-06-29_06-05-49AM_1.log

Lsinventory Output file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2012-06-29_06-05-49AM.txt

——————————————————————————–

Installed Top-level Products (1):

Oracle Grid Infrastructure                                           11.2.0.2.0

There are 1 products installed in this Oracle Home.

Interim patches (1) :

 

Patch  12539000     : applied on Fri Jun 29 06:02:12 EDT 2012

Unique Patch ID:  13976979

Created on 28 Jul 2011, 12:37:42 hrs PST8PDT

Bugs fixed:

12539000

 

Rac system comprising of multiple nodes

Local node = kens-racnode1

Remote node = kens-racnode2

——————————————————————————–

OPatch succeeded.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

[root@kens-racnode1 12539000]# /u01/app/11.2.0.3/grid/rootupgrade.sh

Performing root user operation for Oracle 11g

 

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME=  /u01/app/11.2.0.3/grid

 

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of “dbhome” have not changed. No need to overwrite.

The contents of “oraenv” have not changed. No need to overwrite.

The contents of “coraenv” have not changed. No need to overwrite.

 

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params

Creating trace directory

User ignored Prerequisites during installation

 

ASM upgrade has started on first node.

 

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘kens-racnode1’

CRS-2673: Attempting to stop ‘ora.crsd’ on ‘kens-racnode1’

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘kens-racnode1’

CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘kens-racnode1’

CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘kens-racnode1’

CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘kens-racnode1’

CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘kens-racnode1’

CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘kens-racnode1’

CRS-2673: Attempting to stop ‘ora.cvu’ on ‘kens-racnode1’

CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘kens-racnode1’ succeeded

CRS-2673: Attempting to stop ‘ora.kens-racnode1.vip’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.kens-racnode1.vip’ on ‘kens-racnode1’ succeeded

CRS-2672: Attempting to start ‘ora.kens-racnode1.vip’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.LISTENER_SCAN2.lsnr’ on ‘kens-racnode1’ succeeded

CRS-2673: Attempting to stop ‘ora.scan2.vip’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.scan2.vip’ on ‘kens-racnode1’ succeeded

CRS-2672: Attempting to start ‘ora.scan2.vip’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.LISTENER_SCAN3.lsnr’ on ‘kens-racnode1’ succeeded

CRS-2673: Attempting to stop ‘ora.scan3.vip’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.scan3.vip’ on ‘kens-racnode1’ succeeded

CRS-2672: Attempting to start ‘ora.scan3.vip’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.cvu’ on ‘kens-racnode1’ succeeded

CRS-2672: Attempting to start ‘ora.cvu’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.registry.acfs’ on ‘kens-racnode1’ succeeded

CRS-2676: Start of ‘ora.cvu’ on ‘kens-racnode2’ succeeded

CRS-2676: Start of ‘ora.kens-racnode1.vip’ on ‘kens-racnode2’ succeeded

CRS-2676: Start of ‘ora.scan2.vip’ on ‘kens-racnode2’ succeeded

CRS-2672: Attempting to start ‘ora.LISTENER_SCAN2.lsnr’ on ‘kens-racnode2’

CRS-2676: Start of ‘ora.scan3.vip’ on ‘kens-racnode2’ succeeded

CRS-2672: Attempting to start ‘ora.LISTENER_SCAN3.lsnr’ on ‘kens-racnode2’

CRS-2676: Start of ‘ora.LISTENER_SCAN3.lsnr’ on ‘kens-racnode2’ succeeded

CRS-2676: Start of ‘ora.LISTENER_SCAN2.lsnr’ on ‘kens-racnode2’ succeeded

CRS-2677: Stop of ‘ora.oc4j’ on ‘kens-racnode1’ succeeded

CRS-2672: Attempting to start ‘ora.oc4j’ on ‘kens-racnode2’

CRS-2676: Start of ‘ora.oc4j’ on ‘kens-racnode2’ succeeded

CRS-2677: Stop of ‘ora.DATA.dg’ on ‘kens-racnode1’ succeeded

CRS-2673: Attempting to stop ‘ora.asm’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.asm’ on ‘kens-racnode1’ succeeded

CRS-2673: Attempting to stop ‘ora.ons’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.ons’ on ‘kens-racnode1’ succeeded

CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.net1.network’ on ‘kens-racnode1’ succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘kens-racnode1’ has completed

CRS-2677: Stop of ‘ora.crsd’ on ‘kens-racnode1’ succeeded

CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘kens-racnode1’

CRS-2673: Attempting to stop ‘ora.evmd’ on ‘kens-racnode1’

CRS-2673: Attempting to stop ‘ora.asm’ on ‘kens-racnode1’

CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘kens-racnode1’

CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.asm’ on ‘kens-racnode1’ succeeded

CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.evmd’ on ‘kens-racnode1’ succeeded

CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘kens-racnode1’ succeeded

CRS-2677: Stop of ‘ora.mdnsd’ on ‘kens-racnode1’ succeeded

CRS-2677: Stop of ‘ora.ctssd’ on ‘kens-racnode1’ succeeded

CRS-2673: Attempting to stop ‘ora.cssd’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.cssd’ on ‘kens-racnode1’ succeeded

CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘kens-racnode1’

CRS-2673: Attempting to stop ‘ora.crf’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.crf’ on ‘kens-racnode1’ succeeded

CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.diskmon’ on ‘kens-racnode1’ succeeded

CRS-2677: Stop of ‘ora.gipcd’ on ‘kens-racnode1’ succeeded

CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.gpnpd’ on ‘kens-racnode1’ succeeded

CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘kens-racnode1’ succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘kens-racnode1’ has completed

CRS-4133: Oracle High Availability Services has been stopped.

OLR initialization – successful

Replacing Clusterware entries in inittab

clscfg: EXISTING configuration version 5 detected.

clscfg: version 5 is 11g Release 2.

Successfully accumulated necessary OCR keys.

Creating OCR keys for user ‘root’, privgrp ‘root’..

Operation successful.

Configure Oracle Grid Infrastructure for a Cluster … succeeded

[root@kens-racnode1 12539000]#

 

 

NODE 2

 

[root@kens-racnode2 OPatch]# /u01/app/11.2.0.3/grid/rootupgrade.sh

Performing root user operation for Oracle 11g

 

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME=  /u01/app/11.2.0.3/grid

 

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of “dbhome” have not changed. No need to overwrite.

The contents of “oraenv” have not changed. No need to overwrite.

The contents of “coraenv” have not changed. No need to overwrite.

 

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params

Creating trace directory

User ignored Prerequisites during installation

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘kens-racnode2’

CRS-2673: Attempting to stop ‘ora.crsd’ on ‘kens-racnode2’

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘kens-racnode2’

CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘kens-racnode2’

CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘kens-racnode2’

CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘kens-racnode2’

CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘kens-racnode2’

CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘kens-racnode2’

CRS-2673: Attempting to stop ‘ora.cvu’ on ‘kens-racnode2’

CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.LISTENER_SCAN3.lsnr’ on ‘kens-racnode2’ succeeded

CRS-2673: Attempting to stop ‘ora.scan3.vip’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.scan3.vip’ on ‘kens-racnode2’ succeeded

CRS-2672: Attempting to start ‘ora.scan3.vip’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.cvu’ on ‘kens-racnode2’ succeeded

CRS-2672: Attempting to start ‘ora.cvu’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘kens-racnode2’ succeeded

CRS-2673: Attempting to stop ‘ora.kens-racnode2.vip’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.LISTENER_SCAN2.lsnr’ on ‘kens-racnode2’ succeeded

CRS-2673: Attempting to stop ‘ora.scan2.vip’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.kens-racnode2.vip’ on ‘kens-racnode2’ succeeded

CRS-2672: Attempting to start ‘ora.kens-racnode2.vip’ on ‘kens-racnode1’

CRS-2677: Stop of ‘ora.scan2.vip’ on ‘kens-racnode2’ succeeded

CRS-2672: Attempting to start ‘ora.scan2.vip’ on ‘kens-racnode1’

CRS-2676: Start of ‘ora.cvu’ on ‘kens-racnode1’ succeeded

CRS-2677: Stop of ‘ora.registry.acfs’ on ‘kens-racnode2’ succeeded

CRS-2676: Start of ‘ora.scan3.vip’ on ‘kens-racnode1’ succeeded

CRS-2672: Attempting to start ‘ora.LISTENER_SCAN3.lsnr’ on ‘kens-racnode1’

CRS-2676: Start of ‘ora.scan2.vip’ on ‘kens-racnode1’ succeeded

CRS-2676: Start of ‘ora.kens-racnode2.vip’ on ‘kens-racnode1’ succeeded

CRS-2672: Attempting to start ‘ora.LISTENER_SCAN2.lsnr’ on ‘kens-racnode1’

CRS-2676: Start of ‘ora.LISTENER_SCAN3.lsnr’ on ‘kens-racnode1’ succeeded

CRS-2676: Start of ‘ora.LISTENER_SCAN2.lsnr’ on ‘kens-racnode1’ succeeded

CRS-2677: Stop of ‘ora.oc4j’ on ‘kens-racnode2’ succeeded

CRS-2672: Attempting to start ‘ora.oc4j’ on ‘kens-racnode1’

CRS-2676: Start of ‘ora.oc4j’ on ‘kens-racnode1’ succeeded

CRS-2677: Stop of ‘ora.DATA.dg’ on ‘kens-racnode2’ succeeded

CRS-2673: Attempting to stop ‘ora.asm’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.asm’ on ‘kens-racnode2’ succeeded

CRS-2673: Attempting to stop ‘ora.ons’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.ons’ on ‘kens-racnode2’ succeeded

CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.net1.network’ on ‘kens-racnode2’ succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘kens-racnode2’ has completed

CRS-2677: Stop of ‘ora.crsd’ on ‘kens-racnode2’ succeeded

CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘kens-racnode2’

CRS-2673: Attempting to stop ‘ora.evmd’ on ‘kens-racnode2’

CRS-2673: Attempting to stop ‘ora.asm’ on ‘kens-racnode2’

CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘kens-racnode2’

CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.asm’ on ‘kens-racnode2’ succeeded

CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.evmd’ on ‘kens-racnode2’ succeeded

CRS-2677: Stop of ‘ora.mdnsd’ on ‘kens-racnode2’ succeeded

CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘kens-racnode2’ succeeded

CRS-2677: Stop of ‘ora.ctssd’ on ‘kens-racnode2’ succeeded

CRS-2673: Attempting to stop ‘ora.cssd’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.cssd’ on ‘kens-racnode2’ succeeded

CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘kens-racnode2’

CRS-2673: Attempting to stop ‘ora.crf’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.crf’ on ‘kens-racnode2’ succeeded

CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.diskmon’ on ‘kens-racnode2’ succeeded

CRS-2677: Stop of ‘ora.gipcd’ on ‘kens-racnode2’ succeeded

CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘kens-racnode2’

CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘kens-racnode2’ succeeded

CRS-2677: Stop of ‘ora.gpnpd’ on ‘kens-racnode2’ succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘kens-racnode2’ has completed

CRS-4133: Oracle High Availability Services has been stopped.

OLR initialization – successful

Replacing Clusterware entries in inittab

clscfg: EXISTING configuration version 5 detected.

clscfg: version 5 is 11g Release 2.

Successfully accumulated necessary OCR keys.

Creating OCR keys for user ‘root’, privgrp ‘root’..

Operation successful.

Started to upgrade the Oracle Clusterware. This operation may take a few minutes.

Started to upgrade the CSS.

Started to upgrade the CRS.

The CRS was successfully upgraded.

Oracle Clusterware operating version was successfully set to 11.2.0.3.0

 

ASM upgrade has finished on last node.

 

PRKO-2116 : OC4J is already enabled

Configure Oracle Grid Infrastructure for a Cluster … succeeded

 

TEST

 

We now see that the clusterware processes are now running from the 11.2.0.3  Grid Infrastructure Oracle Home.

We can also use the various crsctl query crs commands to confirm the upgraded software version.

 

[root@kens-racnode1 12539000]# ps -ef |grep css

root     19784     1  0 06:50 ?        00:00:00 /u01/app/11.2.0.3/grid/bin/cssdmonitor

root     19807     1  0 06:50 ?        00:00:00 /u01/app/11.2.0.3/grid/bin/cssdagent

grid     19823     1  0 06:50 ?        00:00:02 /u01/app/11.2.0.3/grid/bin/ocssd.bin

root     26134 13158  0 07:05 pts/0    00:00:00 grep css

 

 

[grid@kens-racnode1 ~]$ crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [11.2.0.3.0]

 

[grid@kens-racnode1 ~]$ crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [11.2.0.3.0]

 

[grid@kens-racnode1 ~]$ crsctl query crs softwareversion

Oracle Clusterware version on node [kens-racnode1] is [11.2.0.3.0]

 

[grid@kens-racnode1 ~]$ crsctl query crs softwareversion kens-racnode2

Oracle Clusterware version on node [kens-racnode2] is [11.2.0.3.0]