Real Application Clusters « Oracle DBA

Building a test 11gR2 RAC environment in minutes using Oracle Virtual Machine templates

This note illustrates how we can set up an Orace 11g Release 2 Real Application Clusters environment for test purposes using the Oracle VM templates which are currently available in under 30 minutes!

The templates are available from the Oracle E-Delivery web site and are available in both OEL 32bit Linux as well as 64bit Linux.

While we are demonstrating a test or development RAC setup where a single Oracle VM server is hosting both the guest nodes, a ‘Production’ type envrionment is also supported using the Oracle VM Templates where we have multiple OVM Servers and the shared disks are configured as ‘phy’ devices which are then passed on to the guest Oracle VM’s. So there are a number of Oracle VM Servers in this case.

In this test environment, however, the shared disk is configured as ‘file’ devices and both guests can run on the same Oracle VM Server.

In other words

Continue reading Building a test 11gR2 RAC environment in minutes using Oracle Virtual Machine templates

11gR2 RAC installation and the ORA-15014 error

In 11g Release 2, we now have to place the Voting Disk and Cluster Registry files on ASM disks and we do not have the option of placing them on OCFS or raw devices as in the case of previous versions.

When you run the root.sh script as part of the 11gR2 RAC installation, it will create an ASM disk group CDATA and will place the OCR and Voting disk files on this ASM disk. We provide an ASM disk discovery string where the raw devices exist on which the ASM disk groups will be built.

Initially we had provided the ASM diskstring as /dev/ocr_disk* as this was where we wanted to place the OCR and Vote files. But when we tried to change this to create another ASM disk group, we kept getting these ORA-15031 and ORA-15014 errors.

We had to change the asm_diskstring parameter to include both the paths

Continue reading 11gR2 RAC installation and the ORA-15014 error

11g ASM rolling upgrade from 11.1.0.6 to 11.1.0.7

In Oracle 11g, we can now do rolling upgrades of the ASM instanaces in a RAC cluster configuration. In this example shown below, one of the ASM instances is being upgraded to 11.1.0.7 while the other instance is still running on version 11.1.0.6. While the ASM software is being upgraded, client connections to the RAC database will continue as the service will get relocated to the other node in the cluster which is not being currently upgraded. So in other words this is an online patch application.

The database name here is racdba and the two instances are racdba1 and racdba2 running on nodes middba1 and middba2. The platform used in this example is AIX 5.3 64 bit on a P series LPAR.

On one of the nodes in the cluster, connect to the ASM instance (+ASM1) and issue the command to start the rolling migration.

SQL> alter system start rolling migration to

Continue reading 11g ASM rolling upgrade from 11.1.0.6 to 11.1.0.7

11.1.0.6 Clusterware upgrade to 11.1.0.7

The following screenshots are taken from a RAC Clusterware upgrade from 11.1.0.6 to 11.1.0.7 on AIX 64 bit platform.

We encountered an issue related to Bug 8925669: ROOT111.SH 0509-150 DEPENDENT MODULE LIBTTSH11.SO COULD NOT BE LOADED. I have provided the workaround we used in this case and which did work for us.

The last screen will indicate that we need to run the root111.sh script which will stop and start the clusterware processes in the upgraded CRS Home

The installer has detected that your Cluster Ready Services (CRS) installation is distributed across the following nodes:

middba1
middba2

Because the software consists of local identical copies distributed across each of the nodes in the cluster, it is possible to patchyour CRS installation in a rolling manner, one node at a time.

To complete the installation of this patchset, you must perform the following tasks on each node:

1. Log in as the root user.
2. As the root user, perform the following

Continue reading 11.1.0.6 Clusterware upgrade to 11.1.0.7

ORA-27504 error creating ASM instance on 11g RAC

While creating the ASM instance using DBCA in an 11g RAC environment on 64 bit AIX 5L, we got the ORA-27504 error along with others as shown below.

The problem was caused by incorrect UDP and TCP packet settings and is documented in Metalink note 300956.1

As root, the following changes need to be made

# no -o tcp_sendspace=262144
# no -o tcp_recvspace=262144
# no -o udp_sendspace=65536
# no -o udp_recvspace=262144
# no -o rfc1323=1

Adding these entries to the /etc/rc.net will ensure that these parameters take effect on every machine reboot.

11g RAC Software Relink Error on AIX

While relinking the Oracle 11g RAC software on 64 bit AIX platform, we got the error as shown below:

Looking at the installation log we found the following lines:

ld: 0706-006 Cannot find or open library file: -l ha_gs_r
ld:open(): A file or directory in the path name does not exist.
ld: 0706-006 Cannot find or open library file: -l ha_em_r
ld:open(): A file or directory in the path name does not exist.
make: 1254-004 The error code from the last command is 255

The cause of this error is that a particluar fileset rsct.basic.rte which is required specifically for a 11g RAC installation has not been installed at the OS level. After the fileset was installed we could relink the software without the same ld errors as mentioned above.

We can check if the fileset has been installed by running the command ‘lslpp’ as shown below.

testdb:/u02/oradata/testdb> lslpp -l rsct.basic.rte
lslpp: Fileset rsct.basic.rte not installed.

After the fileset is

Continue reading 11g RAC Software Relink Error on AIX

11g RAC Transparent Application Failover (TAF)

This example illustrates Transparent Application Failover (TAF) in a two node AIX 11g RAC configuration.

The database name is racdb and the two instances are racdb1 and racdb2.

Node 1 – middba1 which hosts instance racdb1.
Node2 – middba2 which hosts instance racdb2

We have defined two services racdb1 and racdb2. For service racdb1 the preferred instance is racdb1 and
for service racdb2 the preferred instance is racdb2.

The following srvctl commands illustrates the same.

middba1:/u01/oracle/dump> srvctl status service -d racdb -s racdb1
Service racdb1 is running on instance(s) racdb1

middba1:/u01/oracle/dump> srvctl config service -d racdb -s racdb1
racdb1 PREF: racdb1 AVAIL: racdb2

middba1:/u01/oracle/dump> srvctl status service -d racdb -s racdb2
Service racdb2 is running on instance(s) racdb2

middba1:/u01/oracle/dump> srvctl config service -d racdb -s racdb2
racdb2 PREF: racdb2 AVAIL: racdb1

From a SQL*PLUS client, we establish a session as user SYSTEM using the service racdb1. Note the machine name where this service is running from.

testdb:/u01/oracle> sqlplus system/xxx@racdb1

SQL*Plus: Release 11.1.0.6.0 – Production on Fri

Continue reading 11g RAC Transparent Application Failover (TAF)

CRS-0184 error and its resolution

While installing the 11g Clusterware on 64 bit AIX 5L, we encountered the CRS-0184 error while running the root.sh on the second node.

The error was …

Cluster Synchronization Services is active on all the nodes.

Waiting for the Oracle CRSD and EVMD to start

Oracle CRS stack installed and running under init(1M)

Running vipca(silent) for configuring nodeapps

Creating VIP application resource on (2) nodes0:CRS-0184: Cannot communicate with the CRS daemon

The crsd.log file showed the following lines ….

2009-08-10 14:18:36.059: [ CRSMAIN][10286] Failed to spawn a thread for UI connection. error=-11
2009-08-10 14:19:12.143: [ CRSMAIN][10286] Failed to spawn a thread for UI connection. error=-11
2009-08-10 14:20:15.037: [ CRSMAIN][10286] Failed to spawn a thread for UI connection. error=-11
2009-08-10 14:21:11.379: [ CRSMAIN][10286] Failed to spawn a thread for UI connection. error=-11

The fix for this error was to increase the Shell Limits for the user ‘root’ as the Clusterware processes are started as root.

The documentation does mention that we need to so the

Continue reading CRS-0184 error and its resolution

Oracle DBA – Tips and Techniques

That’s Me

Feedback

Categories

Archives

Recent Posts

Administration