ORA-29701 after IP address change and overcoming the SIHA ORA-29787 error on SPFILE creation

I had recently installed 11g Release 2 Grid Infrastructure for a single instance on a RHEL 5 virtual machine set up using VMware server and after I changed the IP address for one of the NIC’s (eth0) I started getting ORA-29701 errors whenever I issued the crsctl check command as well as when I tried to start the ASM instance as shown below.

SQL> startup;
ORA-01078: failure in processing system parameters
ORA-29701: unable to connect to Cluster Synchronization Service

These are the steps I had to take to overcome the problem – basically it is recreating the Oracle High Availability Services and updating the cluster registry with all the infomation about the resources which were originally configured.

[root@gavin install]# ./roothas.pl -deconfig
2010-05-20 12:49:48: Checking for super user privileges
2010-05-20 12:49:48: User has super user privileges
2010-05-20 12:49:48: Parsing the host name
Using configuration parameter file: ./crsconfig_params
CRS resources for listeners are still configured
CRS-2730: Resource 'ora.asm' depends on resource 'ora.cssd'
CRS-4000: Command Delete failed, or completed with errors.
CRS-4133: Oracle High Availability Services has been stopped.
ACFS-9200: Supported
Successfully deconfigured Oracle Restart stack

[root@gavin install]# ./roothas.pl
2010-05-20 12:50:49: Checking for super user privileges
2010-05-20 12:50:49: User has super user privileges
2010-05-20 12:50:49: Parsing the host name
Using configuration parameter file: ./crsconfig_params
LOCAL ADD MODE 
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
CRS-4664: Node gavin successfully pinned.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting

gavin     2010/05/20 12:51:44     /u02/app/oracle/product/11.2.0/grid/cdata/gavin/backup_20100520_125144.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
[root@gavin install]# 

[root@gavin install]# pwd
/u02/app/oracle/product/11.2.0/grid/crs/install
[root@gavin install]# cd ../../bin
[root@gavin bin]# ./crsctl modify resource "ora.cssd" -attr "AUTO_START=1"
[root@gavin bin]# ./crsctl modify resource "ora.diskmon" -attr "AUTO_START=1"
[root@gavin bin]# ./srvctl add asm
[root@gavin bin]# ./crsctl modify resource "ora.asm" -attr "AUTO_START=1"
[root@gavin bin]# 

I then started the ASM instance with a temporary init.ora file. The contents are shown below.

[oracle@gavin ~]$ cat /tmp/init+ASM.ora
asm_diskgroups=’DATA’
instance_type=’asm’
large_pool_size=12M
remote_login_passwordfile=’EXCLUSIVE’

But when we tried to create the spfile from this temporary init.ora we got a strange error as shown below.

SQL> create spfile=’+DATA’ from pfile=’/tmp/init+ASM.ora’;
create spfile=’+DATA’ from pfile=’/tmp/init+ASM.ora’
*
ERROR at line 1:
ORA-29787: SIHA attribute SET failed with error [Attribute ‘SPFILE’ sts[219]
lsts[0]]

This error I think was because the ASM had not been registered with the OCR and when we tried to create a spfile located on an ASM disk +DATA, it could not find that disk as ASM itself was not believed to be running even though a ps -ef |grep +ASM did show the instance running – but the cluster registry was not aware of that.

What I had to do was to add the ASM instance using the srvctl command, but also provide the location of the temporary init.ora file which was used to initially start the ASM instance. Once I did that, the ASM details got registered and I was able to create the spfile on the ASM disk.

[oracle@gavin ~]$ srvctl status asm
PRCR-1001 : Resource ora.asm does not exist
[oracle@gavin ~]$ srvctl add asm -p /tmp/init+ASM.ora
[oracle@gavin ~]$ srvctl status asm
ASM is not running.
[oracle@gavin ~]$ srvctl start asm
[oracle@gavin ~]$ srvctl status asm
ASM is running on gavin

SQL> create spfile=’+DATA’ from pfile=’/tmp/init+ASM.ora’;
File created.