Upgrading Clusterware to patchset 3 (10.2.0.4)

Recently upgraded Oracle 10g RAC (10.2.0.1) to 10.2.0.4 patchset. Noticed something interesting while applying the patchset. I have my 10g RAC running on 10.2.0.1 (base version).I planned to apply patch as a ROLLING Upgrade. The patchset document says the patch can be applied as rolling upgrade but when you invoke the runInstaller for cluster upgrade, you won't get the option to select one of the two nodes (in 2 node cluster). Both the nodes are automatically selected and grayed out (we can't make any change). Initially i thought that my cluster is not being detected by runInstaller.Cancelled it and set the ORACLE_HOME,PATH etc...properly but got the same thing again...But later realised that it works like that only. The runInstaller completes the installation and prompts to shutdown the CRS on node 1 and then run root102.sh.This scripts bring up the cluster after execution and the process is repeated on node 2...So here i go...

I have my 10.2.0.1 clusterware running
[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE ONLINE rac1
ora....l1.inst application ONLINE ONLINE rac1
ora....l2.inst application ONLINE ONLINE rac2
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2

Node-1
[root@rac1 bin]# ./crsctl query crs softwareversion
CRS software version on node [rac1] is [10.2.0.1.0]
[root@rac1 bin]# ./crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.1.0]

Node-2
[root@rac2 bin]# ./crsctl query crs softwareversion
CRS software version on node [rac2] is [10.2.0.1.0]
[root@rac2 bin]# ./crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.1.0]

Applying the patchset.
1) Stop service, db instance, asm and nodeapps on node 1 (rac1)
[oracle@rac1 ~]$ srvctl stop instance -i orcl1 -d orcl
[oracle@rac1 ~]$ srvctl stop asm -n rac1
[oracle@rac1 ~]$ srvctl stop nodeapps -n rac1
[oracle@rac1 ~]$ crs_stat -t

Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE ONLINE rac1
ora....l1.inst application OFFLINE OFFLINE
ora....l2.inst application ONLINE ONLINE rac2
ora....SM1.asm application OFFLINE OFFLINE
ora....C1.lsnr application OFFLINE OFFLINE
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application OFFLINE OFFLINE
ora.rac1.vip application OFFLINE OFFLINE
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2

Don't stop the CRS daemons. They would be brought down later.
[root@rac1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

2) Unzip the patch (p6810189_10204_Linux-x86.zip) and invoke runInstaller

3) In the screenshot as shown below both the nodes are grayed out and there is not option to select one node (this was my assumpiton intially that rolling should apply one by one with node selection option)..Click Next to move ahead...

4) Once the installer completes, it will prompt to run the following two scripts.(Each node one at a time)
/u01/app/oracle/product/10.2.0/crs/bin/crsctl stop crs
/u01/app/oracle/product/10.2.0/crs/install/root102.sh

The first script stops the clusterware and after that second script completesthe patching...
[root@rac1 bin]# /u01/app/oracle/product/10.2.0/crs/bin/crsctl stop crs
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@rac1 bin]#

[root@rac1 bin]# /u01/app/oracle/product/10.2.0/crs/bin/crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
[root@rac1 bin]#

[oracle@rac2 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE ONLINE rac2
ora....l1.inst application OFFLINE OFFLINE
ora....l2.inst application ONLINE ONLINE rac2
ora....SM1.asm application OFFLINE OFFLINE
ora....C1.lsnr application OFFLINE OFFLINE
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application OFFLINE OFFLINE
ora.rac1.vip application OFFLINE OFFLINE
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
[oracle@rac2 ~]$

[root@rac1 install]# /u01/app/oracle/product/10.2.0/crs/install/root102.sh
Creating pre-patch directory for saving pre-patch clusterware files
Completed patching clusterware files to /u01/app/oracle/product/10.2.0/crs
Relinking some shared libraries.
Relinking of patched files is complete.
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
WARNING: directory '/u01' is not owned by root
Preparing to recopy patched init and RC scripts.
Recopying init and RC scripts.
Startup will be queued to init within 30 seconds.
/etc/profile: line 59: ulimit: open files: cannot modify limit: Operation not permitted
Starting up the CRS daemons.
Waiting for the patched CRS daemons to start.
This may take a while on some systems.
.
10204 patch successfully applied.
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: rac1 rac1-priv rac1
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
clscfg -upgrade completed successfully
[root@rac1 install]#

The above scripts will automatically startup the clusterware on node 1 (and nodeapps,asm and instance if they are enabled for startup)

root@rac1 bin]# ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE ONLINE rac2
ora....l1.inst application ONLINE ONLINE rac1
ora....l2.inst application ONLINE ONLINE rac2
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
[root@rac1 bin]#

This is what i have in alertrac1.log under $CRS_HOME/log/rac1
[cssd(30267)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2009-09-10 14:32:18.880
[evmd(29572)]CRS-1401:EVMD started on node rac1.
2009-09-10 14:32:18.927
[crsd(29700)]CRS-1012:The OCR service started on node rac1.
2009-09-10 14:32:20.661
[crsd(29700)]CRS-1201:CRSD started on node rac1.
2009-09-10 14:32:31.424
[client(30665)]CRS-1804:Node rac1 upgraded to version 10.2.0.4.0.
2009-09-10 14:32:33.507

So the software version of clusterware change to 10.2.0.4 but it would be active only after upgrade of second node.

[root@rac1 bin]# ./crsctl query crs softwareversion
CRS software version on node [rac1] is [10.2.0.4.0]
[root@rac1 bin]# ./crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.1.0]
[root@rac1 bin]#

5) Now is the time to repeat the scripts on node 2 (rac2). Before that shutdown the resources i.e. Stop service, db instance, asm and nodeapps on node 2 (rac2)

[root@rac2 bin]# /u01/app/oracle/product/10.2.0/crs/bin/crsctl stop crs
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE OFFLINE
ora....l1.inst application ONLINE OFFLINE
ora....l2.inst application ONLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE OFFLINE
ora....C2.lsnr application ONLINE OFFLINE
ora.rac2.gsd application ONLINE OFFLINE
ora.rac2.ons application ONLINE OFFLINE
ora.rac2.vip application ONLINE OFFLINE

[root@rac2 install]# /u01/app/oracle/product/10.2.0/crs/install/root102.sh
Creating pre-patch directory for saving pre-patch clusterware files
Completed patching clusterware files to /u01/app/oracle/product/10.2.0/crs
Relinking some shared libraries.
Relinking of patched files is complete.
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
WARNING: directory '/u01' is not owned by root
Preparing to recopy patched init and RC scripts.
Recopying init and RC scripts.
Startup will be queued to init within 30 seconds.
/etc/profile: line 59: ulimit: open files: cannot modify limit: Operation not pe rmitted
Starting up the CRS daemons.
Waiting for the patched CRS daemons to start.
This may take a while on some systems.
.
10204 patch successfully applied.
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 2: rac2 rac2-priv rac2
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
clscfg -upgrade completed successfully
[root@rac2 install]#

[root@rac2 bin]# ./crsctl query crs softwareversion
CRS software version on node [rac2] is [10.2.0.4.0]
[root@rac2 bin]# ./crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.4.0]
[root@rac2 bin]#

At the end of this script, everyhting would be brought up automatically.

[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE ONLINE rac1
ora....l1.inst application ONLINE ONLINE rac1
ora....l2.inst application ONLINE ONLINE rac2
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
[oracle@rac1 ~]$

This patched the CRS to 10.2.0.4...

-- Rishi