Change IP address for oracle RAC public and VIP interfaces

Change IP address for oracle RAC public and VIP interfaces Rohit Important X Reply Change IP address for oracle RAC public and VIP interfaces Filed under: [RAC] — Tags: ORACLE RAC CHANGE IP vip public oifcfg — zhefeng @ 4:33 pm Ads by Google 24x7 oracle Switch to Itanium®-based solutions. Download industry case studies. www.ItaniumSolutions.org My company is doing massive re-ip project. So our Oracle RAC ip address has to be changed as well. Fortunately, we don’t need to change hostname, otherwise the story will be more complicated. 1. Current Environment 1). Machine IP: Node1: vanpgvmrac01 Node2: vanpgvmrac02 ## eth1-Public 10.50.96.101 vanpgvmrac01 vanpgvmrac01.product.com 10.50.96.102 vanpgvmrac02 vanpgvmrac02.product.com ## eth0-Private 192.168.199.1 vanpgvmracprv01 vanpgvmracprv01.product.com 192.168.199.2 vanpgvmracprv02 vanpgvmracprv02.product.com ## VIP 10.50.96.103 vanpgvmracvip01 vanpgvmracvip01.product.com 10.50.96.104 vanpgvmracvip02 vanpgvmracvip02.product.com 2). cluster information: cluster name – vm10cls database name – v10c Instance 1 – v10c1 Instance 2 – v10c2 Node1 – vanpgvmrac01 Node2 – vanpgvmrac02 2. New IP changing map(different subnet mask too): 10.50.96.101/255.255.255.0 vanpgvmrac01 –> 10.50.99.41/255.255.252.0 10.50.96.102/255.255.255.0 vanpgvmrac02 –> 10.50.99.42/255.255.252.0 10.50.96.103/255.255.255.0 vanpgvmracvip01 –> 10.50.99.43/255.255.252.0 10.50.96.104/255.255.255.0 vanpgvmracvip02 –> 10.50.99.44/255.255.252.0 3. steps 1 — change RAC IP settings 1). bring service down, make sure everything was offline except css daemon bash-3.1$ srvctl stop database -d v10c bash-3.1$ srvctl stop nodeapps -n vanpgvmrac01 bash-3.1$ srvctl stop nodeapps -n vanpgvmrac02 bash-3.1$ crs_stat -t Name Type Target State Host ———————————————————— ora.v10c.db application OFFLINE OFFLINE ora….c1.inst application OFFLINE OFFLINE ora….c2.inst application OFFLINE OFFLINE ora….SM1.asm application OFFLINE OFFLINE ora….01.lsnr application OFFLINE OFFLINE ora….c01.gsd application OFFLINE OFFLINE ora….c01.ons application OFFLINE OFFLINE ora….c01.vip application OFFLINE OFFLINE ora….SM2.asm application OFFLINE OFFLINE ora….02.lsnr application OFFLINE OFFLINE ora….c02.gsd application OFFLINE OFFLINE ora….c02.ons application OFFLINE OFFLINE ora….c02.vip application OFFLINE OFFLINE 2). backup OCR and Voting disks bash-3.1$ ocrcheck|grep -i file Device/File Name : /dev/raw/raw1 bash-3.1$ crsctl query css votedisk 0. 0 /dev/raw/raw2 located 1 votedisk(s). #dd if=/dev/raw/raw1 of=/database/temp/ocr_vote_bk/ocr.bak #dd if=/dev/raw/raw2 of=/database/temp/ocr_vote_bk/vote.bak 3). get current config: bash-3.1$ oifcfg getif eth0 10.50.96.0 global public –current network for public eth1 192.168.199.0 global cluster_interconnect –we are not going to change this 4). delete current public ip: bash-3.1$ oifcfg delif -global eth0 5). change to new network: bash-3.1$ oifcfg setif -global eth0/10.50.99.0:public 6). change vip address: a. check current one bash-3.1$ srvctl config nodeapps -n vanpgvmrac01 -a VIP exists.: /vanpgvmracvip01/10.50.96.103/255.255.255.0/eth0 bash-3.1$ srvctl config nodeapps -n vanpgvmrac02 -a VIP exists.: /vanpgvmracvip02/10.50.96.104/255.255.255.0/eth0 b. Modify VIP component (has to be the css owner, “root” usually) #srvctl modify nodeapps -n vanpgvmrac01 -A 10.50.99.43/255.255.252.0/eth0 #srvctl modify nodeapps -n vanpgvmrac02 -A 10.50.99.44/255.255.252.0/eth0 c. double verify the changes bash-3.1$ srvctl config nodeapps -n vanpgvmrac01 -a VIP exists.: /vanpgvmracvip01/10.50.99.43/255.255.252.0/eth0 bash-3.1$ srvctl config nodeapps -n vanpgvmrac02 -a VIP exists.: /vanpgvmracvip02/10.50.99.44/255.255.252.0/eth0 7). change the hosts file(on both nodes): ## eth1-Public 10.50.99.41 vanpgvmrac01 vanpgvmrac01.product.com 10.50.99.42 vanpgvmrac02 vanpgvmrac02.product.com ## eth0-Private 192.168.199.1 vanpgvmracprv01 vanpgvmracprv01.product.com 192.168.199.2 vanpgvmracprv02 vanpgvmracprv02.product.com ## VIP 10.50.99.43 vanpgvmracvip01 vanpgvmracvip01.product.com 10.50.99.44 vanpgvmracvip02 vanpgvmracvip02.product.com 8). if the listener is using any IP address, it also needs to be changed. 4. Steps 2 — change OS IP settings 1). change IP [root@vanpgvmrac01]# vi /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 BOOTPROTO=static IPADDR=10.50.99.41 NETMASK=255.255.252.0 HWADDR=00:50:56:BD:05:14 ONBOOT=yes [root@vanpgvmrac02]# vi /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 BOOTPROTO=static IPADDR=10.50.99.42 NETMASK=255.255.252.0 HWADDR=00:50:56:BD:3E:08 ONBOOT=yes 2). change the default gateway on both nodes (if needed, here since they are in same vlan so i didn’t change them) [root@vanpgvmrac01 ~]# cat /etc/sysconfig/network NETWORKING_IPV6=yes HOSTNAME=vanpgvmrac01 NETWORKING=yes NISDOMAIN=vanrdnis GATEWAY=10.50.96.1 <– here is the default gateway to be changed 3) Change the IP Address’es on the known_hosts ssh config files for oracle user $ su – oracle $ cd .ssh $ cp known_hosts known_hosts.bak $ modify the old IP’s to the new IP’s 4). restart network (on both node) #service network restart 5). restart crs daemon (on both node) #crsctl stop crs #crsctl start crs 5. Step3 — verify everything reference: 1. "How to Change Interconnect/Public Interface IP or Subnet in Oracle Clusterware", Doc ID: 283684.1 2. "Modifying the VIP or VIP Hostname of a 10g or 11g Oracle Clusterware Node", DOC ID: 276434.1 3. "How to change Public and VIP component address in case of RAC?" http://orcl-experts.info/index.php?name=FAQ&id_cat=9 How to Change the public IP Address in an Oracle RAC Though this task is well documented (see [1] and [2]) there are still some pitfalls and quirks that can cause problems. Such as.. Should a particular task be done with the root or oracle user? Should it be performed on both nodes? This is a step-by-step description how to perform the change of the public IP address in the Oracle two node RAC. The Oracle version is 10.2.0.3. The OS used in this example is Linux Centos 2.6.9-42.ELsmp Note that due to the change of the private IP address the virtual IP address (VIP) may be required to change as well as it must remain in the same subnet as the public IP address. The configuration Host Current IP Address Changed IP Address Private IP hst1 192.168.1.101 192.168.2.121 Private IP hst2 192.168.1.100 192.168.2.120 VIP hst1-vip 192.168.1.111 192.168.2.111 VIP hst2-vip 192.168.1.110 192.168.2.110 Note that the actual value of IP addresses and host names was changed. The basic steps The change of the private IP address is performed in the following basic steps 1) Shut down everything except the CRS stack 2) Change the public interface 3) Modify the VIP address 4) Shut down CRS 5) Modify IP address on OS level and reconfigure /etc/hosts, listener,.. 6) Restart The order of the steps is very important. E.g. step 3) can be performed only if the CRS stack is up. If you perform step 5) prematurely, CRS will go down automatically (probably trying some reboots beforehand). This will effectively block step 3) to be done. Shut Down Everything Except the CRS Stack The database and nodeapps (on all nodes) are stopped. [oracle@hst1 ~]$ srvctl stop database -d mydb [oracle@hst1 ~]$ srvctl stop nodeapps -n hst1 [oracle@hst1 ~]$ srvctl stop nodeapps -n hst2 Note that if you use ASM, it must be stopped as well. After that we verify the status. [oracle@hst1 ~]$ srvctl status database -d mydb Instance MYDB1 is not running on node hst2 Instance MYDB2 is not running on node hst1 [oracle@hst1 ~]$ srvctl status nodeapps -n hst1 VIP is not running on node: hst1 GSD is not running on node: hst1 Listener is not running on node: hst1 ONS daemon is not running on node: hst1 [oracle@hst1 ~]$ srvctl status nodeapps -n hst2 VIP is not running on node: hst2 GSD is not running on node: hst2 Listener is not running on node: hst2 ONS daemon is not running on node: hst2 Everything is OK, we can go to the next step. Change the Public Interface First let us have a look on the actual status [oracle@hst1 ~]$ oifcfg getif eth0 192.168.1.0 global public eth1 192.168.2.0 global cluster_interconnect We need to change the interface eth0. As there is no modify command, we will delete and redefine the interface. [oraclu@hst1 ~]$ oifcfg delif -global eth0 [oraclu@hst1 ~]$ oifcfg setif -global eth0/192.168.2.0:public the exact syntax of the setif command can be found in [3] Note that the CRS installation user (here oraclu) must be used for this operation. Otherwise an error message is issued: PROC-5: User does not have permission to perform a cluster registry operation on this key. Authentication error [User [oraclu] does not match with initialized u ser] [0] PRIF-11: cluster registry error Finally, we check if the action was done successfully [oracle@hst1 ~]$ oifcfg getif eth0 192.168.2.0 global public eth1 192.168.2.0 global cluster_interconnect Modify the VIP Address As already mentioned, due to the fact that we changed the subnet of the public IP address, we must change the VIP address as well. The following modify statement should be used. [root@hst1 ~]# /appl/oracle/product/10.2.0/db_1/bin/srvctl modify nodeapps -n hst2 -A 192.168.2.110/255.255.255.0/eth0 [root@hst1 ~]# /appl/oracle/product/10.2.0/db_1/bin/srvctl modify nodeapps -n hst1 -A 192.168.2.111/255.255.255.0/eth0 Note that the root user should be used for this action. The variable ORACLE_HOHE must be initialised. Otherwise one of the following errors will be raised. PRKO-2117 : This command should be executed as the system privilege user. ****ORACLE_HOME environment variable not set! ORACLE_HOME should be set to the main directory that contains Oracle products. Set and export ORACLE_HOME, then re-run. Shut Down CRS The stop command must be performed on all nodes: crsctl stop crs Modify the IP Address on OS Level Modify the public IP address (eth0). In Centos with application / system setting / network Perform the change on all nodes. Modify /etc/hosts and listener.ora files if required. Restart Reboot all nodes and verify the status. Shortly after reboot we can see... [oracle@hst2 ~]$ /sbin/ifconfig -a | egrep '(eth|Mask)' eth0 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:FE inet addr:192.168.2.120 Bcast:192.168.2.255 Mask:255.255.255.0 eth0:1 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:FE inet addr:192.168.2.111 Bcast:192.168.2.255 Mask:255.255.255.0 eth0:2 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:FE inet addr:192.168.2.110 Bcast:192.168.2.255 Mask:255.255.255.0 eth1 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:F1 inet addr:192.168.2.100 Bcast:192.168.2.255 Mask:255.255.255.0 inet addr:127.0.0.1 Mask:255.0.0.0 This means both VIP's eth0:1 and eth0:2 are switched to one host. This is not good, normally this appears if one host is down and the second host takes over the VIP. Fortunately, after a few seconds we see.. [oracle@hst2 ~]$ /sbin/ifconfig -a | egrep '(eth|Mask)' eth0 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:FE inet addr:192.168.2.120 Bcast:192.168.2.255 Mask:255.255.255.0 eth0:2 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:FE inet addr:192.168.2.110 Bcast:192.168.2.255 Mask:255.255.255.0 eth1 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:F1 inet addr:192.168.2.100 Bcast:192.168.2.255 Mask:255.255.255.0 inet addr:127.0.0.1 Mask:255.0.0.0 [oracle@hst1 ~]$ /sbin/ifconfig -a | egrep '(eth|Mask)' eth0 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:F2 inet addr:192.168.2.121 Bcast:192.168.2.255 Mask:255.255.255.0 eth0:1 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:F2 inet addr:192.168.2.111 Bcast:192.168.2.255 Mask:255.255.255.0 eth1 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:F3 inet addr:192.168.2.101 Bcast:192.168.2.255 Mask:255.255.255.0 inet addr:127.0.0.1 Mask:255.0.0.0 I.e. the VIP's are distributed correctly. The configuration of IP addresses is as required. Let’s have a final look at the status of the RAC. [oracle@hst1 ~]$ srvctl status database -d mydb Instance MYDB1 is running on node hst2 Instance MYDB2 is running on node hst1 [oracle@hst1 ~]$ srvctl status nodeapps -n hst1 VIP is running on node: hst1 GSD is running on node: hst1 Listener is running on node: hst1 ONS daemon is running on node: hst1 [oracle@hst2 ~]$ srvctl status database -d mydb Instance MYDB1 is running on node hst2 Instance MYDB2 is running on node hst1 [oracle@hst2 ~]$ srvctl status nodeapps -n hst1 VIP is running on node: hst1 GSD is running on node: hst1 Listener is running on node: hst1 ONS daemon is running on node: hst1 [root@hst1 ~]# /appl/oraclu/product/10.2.0/crs/bin/crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....B1.inst application ONLINE ONLINE hst2 ora....B2.inst application ONLINE ONLINE hst1 ora....DB1.srv application ONLINE ONLINE hst2 ora....DB2.srv application ONLINE ONLINE hst1 ora....BTAF.cs application ONLINE ONLINE hst2 ora.MYDB.db application ONLINE ONLINE hst2 ora....B2.lsnr application ONLINE ONLINE hst2 ora.hst2.gsd application ONLINE ONLINE hst2 ora.hst2.ons application ONLINE ONLINE hst2 ora.hst2.vip application ONLINE ONLINE hst2 ora....B1.lsnr application ONLINE ONLINE hst1 ora.hst1.gsd application ONLINE ONLINE hst1 ora.hst1.ons application ONLINE ONLINE hst1 ora.hst1.vip application ONLINE ONLINE hst1 References [1] http://forums.oracle.com/forums/thread.jspa?messageID=2161448 [2] Metalink Note:276434.1 https://metalink.oracle.com/metalink/plsql/showdoc?db=NOT&id=276434.1&blackframe=1 [3] Metalink Note:283684.1 https://metalink.oracle.com/metalink/plsql/showdoc?db=NOT&id=283684.1&blackframe=1 (Account is required for Metalink access) Jaromir D.B. Nemec is a freelancer specializing in data warehouse and integration solutions. He can be reached under http://www.db-nemec.com Changing Hostnames in Oracle RAC By: Alex Gorbachev Update: this procedure is for Linux and should work on any UNIX OS. Ron supplied how he did this on Windows in the comment below. Thanks Ron. Sometimes there is a desperate need to change hostnames for one or all nodes of an Oracle RAC cluster. However, this operation is not officially supported. From Metalink Note 220970.1 RAC Frequently Asked Questions: Can I change the public hostname in my Oracle Database 10g Cluster using Oracle Clusterware? Hostname changes are not supported in Oracle Clusterware (CRS), unless you want to perform a deletenode followed by a new addnode operation. The hostname is used to store among other things the flag files and CRS stack will not start if hostname is changed. One way to do it is to remove a node from a cluster, change its hostname, and then add it back to the cluster as a new node. You will need to make sure that ORACLE_HOME is also added to this node as well as the database instance configuration. If you are brave enough, there is another way to do this. It’s not described anywhere on the Metalink, but there are no major hacks needed to implement it. The idea is to simply re-run the configuration of CRS (including re-formatting OCR and voting disks) and re-create the CRS resources after that. Obviously, this is not an online operation and the whole cluster is down for the duration of rename. I assume that we have an Oracle RAC cluster running, with database(s) running already, optionally including ASM instances. Capture Resource Definitions Before doing anything, we should capture resource definitions from the current CRS resources. This is an optional step, but it will simplify configuration later. A single resource definition can be captured with a command $ORA_CRS_HOME/bincrs_stat -p . Here is a small shell script to capture that for every resource and save it into a .cap file. As you will see later these files can be used to easily recreate resources: for res in `$ORA_CRS_HOME/bin/crs_stat -p | grep "^NAME=" | cut -d = -f 2` ; do $ORA_CRS_HOME/bin/crs_stat -p $res >/opt/oracle/resources/$res.cap done Here is an example list of files created: # ls -l /opt/oracle/resources total 52 -rw-r--r-- 1 root root 813 Feb 27 12:06 ora.A.A1.inst.cap -rw-r--r-- 1 root root 813 Feb 27 12:06 ora.A.A2.inst.cap -rw-r--r-- 1 root root 789 Feb 27 12:06 ora.A.db.cap -rw-r--r-- 1 root root 809 Feb 27 12:06 ora.vs10a.ASM1.asm.cap -rw-r--r-- 1 root root 799 Feb 27 12:06 ora.vs10a.gsd.cap -rw-r--r-- 1 root root 832 Feb 27 12:06 ora.vs10a.LISTENER_VS10A.lsnr.cap -rw-r--r-- 1 root root 804 Feb 27 12:06 ora.vs10a.ons.cap -rw-r--r-- 1 root root 828 Feb 27 12:06 ora.vs10a.vip.cap -rw-r--r-- 1 root root 809 Feb 27 12:06 ora.vs11a.ASM2.asm.cap -rw-r--r-- 1 root root 799 Feb 27 12:06 ora.vs11a.gsd.cap -rw-r--r-- 1 root root 832 Feb 27 12:06 ora.vs11a.LISTENER_VS11A.lsnr.cap -rw-r--r-- 1 root root 804 Feb 27 12:06 ora.vs11a.ons.cap -rw-r--r-- 1 root root 828 Feb 27 12:06 ora.vs11a.vip.cap Sample content from one file: # cat /opt/oracle/resources/ora.A.db.cap NAME=ora.A.db TYPE=application ACTION_SCRIPT=/opt/oracle/A/product/10.2.0/crs/bin/racgwrap ACTIVE_PLACEMENT=0 AUTO_START=1 CHECK_INTERVAL=600 DESCRIPTION=CRS application for the Database FAILOVER_DELAY=0 FAILURE_INTERVAL=60 FAILURE_THRESHOLD=1 HOSTING_MEMBERS= OPTIONAL_RESOURCES= PLACEMENT=balanced REQUIRED_RESOURCES= RESTART_ATTEMPTS=1 SCRIPT_TIMEOUT=600 START_TIMEOUT=0 STOP_TIMEOUT=0 UPTIME_THRESHOLD=7d USR_ORA_ALERT_NAME= USR_ORA_CHECK_TIMEOUT=0 USR_ORA_CONNECT_STR=/ as sysdba USR_ORA_DEBUG=0 USR_ORA_DISCONNECT=false USR_ORA_FLAGS= USR_ORA_IF= USR_ORA_INST_NOT_SHUTDOWN= USR_ORA_LANG= USR_ORA_NETMASK= USR_ORA_OPEN_MODE= USR_ORA_OPI=false USR_ORA_PFILE= USR_ORA_PRECONNECT=none USR_ORA_SRV= USR_ORA_START_TIMEOUT=0 USR_ORA_STOP_MODE=immediate USR_ORA_STOP_TIMEOUT=0 USR_ORA_VIP= Stop Clusterware Now we can stop Oracle Clusterware on all nodes using $ORA_CRS_HOME/bin/crsctl stop crs, and then change hostnames. Note that this will stop all databases, listeners, and other resources registered within CRS, so this is the time when outage starts. Rename Hosts I won’t discuss changing the hostname itself it here — it’s a straight-forward SA task. A few things to pay special attention to: Make sure that aliases in /etc/hosts are amended. Don’t forget to change aliases for VIPs and private IPs. This is not strictly required but you are better off following the standard naming convention (-priv and -vip for interconnect and virtual IP respectively) unless you have really good reason not to. Note that at this stage you should be also able to change IP addresses as well. I didn’t try it, but it should work. Also make sure DNS configuration is also changed by your SA, if your applications use DNS to resolve hostnames. Modify $ORA_CRS_HOME/install/rootconfig $ORA_CRS_HOME/install/rootconfig is called as part of the root.sh script run after Oracle Clusterware installation. We have to modify it so that it uses different host names. Generally, you would simply change every appearance of the old hostnames to the new hostnames. If you want to do that in vi, use :%s/old_node/new_node/g. Be careful not to change existing non-relevant parts of the script matching your old hostname. The variables that should be changed are CRS_HOST_NAME_LIST CRS_NODE_NAME_LIST CRS_PRIVATE_NAME_LIST CRS_NODELIST CRS_NODEVIPS The latter might need modification if you also change IPs. At this stage, you can also change your OCR and voting disks locations. The following lines should be changed: CRS_OCR_LOCATIONS={OCR path},{OCR mirror path} CRS_VOTING_DISKS={voting disk1 path},{voting disk2 path},{voting disk3 path} You can also change your cluster name via the variable CRS_CLUSTER_NAME. Cleanup OCR and Voting Disks You should clear OCR and voting disks, otherwise, the script will refuse to format them (well, it will unless you use a special force option, but that’s more hassle). This can be done using dd. In the example below I have mirrored OCR and 3 voting disks: dd if=/dev/zero of= bs=1024k dd if=/dev/zero of={OCR2 path} bs=1024k dd if=/dev/zero of={voting1 path} bs=1024k dd if=/dev/zero of={voting2 path} bs=1024k dd if=/dev/zero of={voting3 path} bs=1024k “Break” Clusterware Configuration rootconfig has some protection from idiots (as we say in Russia) built in — it checks if Clusterware has been already configured and, if it has, it exits without doing any harm. One way to “break” the configuration and make this script run for a second time is to delete the file /etc/oracle/ocr.loc. (Note that this is a Linux-specific location; other Unix variants might have different path. On HP-UX, for example, it’s something like /var/opt/oracle/ocr.log if I recall correctly.) Run $ORA_CRS_HOME/install/rootconfig If everything has gone alright, you should be able to run $ORA_CRS_HOME/install/rootconfig as the root user without any issues. If there are problems, follow the standard CRS troubleshooting procedure — checking /var/log/messages and $ORA_CRS_HOME/log/{nodename} et cetera. Note that this should be done on every node one by one — sequentially. On the last node of the cluster, the script will try to configure the VIPs, and there is a known bug here if you use a private range IP for VIP. This can be easily fixed by running $ORA_CRS_HOME/bin/vipca manually in graphical mode (i.e. you will need $DISPLAY configured correctly). Verify Clusterware Configuration and Status This is a simple check to make sure that all nodes are up and have VIP components configured correctly: [root@vs10 bin]# $ORA_CRS_HOME/bin/crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.vs10.gsd application ONLINE ONLINE vs10 ora.vs10.ons application ONLINE ONLINE vs10 ora.vs10.vip application ONLINE ONLINE vs10 ora.vs11.gsd application ONLINE ONLINE vs11 ora.vs11.ons application ONLINE ONLINE vs11 ora.vs11.vip application ONLINE ONLINE vs11 Adding Listener Resources to CRS There are two ways to do this — you can either use netca to configure the listener from scratch (you might need to clean it up from listener.ora first), or you can change the configuration manually and register it with CRS from command line. I’ll show how to that manually — obviously, the preferred way when it comes to the real environments. ;-) First of all, we will need to change the $ORACLE_HOME/network/admin/listener.ora file, and you will probably want to change tnsnames.ora at the same time. You need to replace old node aliases with new ones, and change the IPs if they are used instead of aliases, and if you changed them above during clusterware reconfiguration. Note that depending on how your LOCAL_LISTENER and REMOTE_LISTENER init.ora parameters are set, you might need to change them: if they reference connections descriptors from tnsname.ora, then only the latter should be changed, but if there are full connection descriptors, they should also be modified). You should also change listener names to reflect new hostnames. Usually, listeners are named as LISTENER_{hostname}, and you should keep this convention again unless you have a very good reason not to. Do that on both nodes if you don’t have a shared ORACLE_HOME. Now it’s time to get back to the .cap files with the CRS resource definitions we captured when we began. The files we are interested in are in format ora.{hostname}.LISTENER_{HOSTNAME}.lsnr.cap. In my case, one of them is ora.vs10a.LISTENER_VS10A.lsnr (my old hostname was vs10a). If you changed listener names above, you would need to amend it there as well — NAME=ora.vs10.LISTENER_VS10.lsnr, and rename the file according to the new host name following the same naming convention. Your VIP name has probably changed, so this line should be modified as well: REQUIRED_RESOURCES=ora.vs10.vip. And finally, the hosting member will change: HOSTING_MEMBERS=vs10. Check the whole file carefully — you should simply modify the old hostname to the new one in both lower and upper case. Now it’s time to register the resource — the crs_register command does just that. This command specifies the resource name to register and the directory where the .cap file is located. It should be named exactly like resource name plus a “.cap” extension. Each node’s listener can be added from the same node. It’s important that the content of the .cap file is modified appropriately. Assuming I have files ora.vs10.LISTENER_VS10.lsnr and ora.vs11.LISTENER_VS11.lsnr in directory /opt/oracle/A/resources, I run: $ORA_CRS_HOME/bin/crs_register ora.vs10.LISTENER_VS10.lsnr -dir /opt/oracle/A/resources $ORA_CRS_HOME/bin/crs_register ora.vs11.LISTENER_VS11.lsnr -dir /opt/oracle/A/resources Now the output from crs_stat -t should be: Name Type Target State Host ------------------------------------------------------------ ora....10.lsnr application OFFLINE OFFLINE ora.vs10.gsd application ONLINE ONLINE vs10 ora.vs10.ons application ONLINE ONLINE vs10 ora.vs10.vip application ONLINE ONLINE vs10 ora....11.lsnr application OFFLINE OFFLINE ora.vs11.gsd application ONLINE ONLINE vs11 ora.vs11.ons application ONLINE ONLINE vs11 ora.vs11.vip application ONLINE ONLINE vs11 It’s now time to start the listeners: $ORA_CRS_HOME/bin/srvctl start nodeapps -n vs10 $ORA_CRS_HOME/bin/srvctl start nodeapps -n vs11 crs_stat -t should show the listeners online: Name Type Target State Host ------------------------------------------------------------ ora....10.lsnr application ONLINE ONLINE vs10 ora.vs10.gsd application ONLINE ONLINE vs10 ora.vs10.ons application ONLINE ONLINE vs10 ora.vs10.vip application ONLINE ONLINE vs10 ora....11.lsnr application ONLINE ONLINE vs11 ora.vs11.gsd application ONLINE ONLINE vs11 ora.vs11.ons application ONLINE ONLINE vs11 ora.vs11.vip application ONLINE ONLINE vs11 Adding ASM Instances to CRS This step is optional, and it you don’t use ASM, skip it. Unfortunately, we can’t simply use .cap files to register ASM resources. There are more pieces required and the only way I could find to register ASM instances is to use srvctl which is, actually, a more supported option. This is simple: $ORACLE_HOME/bin/srvctl add asm -n vs10 -i ASM1 -o $ORACLE_HOME $ORACLE_HOME/bin/srvctl add asm -n vs11 -i ASM1 -o $ORACLE_HOME $ORACLE_HOME/bin/srvctl start asm -n vs10 $ORACLE_HOME/bin/srvctl start asm -n vs11 There is a catch — sometimes I had to prefix the name of the ASM instance with a “+” (i.e. making it like -i +ASM1) and sometimes no plus-sign was required. I couldn’t figure out why, so if you have a clue, let me know. crs_stat -t should show now: Name Type Target State Host ------------------------------------------------------------ ora....SM1.asm application ONLINE ONLINE vs10 ora....10.lsnr application ONLINE ONLINE vs10 ora.vs10.gsd application ONLINE ONLINE vs10 ora.vs10.ons application ONLINE ONLINE vs10 ora.vs10.vip application ONLINE ONLINE vs10 ora....SM2.asm application ONLINE ONLINE vs11 ora....11.lsnr application ONLINE ONLINE vs11 ora.vs11.gsd application ONLINE ONLINE vs11 ora.vs11.ons application ONLINE ONLINE vs11 ora.vs11.vip application ONLINE ONLINE vs11 Register Databases For each database, you need to register a database resource. Then, for every instance, you need to register an instance resource. So for database A, my two-node cluster, I use: $ORACLE_HOME/bin/srvctl add database -d A -o $ORACLE_HOME $ORACLE_HOME/bin/srvctl add instance -d A -i A1 -n vs10 $ORACLE_HOME/bin/srvctl add instance -d A -i A2 -n vs11 $ORACLE_HOME/bin/srvctl start database -d A Finally, crs_stat -t should show all resources online: Name Type Target State Host ------------------------------------------------------------ ora.A.A1.inst application ONLINE ONLINE vs10 ora.A.A2.inst application ONLINE ONLINE vs11 ora.A.db application ONLINE ONLINE vs10 ora....SM1.asm application ONLINE ONLINE vs10 ora....10.lsnr application ONLINE ONLINE vs10 ora.vs10.gsd application ONLINE ONLINE vs10 ora.vs10.ons application ONLINE ONLINE vs10 ora.vs10.vip application ONLINE ONLINE vs10 ora....SM2.asm application ONLINE ONLINE vs11 ora....11.lsnr application ONLINE ONLINE vs11 ora.vs11.gsd application ONLINE ONLINE vs11 ora.vs11.ons application ONLINE ONLINE vs11 ora.vs11.vip application ONLINE ONLINE vs11 Other Resources If you had other resources like services, user VIPs, or user-defined resources, you will probably be fine using the crs_register command to get them back into CRS. I didn’t try it, but it should work. Final Check To make sure that everything is working, you should at least reboot every node and see if everything comes up. I don’t know if that operation is considered to be supported. The only slippery bit is modifying the $ORA_CRS_HOME/install/rootconfig file, because it’s usually created by the Universal Installer. Another tricky place is the “unusual” registration of listeners. Otherwise, all the commands are pretty much usual stuff, I think. Good luck! Jun 11, 2007 Category: Group Blog Posts, Oracle Tags: 17 Responses Doug Burns says: June 12, 2007 at 7:22 am I recognise those hostnames ;-) Alex Gorbachev says: June 12, 2007 at 8:18 am I bet you do! ;-) Shu says: October 18, 2007 at 7:39 pm I had to change a few other files in $CRS_HOME before it works for me, such as ./inventory/Components21/oracle.has.crs/10.2.0.1.0/context.xml ./inventory/Components21/oracle.crs/10.2.0.1.0/installIP1Log.xml ./inventory/Components21/oracle.crs/10.2.0.1.0/context.xml ./inventory/ContentsXML/oraclehomeproperties.xml ./install/paramfile.crs But overall, this is a great recipe, played magics for me. Thanks a lot! Alex Gorbachev says: October 19, 2007 at 2:50 pm Glad to hear it worked for you, Shu. Thanks for feedback! Indeed, inventory needs to be changed as well. I don’t see you mentioned file {inventory_dir}/ContentsXML/inventory.xml - it has cluster nodes in it. P Kumar says: January 24, 2008 at 3:42 am Alex, I have tried your method and am running into a problem with the ASM on one node. I get a “PRKS 1009: Failed to start asm interface” error and when I try to start or stop it, I get a “unrecoverable failure. Human intervention required” message. The other node is fine. I have tried it a few times from scratch with the same result, even to the point of doing it in reverse and going back to the old hostnames. Everything comes up fine with the old hostnames. With the new hostnames, I consistently get this ASM error on one node. Can you think of anything that I might be doing wrong? Thanks PKumar Alex Gorbachev says: January 24, 2008 at 9:35 am PKumar, PRKS error doesn’t provide enough details to find out what’s wrong. First thing I would do is to try starting ASM instance manually - set your environment and use sqlplus. If that doesn’t work - you’d see the error. Maybe local_listener or remote_listener is set to an unresolvable value (I think just a hostname can be used with EZconnect) or something else host related. If that works fine, then you need to check the trace file for ora.{nodename}.asm resource. First, make sure you are using srvctl command from ASM home (if it’s different from database home). Than in that home, see the log file $ORACLE_HOME/log/{nodename}/racg/ora.{nodename}.ora.log. You should see the real reason behind the failure. If that doesn’t uncover enough, you can enable CRS tracing on that resource. The easy way to do that would be: crsctl debug log res ora.{nodename}.vip:1 Disable tracing: crsctl debug log res ora.node1.vip:0 Let me know how it goes. Good Luck! P Kumar says: February 2, 2008 at 12:31 pm Sorry could get nothing in the logs. I didn’t see any error anywhere other than when I tried to start it up. I couldn’t spend too much time on it and went ahead and changed the ASM instance to 3 instead of 1, and that worked. Do you know why that would happen? Is the ASM info stored somewhere, perhaps tied to the hostname, that I needed to clear? Mark J. Bobak says: July 10, 2008 at 12:50 pm Worked great, thanks Alex! rush2me says: November 14, 2008 at 3:43 pm Can you tell me about - change in ip adress? i installed orcle 10 g RAC- without RAC option(single node) on hp unix 11i v3. had to change its ip adress. now all seems ok only NETCA is not working.. am even able to create a database and connect to it… Alex Gorbachev says: November 16, 2008 at 9:28 pm rush2me: this article addresses RAC environment. It doesn’t apply to non-RAC installations and change of IP address in single node should be a matter of changing listener configuration unless you are using more advanced features. HunterX says: December 1, 2008 at 3:47 am Thank you for good stuff. I used this stuff to test with 11G RAC. HunterX says: December 2, 2008 at 10:57 am thank you again. I used idea on this article to change hostnames on RAC 10Gr2 (Sparc). Alex Gorbachev says: December 9, 2008 at 12:11 am Glad it helped. Please don’t forget that this is a “creative” way of doing it and might render your environment unsupported. On the other hand, pretty much all operations are “legitimate”. Ron says: December 16, 2008 at 12:54 pm Hi i have done the operation on a Oracle Window Cluster with the info you give us. They are some variant in the step. It work fine for us (change of all Ip address and subnet mask) it can be use for a hostname change like you have said. Thanks. Here we go (for a 2 nodes cluster 10.2.0.3) Stop all Oracle service on all nodes before the change of the address, the low format of the ocrcfg and vote disk change the address in the file host in C:\WINDOWS\system32\drivers\etc with the new address change the ip address in the window network definition connection with the new address ocopy \\.\ocrcfg d:\backup\ocrcfg.dmp ocopy \\.\votedsk1 d:\backup\votedsk1.dmp rem go to directory of the cluster binary crs\bin cd /D D:\Oracle\Ora10g\crs/bin rem Format raw ocrcfg and vote disk LogPartFormat.exe \\.\ocrcfg /q LogPartFormat.exe \\.\votedsk1 /q rem Run in prompt mode the cluster setup rem crssetup.exe config -nn %s_nodenumberlist% -pn %s_privatenamelist% -ch %ORA_CRS_HOME% -oh %s_OracleHome% -c %s_clustername% -l “%s_languageid%” -q %s_votedisklist% -g %s_usergroup% rem example for 2 node crssetup.exe config -nn host_name_node_1 ,1,host_name_node_2,2 -pn host_name_node_1-priv,1,host_name_node_2-priv,2 -ch D:\Oracle\Ora10g\crs -oh D:\Oracle\Ora10g\crs -c crs_lab -l “CANADIAN FRENCH_CANADA.WE8MSWIN1252″; -q “\\.\votedsk1″ rem ensures that all nodes are configured with the same port for the ONS racgons.exe add_config host_name_node_1:6200 host_name_node_2:6200 rem oifcfg to direct components to use specific network interfaces oifcfg.exe setif -global “Public”/10.200.5.0:public “Prive”/192.169.255.0:cluster_interconnect rem to rebuild the vip address vipca in wondow componant and make the incription in the cluster (in prompt mode or online) rem at the same time it will create the gsd and ons service vipca.bat -silent -nodelist “host_name_node_1,host_name_node_2″ -nodevips “host_name_node_1/host_name_node_1-vip/255.255.255.0/Public,host_name_node_2/host_name_node_2-vip/255.255.255.0/Public”; rem vipca rem check the state of the crs services (inscription) crs_stat -t rem now make the registration of the listener in the cluster. rem important don’t use the prompt command below, it will give you a crs-0223 error at the startup service Rem to solve this problem use netca on line rem crs_register ora.host_name_node_1.LISTENER_host_name_node_1.lsnr -dir D:\Oracle\Ora10g\crs\crs\profile rem crs_register ora.host_name_node_2.LISTENER_host_name_node_2.lsnr -dir D:\Oracle\Ora10g\crs\crs\profile rem srvctl start nodeapps -n host_name_node_1 ren srvctl start nodeapps -n host_name_node_2 netca rem check the state of the crs services (inscription) crs_stat -t rem start oracle object service and cluster volume before registering and accessing to asm services sc start †Oracle Object Service†host_name_node_1 sc start OracleClusterVolumeService host_name_node_1 sc start †Oracle Object Service†host_name_node_2 sc start OracleClusterVolumeService host_name_node_2 rem go the the rdbms directory to make the registration of asm and the database set ORACLE_HOME=D:\Oracle\Ora10g\rdbms rem add the service asm in the cluster in my case +asm1 and +asm2 for my 2 nodes srvctl add asm -n host_name_node_1 -i +ASM1 -o %ORACLE_HOME% srvctl add asm -n host_name_node_2 -i +ASM2 -o %ORACLE_HOME% rem start the asm service on each node srvctl start asm -n host_name_node_1 srvctl start asm -n host_name_node_2 rem check the state of the crs services (inscription) crs_stat -t rem go to the bin directory of rdbms directory to make the registration of the database cd /D D:\Logiciels\Oracle\Ora10g\rdbms ...