A script for Relocating and Restoring Database Services

Posted in: Technical Track

Due to complex requirements for service relocate and restore as part of patching, I decided to script the process.

Requirements:

Critical services should only be relocated once as part of patching.
After patching, non-critical services will need to be relocated to other instance.
Critical services are defined as svc_21-29.

Existing configuration:

[[email protected] patch]$ . /media/patch/hawk.env
The Oracle base has been set to /u01/app/oracle
ORACLE_UNQNAME=hawk
ORACLE_SID=hawk1
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/12.2.0.1/db1
Oracle Instance alive for sid "hawk1"
[[email protected] patch]$
[[email protected] patch]$ srvctl status database -d $ORACLE_UNQNAME -v
Instance hawk1 is running on node racnode-dc2-1 with online services svc_11,svc_12,svc_13,svc_14,svc_15,svc_16,svc_17,svc_18,svc_19. Instance status: Open.
Instance hawk2 is running on node racnode-dc2-2 with online services svc_21,svc_22,svc_23,svc_24,svc_25,svc_26,svc_27,svc_28,svc_29. Instance status: Open.
[[email protected] patch]$

Save existing services affinity:

[[email protected] patch]$ ./save_service.sh $ORACLE_UNQNAME
##### Save services affinity: /tmp/service_hawk1.conf
Instance hawk1 is running on node racnode-dc2-1 with online services svc_11,svc_12,svc_13,svc_14,svc_15,svc_16,svc_17,svc_18,svc_19. Instance status: Open.
##### Save services affinity: /tmp/service_hawk2.conf
Instance hawk2 is running on node racnode-dc2-2 with online services svc_21,svc_22,svc_23,svc_24,svc_25,svc_26,svc_27,svc_28,svc_29. Instance status: Open.
-r-------- 1 oracle oinstall 156 Jun  3 17:40 /tmp/service_hawk1.conf
-r-------- 1 oracle oinstall 156 Jun  3 17:40 /tmp/service_hawk2.conf
[[email protected] patch]$

Note: /tmp/service_hawk*.conf is read-only to prevent accidental overwrite.

Patching starts from instance1 and relocate all services to instance2.

[[email protected] patch]$ ./relocate_service.sh $ORACLE_UNQNAME 1 2
********************************
***** SERVICES CURRENT LOCATION:
********************************
Instance hawk1 is running on node racnode-dc2-1 with online services svc_11,svc_12,svc_13,svc_14,svc_15,svc_16,svc_17,svc_18,svc_19. Instance status: Open.
Instance hawk2 is running on node racnode-dc2-2 with online services svc_21,svc_22,svc_23,svc_24,svc_25,svc_26,svc_27,svc_28,svc_29. Instance status: Open.
******************************
***** CMD RELOCATE SERVICES TO: hawk2
******************************
+ srvctl relocate service -d hawk -service svc_11 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_12 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_13 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_14 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_15 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_16 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_17 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_18 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_19 -oldinst hawk1 -newinst hawk2
+ set +x
Instance hawk1 is running on node racnode-dc2-1. Instance status: Open.
Instance hawk2 is running on node racnode-dc2-2 with online services svc_11,svc_12,svc_13,svc_14,svc_15,svc_16,svc_17,svc_18,svc_19,svc_21,svc_22,svc_23,svc_24,svc_25,svc_26,svc_27,svc_28,svc_29. Instance status: Open.
[[email protected] patch]$

Patching completed from instance1.
Patching starts from instance2 and relocate all services to instance1.

[[email protected] patch]$ ./relocate_service.sh $ORACLE_UNQNAME 2 1
********************************
***** SERVICES CURRENT LOCATION:
********************************
Instance hawk1 is running on node racnode-dc2-1. Instance status: Open.
Instance hawk2 is running on node racnode-dc2-2 with online services svc_11,svc_12,svc_13,svc_14,svc_15,svc_16,svc_17,svc_18,svc_19,svc_21,svc_22,svc_23,svc_24,svc_25,svc_26,svc_27,svc_28,svc_29. Instance status: Open.
******************************
***** CMD RELOCATE SERVICES TO: hawk1
******************************
+ srvctl relocate service -d hawk -service svc_11 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_12 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_13 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_14 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_15 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_16 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_17 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_18 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_19 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_21 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_22 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_23 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_24 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_25 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_26 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_27 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_28 -oldinst hawk2 -newinst hawk1
+ set +x
+ srvctl relocate service -d hawk -service svc_29 -oldinst hawk2 -newinst hawk1
+ set +x
Instance hawk1 is running on node racnode-dc2-1 with online services svc_11,svc_12,svc_13,svc_14,svc_15,svc_16,svc_17,svc_18,svc_19,svc_21,svc_22,svc_23,svc_24,svc_25,svc_26,svc_27,svc_28,svc_29. Instance status: Open.
Instance hawk2 is running on node racnode-dc2-2. Instance status: Open.
[[email protected] patch]$

Patching is completed and all services are running from instance1.
Instead of relocating services back to its original instance, non-critical services are relocated to the other instance.
Essentially, svc_11-19 is relocated 3x while svc_21-29 is relocated 1x.

Non-Critical services were running from instance1 per /tmp/service_hawk1.conf when save_service.sh was run.
Non-Critical services are currently running from instance1 and will need to be restored to instance2.

[[email protected] patch]$ ./restore_service.sh $ORACLE_UNQNAME 1 2 /tmp/service_hawk1.conf
********************************
***** SERVICES CURRENT LOCATION:
********************************
Instance hawk1 is running on node racnode-dc2-1 with online services svc_11,svc_12,svc_13,svc_14,svc_15,svc_16,svc_17,svc_18,svc_19,svc_21,svc_22,svc_23,svc_24,svc_25,svc_26,svc_27,svc_28,svc_29. Instance status: Open.
Instance hawk2 is running on node racnode-dc2-2. Instance status: Open.
**************************
***** SAVED CONFIGURATION: /tmp/service_hawk1.conf
**************************
Instance hawk1 is running on node racnode-dc2-1 with online services svc_11,svc_12,svc_13,svc_14,svc_15,svc_16,svc_17,svc_18,svc_19. Instance status: Open.
*************************************
***** CMD - RESTORE SERVICES TO: hawk2
*************************************
+ srvctl relocate service -d hawk -service svc_11 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_12 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_13 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_14 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_15 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_16 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_17 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_18 -oldinst hawk1 -newinst hawk2
+ set +x
+ srvctl relocate service -d hawk -service svc_19 -oldinst hawk1 -newinst hawk2
+ set +x
Instance hawk1 is running on node racnode-dc2-1 with online services svc_21,svc_22,svc_23,svc_24,svc_25,svc_26,svc_27,svc_28,svc_29. Instance status: Open.
Instance hawk2 is running on node racnode-dc2-2 with online services svc_11,svc_12,svc_13,svc_14,svc_15,svc_16,svc_17,svc_18,svc_19. Instance status: Open.
[[email protected] patch]$

Hopefully, patching is not always this complex and if it is, maybe the scripts will be useful.

Scripts used:

[[email protected] patch]$ cat save_service.sh
#!/bin/sh -e
# MDinh : Mar 27, 2019
#
DN=`dirname $0`
BN=`basename $0`
DB=${1:?"---> USAGE: $DN/$BN <db_unique_name>"}
IFS=","
save_service ()
{
OUTF=/tmp/service_$i.conf
srvctl status instance -d ${DB} -instance ${i} -v > $OUTF
echo
echo "##### Save services affinity: $OUTF"
chmod 400 $OUTF
cat $OUTF
echo
}
#
# Get instande name from, e,g. Database instances: hawk1,hawk2
#
inst=$(srvctl config database -db ${DB} |grep "^Database instances" | awk -F' ' '{print $3}')
# Loop through instance
for i in ${inst}; do
save_service
done
ls -lh /tmp/service*.conf
exit
[[email protected] patch]$
[[email protected] patch]$ cat relocate_service.sh
#!/bin/sh
# relocate_service.sh
# MDinh : Jun 02, 2019
#
DN=`dirname $0`
BN=`basename $0`
DB=${1:?"---> USAGE: $DN/$BN <db_unique_name> <oldinst> <newinst>"}
OLD=${2:?"---> USAGE: $DN/$BN <db_unique_name> <oldinst> <newinst>"}
NEW=${3:?"---> USAGE: $DN/$BN <db_unique_name> <oldinst> <newinst>"}
echo
echo "********************************"
echo "***** SERVICES CURRENT LOCATION:"
echo "********************************"
echo
srvctl status database -d ${DB} -v
echo
echo "******************************"
echo "***** CMD RELOCATE SERVICES TO: ${DB}${NEW}"
echo "******************************"
echo
OUTF=/tmp/old_service_${DB}${OLD}.loc
srvctl status instance -d ${DB} -instance ${DB}${OLD} -v  > $OUTF
svc=$(tail -1 $OUTF| awk -F" " '{print $11}'| awk '{$0=substr($0,1,length($0)-1); print $0}')
IFS=","
for s in ${svc}
do
set -x
  srvctl relocate service -d ${DB} -service ${s} -oldinst ${DB}${OLD} -newinst ${DB}${NEW}
set +x
done
echo
srvctl status database -d ${DB} -v
echo
exit
[[email protected] patch]$
[[email protected] patch]$ cat restore_service.sh
#!/bin/sh
# restore_service.sh
# MDinh : Jun 02, 2019
#
DN=`dirname $0`
BN=`basename $0`
DB=${1:?"---> USAGE: $DN/$BN <db_unique_name> <oldinst> <newinst> <conf>"}
OLD=${2:?"---> USAGE: $DN/$BN <db_unique_name> <oldinst> <newinst> <conf>"}
NEW=${3:?"---> USAGE: $DN/$BN <db_unique_name> <oldinst> <newinst> <conf>"}
CONF=${4:?"---> USAGE: $DN/$BN <db_unique_name> <oldinst> <newinst> <conf>"}
echo
echo "********************************"
echo "***** SERVICES CURRENT LOCATION:"
echo "********************************"
echo
srvctl status database -d ${DB} -v
echo
echo "**************************"
echo "***** SAVED CONFIGURATION: $CONF"
echo "**************************"
echo
cat $CONF
echo
echo "*************************************"
echo "***** CMD - RESTORE SERVICES TO: ${DB}${NEW}"
echo "*************************************"
echo
svc=$(tail -1 $CONF| awk -F" " '{print $11}'| awk '{$0=substr($0,1,length($0)-1); print $0}')
IFS=","
for s in ${svc}
do
set -x
  srvctl relocate service -d ${DB} -service ${s} -oldinst ${DB}${OLD} -newinst ${DB}${NEW}
set +x
done
echo
srvctl status database -d ${DB} -v
echo
exit
[[email protected] patch]$
email
Want to talk with an expert? Schedule a call with our team to get the conversation started.

2 Comments. Leave new

Simple and nice scripts to do the work.

Reply
Michael Dinh
March 17, 2020 9:52 am

Much appreciated Cong.

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *