In my exadata environnment, it has 2 compute nodes, 3 cells, 3 VM clusters.
Let’s see how to add new VMs cluster .
First, add new IPs in DNS server. In my case:
exavm01clu04: 172.22.1.241
exavm02clu04: 172.22.1.242
exavm01clu04-vip:172.22.1.244
exavm02clu04-vip:172.22.1.243
scan: 172.22.1.245
172.22.1.246
172.22.1.247
Second, downloading new OEDA(Patch 30640393) , start oeda server and create new VM config file (tvd-secured-3a-exa-clu4.xml).
Then, copy new VM confoig file (tvd-secured-3a-exa-clu4.xml) to dom0 (/opt/oracle.SupportTools/onecommand/linux-x64).
Follow below steps to create new VM cluster.
- Validate Configuration File
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 1 Initializing ===== Executing Validate Configuration File ===== Validating cluster: clu4 Locating machines... Validating cluster networks... Verifying operating systems... Validating network connectivity... Validating private ips on virtual cluster Validating guest private networks ||||| Validating NTP setup... Validating physical disks on storage cells... Validating users... Validating platinum... Validating switches... Validating quorum disk requirements... Checking disk reclaim status... Checking Disk Tests Status.... Completed validation... SUCCESS: Ip address: 172.22.1.122 is configured correctly SUCCESS: Ip address: 172.22.1.241 is configured correctly SUCCESS: Ip address: 172.22.1.123 is configured correctly SUCCESS: Ip address: 172.22.1.242 is configured correctly SUCCESS: Ip address: 172.22.1.247 is configured correctly SUCCESS: Ip address: 172.22.1.245 is configured correctly SUCCESS: Ip address: 172.22.1.246 is configured correctly SUCCESS: Ip address: 172.22.1.243 is configured correctly SUCCESS: Ip address: 172.22.1.244 is configured correctly SUCCESS: Validated NTP server 172.22.1.9 SUCCESS: Required System image file /EXAVMIMAGES/System.first.boot.21.2.3.0.0.210817.img found on exadbadm01.oracle.democenter.myexadata.ch SUCCESS: Required System image file /EXAVMIMAGES/System.first.boot.21.2.3.0.0.210817.img found on exadbadm02.oracle.democenter.myexadata.ch SUCCESS: Required image file db-klone-Linux-x86-64-19000211019.zip found on exadbadm02.oracle.democenter.myexadata.ch SUCCESS: Required image file db-klone-Linux-x86-64-19000211019.zip found on exadbadm02.oracle.democenter.myexadata.ch SUCCESS: Required image file grid-klone-Linux-x86-64-19000211019.zip found on exadbadm01.oracle.democenter.myexadata.ch SUCCESS: Required image file grid-klone-Linux-x86-64-19000211019.zip found on exadbadm01.oracle.democenter.myexadata.ch SUCCESS: Required file /EXAVMIMAGES/onecommand/linux-x64/WorkDir/ahf_setup exists... SUCCESS: Found Operating system LinuxPhysical and matches expected Operating system in AWS SUCCESS: Found Operating system LinuxPhysical and matches expected Operating system in AWS SUCCESS: Found Operating system LinuxPhysical and matches expected Operating system in AWS SUCCESS: Expected machine exadbadm02.oracle.democenter.myexadata.ch to have OS Type of Linux Dom0/KVM host, and found OsType LinuxPhysical SUCCESS: Expected machine exadbadm01.oracle.democenter.myexadata.ch to have OS Type of Linux Dom0/KVM host, and found OsType LinuxPhysical SUCCESS: SUCCESS: NTP servers on machine exaceladm03.oracle.democenter.myexadata.ch verified successfully SUCCESS: NTP servers on machine exaceladm02.oracle.democenter.myexadata.ch verified successfully SUCCESS: NTP servers on machine exaceladm01.oracle.democenter.myexadata.ch verified successfully SUCCESS: NTP servers on machine exadbadm02.oracle.democenter.myexadata.ch verified successfully SUCCESS: NTP servers on machine exadbadm01.oracle.democenter.myexadata.ch verified successfully SUCCESS: Sufficient memory for all the guests on database node exadbadm02.oracle.democenter.myexadata.ch SUCCESS: Sufficient memory for all the guests on database node exadbadm01.oracle.democenter.myexadata.ch SUCCESS: Switch IP 172.22.1.112 resolves successfully to host exasw-iba0.oracle.democenter.myexadata.ch on node exadbadm02.oracle.democenter.myexadata.ch SUCCESS: Switch IP 172.22.1.113 resolves successfully to host exasw-ibb0.oracle.democenter.myexadata.ch on node exadbadm02.oracle.democenter.myexadata.ch SUCCESS: Switch IP 172.22.1.113 resolves successfully to host exasw-ibb0.oracle.democenter.myexadata.ch on node exadbadm01.oracle.democenter.myexadata.ch SUCCESS: Switch IP 172.22.1.112 resolves successfully to host exasw-iba0.oracle.democenter.myexadata.ch on node exaceladm03.oracle.democenter.myexadata.ch SUCCESS: Switch IP 172.22.1.113 resolves successfully to host exasw-ibb0.oracle.democenter.myexadata.ch on node exaceladm03.oracle.democenter.myexadata.ch SUCCESS: Switch IP 172.22.1.112 resolves successfully to host exasw-iba0.oracle.democenter.myexadata.ch on node exadbadm01.oracle.democenter.myexadata.ch SUCCESS: Switch IP 172.22.1.112 resolves successfully to host exasw-iba0.oracle.democenter.myexadata.ch on node exaceladm01.oracle.democenter.myexadata.ch SUCCESS: Switch IP 172.22.1.113 resolves successfully to host exasw-ibb0.oracle.democenter.myexadata.ch on node exaceladm01.oracle.democenter.myexadata.ch SUCCESS: Switch IP 172.22.1.112 resolves successfully to host exasw-iba0.oracle.democenter.myexadata.ch on node exaceladm02.oracle.democenter.myexadata.ch SUCCESS: Switch IP 172.22.1.113 resolves successfully to host exasw-ibb0.oracle.democenter.myexadata.ch on node exaceladm02.oracle.democenter.myexadata.ch SUCCESS: # of nodes matches quorum disk requirement - computes2 storage 3 SUCCESS: Disk Tests are not running/active on any of the Storage Servers. SUCCESS: Required Kernel Version 4.1.12.124.52.4 for Oracle19c found on exadbadm01 SUCCESS: Required Kernel Version 4.1.12.124.52.4 for Oracle19c found on exadbadm02 SUCCESS: Disk size 10000GB on cell exaceladm01.oracle.democenter.myexadata.ch matches the value specified in the OEDA configuration file SUCCESS: Disk size 10000GB on cell exaceladm02.oracle.democenter.myexadata.ch matches the value specified in the OEDA configuration file SUCCESS: Disk size 10000GB on cell exaceladm03.oracle.democenter.myexadata.ch matches the value specified in the OEDA configuration file SUCCESS: Number of physical disks on exaceladm01.oracle.democenter.myexadata.ch matches the value specified in OEDA configuration file SUCCESS: Number of physical disks on exaceladm02.oracle.democenter.myexadata.ch matches the value specified in OEDA configuration file SUCCESS: Number of physical disks on exaceladm03.oracle.democenter.myexadata.ch matches the value specified in OEDA configuration file Following warnings were found... WARNING: The version of OEDA currently in use has not been updated in over 45 days. Please download and use the latest OEDA version. WARNING: The OEDA version being used for deployment is older than the version that was used to create the configuration XML file. Please use the same OEDA versions. ===== Successfully completed execution of step Validate Configuration File [elapsed Time [Elapsed = 124452 mS [2.0 minutes] Wed Jan 19 15:59:32 CET 2022]] =====
2. Create Virtual Machine
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 2 Initializing ===== Executing Create Virtual Machine ===== Creating guests for cluster clu4 Checking status of remote files... Checking status of existing files on remote nodes... Getting status of local files... Checking versions on IB switches ... Creating guest using /EXAVMIMAGES/conf/final-exavm02clu04.oracle.democenter.myexadata.ch-vm.xml on node exadbadm02.oracle.democenter.myexadata.ch Creating guest using /EXAVMIMAGES/conf/final-exavm01clu04.oracle.democenter.myexadata.ch-vm.xml on node exadbadm01.oracle.democenter.myexadata.ch Guests created, waiting for guests to start... Waiting for machine exavm01clu04.oracle.democenter.myexadata.ch to be pingable... Waiting for machine exavm02clu04.oracle.democenter.myexadata.ch to be pingable... Machine exavm01clu04.oracle.democenter.myexadata.ch is pingable, waiting for login... Machine exavm02clu04.oracle.democenter.myexadata.ch is pingable, waiting for login... Successfully logged into machine exavm01clu04.oracle.democenter.myexadata.ch Successfully logged into machine exavm02clu04.oracle.democenter.myexadata.ch Creating cellroute.ora for OVMs Restarting machines... Going to reboot guests from dom0 [exavm01clu04.oracle.democenter.myexadata.ch, exavm02clu04.oracle.democenter.myexadata.ch] Waiting for machine exavm02clu04.oracle.democenter.myexadata.ch to be NOT pingable... Waiting for machine exavm01clu04.oracle.democenter.myexadata.ch to be NOT pingable... Starting guests [exavm01clu04.oracle.democenter.myexadata.ch, exavm02clu04.oracle.democenter.myexadata.ch]... Waiting for machine exavm01clu04.oracle.democenter.myexadata.ch to be pingable... Waiting for machine exavm02clu04.oracle.democenter.myexadata.ch to be pingable... Machine exavm01clu04.oracle.democenter.myexadata.ch is pingable, waiting for login... Machine exavm02clu04.oracle.democenter.myexadata.ch is pingable, waiting for login... Successfully logged into machine exavm01clu04.oracle.democenter.myexadata.ch Successfully logged into machine exavm02clu04.oracle.democenter.myexadata.ch No config Keys in the configuration file.. Creating databasemachine.xml for EM discovery Done Creating databasemachine.xml for EM discovery Verifying Cell Services on Storage Servers.... ===== Successfully completed execution of step Create Virtual Machine [elapsed Time [Elapsed = 1716785 mS [28.0 minutes] Wed Jan 19 19:36:26 CET 2022]] =====
3. Create Users
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 3 Initializing ===== Executing Create Users ===== Validating existing users and groups... ===== Successfully completed execution of step Create Users [elapsed Time [Elapsed = 5316 mS [0.0 minutes] Wed Jan 19 20:13:04 CET 2022]] =====
4. Setup Cell Connectivity
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 4 Initializing ===== Executing Setup Cell Connectivity ===== Updating /etc/hosts on nodes in cluster clu4 Creating cellip.ora and cellinit.ora ... Creating cellip.ora for cluster clu4 Creating cellinit.ora for cluster clu4 Done creating cellip.ora and cellinit.ora... ===== Successfully completed execution of step Setup Cell Connectivity [elapsed Time [Elapsed = 2970 mS [0.0 minutes] Wed Jan 19 20:15:36 CET 2022]] =====
5. Calibrate Cells
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 5 Initializing ===== Executing Calibrate Cells ===== Calibrating cells... ===== Successfully completed execution of step Calibrate Cells [elapsed Time [Elapsed = 480424 mS [8.0 minutes] Wed Jan 19 20:26:08 CET 2022]] =====
6. Create Cell Disks
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 6 Initializing ===== Executing Create Cell Disks ===== Validating Self-Signed Certificates on cell servers... Checking physical disks for errors before creating celldisks Creating cell disks... WriteBack flash cache mode has already been enabled. ===== Successfully completed execution of step Create Cell Disks [elapsed Time [Elapsed = 58351 mS [0.0 minutes] Wed Jan 19 20:30:28 CET 2022]] =====
7. Create Grid Disks
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 7 Initializing ===== Executing Create Grid Disks ===== Creating grid disks for cluster clu4 Checking Cell Disk status... Creating quorum devices in cluster clu4 Configuring ASM-Scoped Security on cluster clu4 Configuring ASM Security key on the cell servers [exaceladm01.oracle.democenter.myexadata.ch, exaceladm02.oracle.democenter.myexadata.ch, exaceladm03.oracle.democenter.myexadata.ch] Setting up cellkey.ora on compute nodes.. Securing Grid Disks.... ===== Successfully completed execution of step Create Grid Disks [elapsed Time [Elapsed = 197052 mS [3.0 minutes] Wed Jan 19 20:36:01 CET 2022]] =====
8. Install Cluster Software
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 8 Initializing ===== Executing Install Cluster Software ===== Setting up cluster users... Setting up users in cluster clu4 Validating existing users and groups... Creating required directories on nodes in cluster clu4 Setting up ssh for users in cluster clu4 Creating cell diag collection user CELLDIAG on cell servers.. Completed setting up all users... Installing cluster clu4 ===== Successfully completed execution of step Install Cluster Software [elapsed Time [Elapsed = 97733 mS [1.0 minutes] Thu Jan 20 09:24:02 CET 2022]] =====
9. Initialize Cluster Software
-----[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 9 Initializing ===== Executing Initialize Cluster Software ===== Initializing cluster clu4 Getting grid disks using utility in /u01/app/19.0.0.0/grid/bin Writing grid response file for cluster clu4 Getting grid disks using utility in /u01/app/19.0.0.0/grid/bin Running root.sh on node exavm01clu04.oracle.democenter.myexadata.ch Checking file root_exavm01clu04.oracle.democenter.myexadata.ch_2022-01-20_09-36-04-108033583.log on node exavm01clu04.oracle.democenter.myexadata.ch Running root.sh on node exavm02clu04.oracle.democenter.myexadata.ch Checking file root_exavm02clu04.oracle.democenter.myexadata.ch_2022-01-20_09-44-03-615051416.log on node exavm02clu04.oracle.democenter.myexadata.ch Generating response file for Configuration Tools... Getting grid disks using utility in /u01/app/19.0.0.0/grid/bin Writing grid response file for cluster clu4 Getting grid disks using utility in /u01/app/19.0.0.0/grid/bin Running Configuration Assistants on exavm01clu04.oracle.democenter.myexadata.ch Checking status of cluster... Cluster Verification completed successfully ===== Successfully completed execution of step Initialize Cluster Software [elapsed Time [Elapsed = 1141259 mS [19.0 minutes] Thu Jan 20 09:50:14 CET 2022]] =====
10. Install Database Software
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 10 Initializing ===== Executing Install Database Software ===== Installing database software ... Installing database software with database home name clu4_dbh1 ===== Successfully completed execution of step Install Database Software [elapsed Time [Elapsed = 87758 mS [1.0 minutes] Thu Jan 20 10:00:16 CET 2022]] =====
11. Relink Database with RDS
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 11 Initializing ===== Executing Relink Database with RDS ===== ===== Successfully completed execution of step Relink Database with RDS [elapsed Time [Elapsed = 35062 mS [0.0 minutes] Thu Jan 20 10:03:08 CET 2022]] =====
12. Create ASM Diskgroups
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 12 Initializing ===== Executing Create ASM Diskgroups ===== Getting grid disks using utility in /u01/app/19.0.0.0/grid/bin Validating ASM Diskgroups.. ===== Successfully completed execution of step Create ASM Diskgroups [elapsed Time [Elapsed = 72411 mS [1.0 minutes] Thu Jan 20 10:04:50 CET 2022]] =====
13. Create Databases
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 13 Initializing ===== Executing Create Databases ===== Setting up Huge Pages for Database..[vm4cdb1] Creating database [vm4cdb1]... Running datapatch on database [vm4cdb1] Validating Datapatch.. Datapatch validation completed successfullly Restarting Cluster database [vm4cdb1] Recompiling Invalid Objects (if any) on database [vm4cdb1] ===== Successfully completed execution of step Create Databases [elapsed Time [Elapsed = 2731731 mS [45.0 minutes] Thu Jan 20 10:53:20 CET 2022]] =====
14. Create Pluggable Databases
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 14 Initializing ===== Executing Create Pluggable Databases ===== Creating Pdb database... Creating non-default Service vm4cdb1_pdb1_svc for [pdb1] Starting Pdb Service vm4cdb1_pdb1_svc Configuring Temporary Tablespace on PDB [pdb1] ===== Successfully completed execution of step Create Pluggable Databases [elapsed Time [Elapsed = 168291 mS [2.0 minutes] Thu Jan 20 11:22:15 CET 2022]] =====
15. Apply Security Fixes
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 15 Initializing ===== Executing Apply Security Fixes ===== Checking database [vm4cdb1] status on cluster clu4 Setting up Huge Pages for ASM Instance.. Bouncing clusterware to set required parameters... Going to reboot guests from dom0 [exavm01clu04.oracle.democenter.myexadata.ch, exavm02clu04.oracle.democenter.myexadata.ch] Waiting for machine exavm01clu04.oracle.democenter.myexadata.ch to be NOT pingable... Waiting for machine exavm02clu04.oracle.democenter.myexadata.ch to be NOT pingable... Starting guests [exavm01clu04.oracle.democenter.myexadata.ch, exavm02clu04.oracle.democenter.myexadata.ch]... Waiting for machine exavm01clu04.oracle.democenter.myexadata.ch to be pingable... Waiting for machine exavm02clu04.oracle.democenter.myexadata.ch to be pingable... Machine exavm02clu04.oracle.democenter.myexadata.ch is pingable, waiting for login... Machine exavm01clu04.oracle.democenter.myexadata.ch is pingable, waiting for login... Successfully logged into machine exavm02clu04.oracle.democenter.myexadata.ch Successfully logged into machine exavm01clu04.oracle.democenter.myexadata.ch Checking and enabling turbo mode if required on exavm01clu04.oracle.democenter.myexadata.ch,exavm02clu04.oracle.democenter.myexadata.ch... exavm01clu04.oracle.democenter.myexadata.ch Command: /opt/oracle.SupportTools/fix_17898503_Enable_Turbo_Mode.sh produced null output but executed successfully on exavm01clu04.oracle.democenter.myexadata.ch exavm02clu04.oracle.democenter.myexadata.ch Command: /opt/oracle.SupportTools/fix_17898503_Enable_Turbo_Mode.sh produced null output but executed successfully on exavm02clu04.oracle.democenter.myexadata.ch Copying file tvd-secured-3a-exa-clu4.xml to /etc/exadata/config/tvd-secured-3a-exa-clu4_220120_114513.xml on nodes in [exaceladm01.oracle.democenter.myexadata.ch, exaceladm02.oracle.democenter.myexadata.ch, exaceladm03.oracle.democenter.myexadata.ch, exadbadm01.oracle.democenter.myexadata.ch, exadbadm02.oracle.democenter.myexadata.ch] ===== Successfully completed execution of step Apply Security Fixes [elapsed Time [Elapsed = 802908 mS [13.0 minutes] Thu Jan 20 11:45:17 CET 2022]] =====
16. Install Autonomous Health Framework
[root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 16 Initializing ===== Executing Install Autonomous Health Framework ===== Copying over AHF to all nodes in the Cluster..[exadbadm01] Copying over AHF to all nodes in the Cluster..[exadbadm02] Configuring Autonomous Health Framework(AHF) on all computes nodes.. Configuring Autonomous Health Framework(AHF) on all computes nodes.. AHF has been installed on all compute nodes at: /opt/oracle.ahf . EXAchk can be run by invoking ./exachk AHF has been installed on all compute nodes at: /opt/oracle.ahf . EXAchk can be run by invoking ./exachk Copying over AHF to all nodes in the Cluster..[exavm01clu04, exavm02clu04] Configuring Autonomous Health Framework(AHF) on all computes nodes.. AHF has been installed on all compute nodes at: /opt/oracle.ahf . EXAchk can be run by invoking ./exachk Generating an EXAchk report... EXAchk zip file in exavm01clu04:/u01/app/oracle/oracle.ahf/data/exavm01clu04/exachk/user_root/output/exachk_exavm01clu04_vm4cdb1_012022_11517.zip Generating the EXAchk Infrastructure Report... EXAchk zip file in exadbadm01:/opt/oracle.ahf/data/exadbadm01/exachk/exachk_exadbadm01_012022_120332_infrastructure.zip ===== Successfully completed execution of step Install Autonomous Health Framework [elapsed Time [Elapsed = 1574291 mS [26.0 minutes] Thu Jan 20 12:14:49 CET 2022]
17. Create Installation Summary
root@exadbadm01 linux-x64]# ./install.sh -cf tvd-secured-3a-exa-clu4.xml -s 17 Initializing ===== Executing Create Installation Summary ===== Getting system details... Generating Installation Summary report: /EXAVMIMAGES/onecommand/linux-x64/ExadataConfigurations/tvd-secured-3a-3a-InstallationReport.xml... Creating Installation template /EXAVMIMAGES/onecommand/linux-x64/ExadataConfigurations/tvd-secured-3a-InstallationTemplate.html... Created Installation template /EXAVMIMAGES/onecommand/linux-x64/ExadataConfigurations/tvd-secured-3a-InstallationTemplate.html All deployment reports are stored in /EXAVMIMAGES/onecommand/linux-x64/ExadataConfigurations/tvd-secured-3a-UNKNOWN-deploymentfiles.zip Generating Platinum CSV file and copying it over to /opt/oracle.SupportTools on all compute nodes Writing platinum file : /EXAVMIMAGES/onecommand/linux-x64/WorkDir/tvd-secured-3a_null-platinum.csv ===== Successfully completed execution of step Create Installation Summary [elapsed Time [Elapsed = 13425 mS [0.0 minutes] Thu Jan 20 13:31:27 CET 2022]] =====
Oracle DBA Insights 