O ha ha ha ha ha ha ha ha ha ha ha ha ha

[weixin_38067396]

也没有作成盘陈，
不过也出来了，
不太会用Gsx.
ORACLE9i ™ REAL APPLICATION CLUSTER
RELEASE 2 INSTALLATION WITH A NETAPP®
FILER IN RED HAT ADVANCED SERVER™
LINUX® ENVIRONMENT
Sunil Mahale, Network Appliance
Vasu Subbiah, Network Appliance
Petros Xides, Oracle Corporation
September 2002 - TR3189
The alliance is based on a common vision—
reducing complexity and providing simplicity
for greater reliability, a better return on
investment (ROI), and a lower total cost of
ownership (TCO).
oracle and
netapp alliance
Table of Contents
Introduction 1
Assumption 1
The Server/Filer Environment 1
Validated Configurations 2
Cluster Nodes 2
Storage Infrastructure 2
Software Requirements 2
Preinstallation Setup Tasks (For Network Appliance Filer) 2
Preinstallation Setup Tasks (For Cluster Nodes) 3
Installation Procedure 5
Known Issues and Workarounds 10
Disclaimer 11
technical report
Introduction
This technical report will cover the installation of a NetApp filer and Intel architecture-based
enterprise-class servers using Red Hat Advanced Server 2.1, Linux operating system running
Oracle’s 9i Real Application Cluster, Release 2 (hereafter referred to as Oracle9i RAC). This is a
certified configuration and, as such, the components presented in this paper have to be used in
the same combination to gain support from all parties involved. The only exception to this is the
application of certain patches (as defined and required by all the vendors in this configuration).
Assumption
We assume that you are familiar with Oracle9i Release 2 and the operation of Network Appliance™
filers. We also assume that you are familiar with the operation of the Linux OS and that you are
familiar with installation of Oracle® patches and any relevant Linux patches. It is also important
to be familiar with all networking terminology and implementations.
The Server/Filer Environment
The configuration presented in this document is based on the Oracle9i RAC certification
environment specified by Oracle and Network Appliance.
Figure 1)
Oracle9i RAC on Red Hat Advanced Server with a Network Appliance filer.
Figure 1 illustrates a typical configuration of Oracle9i RAC with a NetApp filer and Intel
architecture-based enterprise-class servers running Red Hat Advanced Server, 2.1. This is a
scalable configuration and allows the user to scale horizontally and internally in terms of
processor, memory, and storage.
As shown in the network diagram, we recommend that you dedicate a private network connection
between the Oracle9i RAC servers and the filer. This is accomplished using a dedicated gigabit
network (with a gigabit switch) to the filer. A dedicated network connection is beneficial for the
following reasons:
1
technical report
DATA1
(Filer 880)
10/100 Switch
CPQRH1
Gigabit Switch
Private Network Eth 1 Eth 1
Eth 0 Eth 0
Eth 2: Heartbeat
E5
Dedicated Gigabit
Network to the Filer
Cluster Interconnect Network
CPQRH2
Public Network
• In a 9i RAC environment it is important to eliminate any contentions and latencies.
• By providing a separate network, security is ensured.
The cluster interconnect is used to monitor the heartbeat of the two servers in the cluster.
This is a typical configuration that can be deployed in a customer’s environment.
Validated Configurations
Cluster Nodes
• 2 x 2-way 900 MHz, 2GB RAM Intel architecture-based enterprise-class servers
• 2 x Intel III Gigabit Ethernet NICs (for cluster interconnect)
• 2 x Intel III Gigabit Ethernet NICs (for Network Appliance filer I/O)
Storage Infrastructure
• 1 x Network Appliance 880 filer with Data ONTAP™ 6.1.3R2
• 1 x 4/8 port gigabit switch
• 1 x gigabit NIC in the filer
• 1 or more disk shelves, based on the disk space requirements
Software Requirements
(For both nodes in the participating cluster unless specified otherwise)
• Red Hat Advanced Server 2.1, kernel 2.4.9-e.3
• Oracle9i, Release 2 (9.2.0.1) with Real Application Cluster license
• Intel III Gigabit Ethernet driver for the gigabit NICs
Preinstallation Setup Tasks (For Network Appliance Filer)
(For more information, refer to Network Appliance installation and setup guides:
http://now.netapp.com )
1. Install the filer and connect the disk shelves to it.
2. Make sure each shelf has a unique ID. This should be on the back of the shelf.
3. Connect the shelves to the Fibre Channel Adapter in a daisy chain. The output of one shelf
should be connected to the input of the other. The last shelf’s output should be terminated.
The input to the first shelf should come from the Fibre Channel Adapter in the filer.
4. Install Data ONTAP 6.1.3R2 and configure the filer by giving it host names such as Data1.
5. Install the NFS and SnapRestore® license keys.
2
technical report
6. Configure the gigabit NIC in the filer.
a. Ensure that the gigabit NIC is installed in the filer. This document assumes the NIC is
inserted in slot 5 in the filer Data1.
b. Configure the gigabit interfaces: i.e., e5 on Data1.
c. Specify the following IP address for the interface: (Run the "setup" command on the
filer console to configure the network interfaces.)
Data1>;e5 - ip: 10.1.1.103, netmask: 255.255.255.0
7. Create and export volumes for storing Oracle database files on the filers:
a. Create a volume on Data1 filer for storing Oracle database files. This volume will be
used to store all the data files, control files, and log files for the Oracle database. Specify
the default RAID group size as 14. Use the following commands at the filer console.
Data1>; vol create ordata -r 14 16
b. Edit the /etc/exports file on Data1 and add the following entries to that file:
/vol/ordata -anon=0
c. Execute the following command at the filer console:
Data1>; exportfs -a
Preinstallation Setup Tasks (For Cluster Nodes)
This information is intended for both the nodes in the participating cluster unless
specified otherwise.
1. Install the Intel III gigabit NICs in the cluster nodes (two per node).
a. Connect one gigabit NIC back-to-back to the other node for cluster interconnects.
b. Connect one gigabit NIC to the gigabit switch, which will connect to the gigabit NIC
on the filer.
2. Install Red Hat Advanced Server 2.1, kernel 2.4.9-e.3, on the cluster nodes. (The cluster
nodes in this document will be referred to as cpqrh1 and cpqrh2). Refer to Red Hat
Advanced Server installation manuals for more details.
3. Configure the two gigabit NICs on cpqrh1 and cpqrh2 using Red Hat Advanced Server LAN
management tools.
a. Configure the two network interfaces and specify the following IP addresses:
cpqrh1
eth1 - ip: 10.1.1.101, netmask: 255.255.255.0
eth2 - ip: 20.1.1.101, netmask: 255.255.255.0
3
technical report
cpqrh2
eth1 - ip: 10.1.1.102, netmask: 255.255.255.0
eth2 - ip: 20.1.1.102, netmask: 255.255.255.0
Where:
Interface eth1 on both the cluster nodes is connected to the gigabit switch for filer I/O.
Interface eth2 on both the cluster nodes is connected back-to-back for cluster interconnects.
b. Update the /etc/hosts file on the cluster nodes and add the following entries:
cpqrh1
cpqrh1-eth1 10.1.1.101
cpqrh1-eth2 20.1.1.101
data1 10.1.1.103
cpqrh2-eth1 10.1.1.102
cpqrh2-eth2 20.1.1.102
cpqrh2
cpqrh1-eth1 10.1.1.101
cpqrh1-eth2 20.1.1.101
data1 10.1.1.103
cpqrh2-eth1 10.1.1.102
cpqrh2-eth2 20.1.1.102
4. Make sure the eth1 interface is connected to the switch and can communicate with the filer
Data1. Use the /usr/sbin/ping command to verify the connectivity to the filer over the
gigabit network from both the cluster nodes, i.e., cpqrh1 and cpqrh2.
$/usr/sbin/ping data1
5. The other interface, eth2 on cpqrh1 should be able to communicate with its counterpart,
eth2 on cpqrh2. Use the /usr/sbin/ping command to verify the cluster interconnect
from both the cluster nodes, i.e., cpqrh1 and cpqrh2.
$/usr/sbin/ping cpqrh1-eth2
$/usr/sbin/ping cpqrh2-eth2
cpqrh1-eth2 and cpqrh2-eth2 will be used as private node names for the cluster interconnects.
These interface names will be used for HostName and PrivateNodeNames
parameters in the cluster configuration file, cmcfg.ora, for Oracle9i RAC.
6. Create mount points and mounting volumes.
a. Update the /etc/fstab file on all the server nodes and add the following entries:
4
technical report
data1:/vol/ordata - /oradb nfs - yes
rw,bg,hard,intr,rsize=32768,wsize=32768, tcp, noac,vers=3
Where:
Data1 is the name of the filer
oradb is the mount point on the cluster nodes
The mount options that are required for Oracle9i RAC are:
noac: This mount option disables caching on the client side
tcp: Mount the file system using the tcp option
b. Create the following mount point:
#mkdir /oradb
c. Mount the exported volume on the mount point created above.
#mount /oradb
Installation Procedure
(For more information about installing Oracle9i Release 2, on Linux, refer to Oracle9i Installation
Guide Release 2 (9.2.0.1) for UNIX Systems [Part No:A96167-01] at
http://otn.oracle.com/docs/content.html )
1. Preparing to install Oracle9i RAC on cluster nodes.
a. This document assumes the Oracle user account and the group to be "oracle" and
"dba" respectively, on both the cluster nodes, cpqrh1 and cpqrh2. The user ID and
group name for the "oracle" account should be same on both the cluster nodes.
b. The $ORACLE_HOME is not shared across the cluster nodes. The $ORACLE_HOME
directory is local to the cluster nodes. Only the $ORACLE_HOME/dbs directory is
shared on the cluster nodes.
c. Grant appropriate permissions to the "oracle" user on the shared mounted volume,
/oradb.
#chown -R oracle:dba /oradb
#chmod -R 755 /oradb
d. Set up account equivalence between the cluster nodes for the "oracle" user account.
Add the following entries to the /etc/hosts.equiv file on both the cluster nodes.
cpqrh1 oracle
cpqrh2 oracle
cpqrh1-eth2 oracle
cpqrh2-eth2 oracle
5
technical report
e. Test the "oracle" account equivalence using a remote shell utility such as rsh after
logging in as "oracle" user from both the cluster nodes.
cpqrh1:
#su - oracle
$rsh cpqrh1 pwd
$rsh cpqrh2 pwd
$rsh cpqrh1-eth2 pwd
$rsh cpqrh2-eth2 pwd
cpqrh2:
#su - oracle
$rsh cpqrh1 pwd
$rsh cpqrh2 pwd
$rsh cpqrh1-eth2 pwd
$rsh cpqrh2-eth2 pwd
f. Create the following two shared files on /oradb. The "oracle" user on both the cluster
nodes should have read/write permissions on these files.
#su - oracle
$touch /oradb/SharedConfigFile
$touch /oradb/CmDiskFile
Where:
/oradb/SharedConfigFile will be used by cluster management utilities such as srvctl and oem
to store the cluster configuration, and…
/oradb/CmDiskFile will be used by Oracle Cluster Manager to store the quorum disk information.
2. Installing Oracle9i Release 2 Cluster Manager.
In Oracle9i Release 2, the Oracle Cluster Manager has to be installed, configured, and started
before you install Oracle9i RAC. The Oracle9i Release 2 Cluster Manager is packaged along
with Oracle9i Release 2 software distribution. Refer to Oracle9i Installation Guide Release 2
(9.2.0.1) for UNIX Systems [ Part No:A96167-01] at http://otn.oracle.com/docs/content.html ) for
more information on installing Oracle Cluster manager on Linux. This section highlights the
specifics when integrating with Network Appliance filer.
a. Start the Oracle9i Release 2 installer from either of the nodes. This document assumes
the Oracle installer is started from cpqrh1.
b. From the Available Products window, select Oracle Cluster Manager 9.2.0.1.0.
c. When prompted to enter the Public Node names, enter the names of nodes accessible
from the public network, in the case, cpqrh1 and cpqrh2.
6
technical report
d. For the Private Node names enter the name of the nodes accessible over the private
network between the cluster nodes or the cluster interconnect, in this case, cpqrh1-
eth2 and cpqrh2-eth2.
e. Accept the default value for the Watchdog Parameter Value.
f. When prompted to enter a Quorum Disk file name, enter the full path to the quorum
disk file created above, i.e., /oradb/CmDiskFile
g. The Oracle installer will install all the required files for Oracle Cluster Manager on both
the nodes.
3. Configuring and starting Oracle Cluster Manager.
After installing Oracle Cluster Manager, you have to configure and start it before you can install
Oracle9i RAC. Refer to the Oracle9i Administrator’s Reference Release 2 (9.2.0.1) for UNIX
Systems [ Part No:A97297-01], Appendix F for more information on configuring Oracle Cluster
Manager on Linux. This document will highlight the specifics when integrating with Network
Appliance filers.
a. Create the watchdog device and install the softdog module required by Oracle Cluster
Manager. Execute the following commands on cpqrh1 and cpqrh2.
#mknod /dev/watchdog c 10 130
#/sbin/insmod softdog soft_margin=60
b. Rename the $ORACLE_HOME/oracm/admin/cmcfg.ora.tmp file on cpqrh1 to
$ORACLE_HOME/oracm/admin/cmcfg.ora
cpqrh1
$mv $ORACLE_HOME/oracm/admin/cmcfg.ora.tmp
$ORACLE_HOME/oracm/admin/cmcfg.ora
c. Copy the cmcfg.ora file over to the other cluster node. In this case copy it over to
cpqrh2.
$rcp $ORACLE_HOME/oracm/admin/cmcfg.ora cpqrh2ORACLE_HOME/oracm/admin
d. Edit the cmcfg.ora file on the cluster nodes and make appropriate changes.
cpqrh1:
$ORACLE_HOME/oracm/admin/cmcfg.ora
HeartBeat=15000
ClusterName=Oracle Cluster Manager, version 9i
PollInterval=1000
MissCount=20
PrivateNodeNames=cpqrh1-eth2 cpqrh2-eth2
PublicNodeNames=cpqrh1 cpqrh2
7
technical report
ServicePort=9998
WatchdogSafetyMargin=5000
WatchdogTimerMargin=60000
CmDiskFile=/oradb/CmDiskFile
HostName=cpqrh1-eth2
cpqrh2:
$ORACLE_HOME/oracm/admin/cmcfg.ora
HeartBeat=15000
ClusterName=Oracle Cluster Manager, version 9i
PollInterval=1000
MissCount=20
PrivateNodeNames=cpqrh1-eth2 cpqrh2-eth2
PublicNodeNames=cpqrh1 cpqrh2
ServicePort=9998
WatchdogSafetyMargin=5000
WatchdogTimerMargin=60000
CmDiskFile=/oradb/CmDiskFile
HostName=cpqrh2-eth2
Where:
/oradb/CmDiskFile is the quorum disk file accessible from both cluster nodes.
e. Start the Oracle Cluster Manager by running the following script on cpqrh1 and cpqrh2.
$su - oracle
$su
#. .profile
#ocmstart.sh
f. Verify that the Oracle Cluster Manager is started by checking the
$ORACLE_HOME/oracm/log/cm.log file on cluster nodes.
4. Installing Oracle9i RAC.
a. After ensuring that Oracle Cluster Manager has started on the cluster nodes, restart
Oracle9i Release 2 installer from cpqrh1.
b. In the Cluster Node Selection you should see all the cluster nodes, in this case, cpqrh1
and cpqrh2. If the nodes are not displayed in the cluster node selection, then Oracle
Cluster Manager is not configured or started on those cluster nodes. Configure and
restart the Oracle Cluster Manager and rerun the Oracle installer.
c. Select all the available nodes in the Cluster Node Selection window, i.e., cpqrh1
and cpqrh2.
8
technical report
d. Select Oracle9i Database 9.2.0.1.0 ->; Enterprise Edition ->; Customized in the next
three Oracle installer windows.
e. When prompted to enter the Shared Configuration file name, enter the full path to the
shared configuration file created above, i.e., /oradb/SharedConfigFile.
f. After installing the required Oracle 9i Release 2 executables and libraries on the cluster
nodes, Oracle installer will start executing the tools and utilities in following order:
1. Cluster Configuration Assistant (CLUSTCA)
2. Network Configuration Assistant (NETCA)
3. Database Configuration Assistant (DBCA)
Let Oracle installer complete running and configuring CLUSTCA and NETCA. But when it comes
to running DBCA, cancel out of the installation. The DBCA will have to be run later.
5. Running DBCA to create Oracle9i RAC database.
The Oracle9i Release 2 Database Configuration Assistant (DBCA) can be used to create a clusteraware
database. Refer to Oracle 9i Real Application Clusters Setup and Configuration Guide
Release 2 [Part No: A96600-01] for more information. This document will highlight the specifics
when storing all the Oracle database files in a shared directory on the Network Appliance filer.
a. Create a symbolic link to /oradb on cpqrh1 and cpqrh2. Execute the following commands:
#su - oracle
$mv $ORACLE_HOME/dbs $ORACLE_HOME/dbs.org
$ln -s /oradb $ORACLE_HOME/dbs
b. Before you run the DBCA utility to create the Oracle9i RAC database, check if the global
services daemon (gsd) and the listener have started on the cluster nodes. Execute the
following command to check the status of the gsd on cpqrh1 and cpqrh2.
$gsdctl stat
If the gsd has not started on the cluster nodes, start it before running the DBCA utility
by executing the following command on cpqrh1 and cpqrh2.
$gsdctl start
To check the status of the listener, execute the following command on the cluster nodes.
$lsnrctl status LISTENER
If the listener has not started, start it by executing the following command on cpqrh1 and
cpqrh2.
$lsnrctl start LISTENER
9
technical report
c. Run the DBCA utility to create an Oracle9i RAC database from cpqrh1.
$dbca -datafileDestination $ORACLE_HOME/dbs
The datafileDestination argument specifies the destination directory where the entire Oracle
database files, such as data files, log files, control files, server parameter file, initialization
parameter files, and password files, will be stored. This directory has to be shared by all the
cluster nodes. Refer to the Oracle9i Real Application Clusters Setup and Configuration Guide
Release 2 [Part No: A96600-01] for more details on using DBCA or creating Oracle9i RAC database
manually using scripts.
Known Issues and Workarounds
a. Cluster Manager installation.
Issue:
After the Oracle Cluster Manager installation, the $ORACLE_HOME/oracm/log directory
is missing from the other cluster nodes (cpqrh2). When you to try to start the Oracle
Cluster Manager on that node, it will fail to start.
Workaround:
Create the $ORACLE_HOME/oracm/log directory on the other cluster nodes (cpqrh2)
before starting the Oracle Cluster Manager.
$mkdir $ORACLE_HOME/oracm/log
b. Oracle9i RAC installation.
Issue:
After the Oracle9i RAC installation, the following directories are missing from the other
cluster nodes (cpqrh2).
$ORACLE_HOME/network/log, $ORACLE_HOME/rdbms/log and
$ORACLE_HOME/rdbms/audit
This will prevent you from starting the listener or connecting to any database from that
node (cpqrh2).
Workaround:
Create the following directories on the other cluster nodes (cpqrh2):
$mkdir $ORACLE_HOME/network/log
$mkdir $ORACLE_HOME/rdbms/audit
$mkdir $ORACLE_HOME/rdbms/log
10
technical report
Disclaimer
The reader understands that each environment has its own specific set of requirements and
no guarantees can be given that the results presented in this report will work as expected on
other platforms. This paper should assist in the research and troubleshooting that may be
required in a particular case and serve as a checklist of items to be aware of. Please forward
any errors, omissions, differences, new discoveries, and comments about this paper to
[e-mail].
11
technical report
Network Appliance, Inc.
495 East Java Drive
Sunnyvale, CA 94089
www.netapp.com
© 2002 Network Appliance, Inc. All rights reserved. Specifications subject to change without notice. NetApp, SnapRestore, and the Network
Appliance logo are registered trademarks and Network Appliance, DataONTAP, and The evolution of storage are trademarks of Network
Appliance, Inc., in the U.S. and other countries. Oracle is a registered trademark, and Oracle9i is a trademark of Oracle Corporation. Linux is a
registered trademark of Linus Torvalds. Red Hat Advanced Server is a trademark of Red Hat, Inc. All other brands or products are trademarks
or registered trademarks of their respective holders and should be treated as such. TR-3189 Rev. 09/02
technical report