Building Oracle Grid with Oracle VM on Blade Servers and iscsi Storage. Kai Yu Dell Oracle Solutions Engineering

Transcription

1 Building Oracle Grid with Oracle VM on Blade Servers and iscsi Storage Kai Yu Dell Oracle Solutions Engineering

2 About Author Kai Yu 15 years with Oracle technology, specialized in Oracle Grid/RAC, Oracle Apps and Oracle VM Oracle ACE Director, Oracle papers author and frequent presenter at OOW and Collaborate IOUG Oracle RAC SIG president Dell Oracle Solutions Engineering Lab Dell s Oracle Technology solutions based on Dell PE servers and Dell storages. The solutions stack: servers, storage, network, OS, virtualization, Oracle RAC, Oracle Apps

3 Agenda Introduction to Oracle Grid Grid Reference Configuration POC Project Grid Implementation Consolidate Applications on Grid QA

4 Introduction to Oracle Grid Enterprise Grid Computing Group independent hardware and software components in to a single resource pool Can be provisioned, distributed and redistributed on demand Standardization, Virtualization, Automation Benefit of Grid Computing Quickly meet the changing of business needs Dynamically allocate the resources for changing workloads Resource consolidation and sharing reduce the IT cost Providing Predictable service levels Key Technologies for Oracle Grid Sever clustering: Real Applications Cluster (RAC) Server Virtualization: Oracle VM Storage Virtualization: Automatic Storage Management (ASM) Management and Provisioning: Oracle Enterprise Manager

5 Introduction to Oracle Grid Oracle Real Application Clusters Multiple instances running on many hosts interconnected by high speed private network share a single database Oracle Cache Fusion Technology for node synchronization All Instances can execute transactions simultaneously on the single database in the shared storage Provide High Availability for the applications Scalability, special useful for commodity hardware. Automatic Storage Management (ASM) An integrated file system and volume manager Spread data evenly on disks; automatically rebalance Easy management of file system Provide shared storage for RAC nodes

6 Introduction to Oracle Grid

7 Introduction to Oracle Grid Oracle VM Technology Virtual Server: Xen Hypervisor, management domain dom0 Linux kernel with support of devices, IO, networking, etc. Virtual Server Pool: an autonomous region of VM servers VM: guest OS with applications running on domu

8 Introduction to Oracle Grid

9 Introduction to Oracle Grid Enterprise Grid vs Cloud Computing Dynamic shared infrastructure as the basis of Cloud computing Provide the key building blocks for cloud computing Help IT build private cloud and cloud-like infrastructure Engineering solution on Grid Computing Provide customers a solution that builds cloud like services Complete stack of software and hardware platform Provide building, deploying and managing solutions Ensure customers the enterprise grade configuration Engineering Solution through the Oracle Grid Proof of Concept project

10 Grid Architecture Design Grid Reference POC Project Engineering solution for Grid Architecture and implementation Design/implement an enterprise grade Grid infrastructure Dynamically provision multi-tier applications in a single Grid Based on commodity hardware and SAN storage Grid Architecture Physical Grid on clustering: 8 + nodes Oracle 11g RAC Virtual Grid: a set of virtual machines running on Oracle VM servers based on Dell Blades Grid control as the unified management Grid Hardware Platform Commodity Hardware: Dell 11g Blade Servers: Shared SAN storage: Dell EqualLogic iscsi storage

11 Physical View Logical View Grid Architecture Design

12 Grid Architecture Design

13 Grid Architecture Design

14 Implementation Tasks Configure Grid Control Management Infrastructure Configure EqualLogic Shared storage Configure Physical Grid Configure Virtual Grid Grid Control Management Configuration Oracle Enterprise Linux OEL bit Oracle Enterprise Manager Grid control Grid control upgrade Enable Virtual Management Pack Restart OMS server

15 Blade servers connect to EqualLogic SAN Storage

16 EqualLogic shared storage configuration Storage volumes for the Physical Grid Volume Size Raid Used for OS Mapping blade_crs 2GB 10 OCR/Votingdisk Yes blade_data1 100GB 10 Data for DB1 ASM diskgroup1 blade_data2 100GB 10 Data for DB2 ASM diskgroup2 blade_data3 150GB 10 Data for DB3 ASM diskgroup3 blade_data5 150GB 10 Data for DB4 ASM diskgroup5 Volume Size Raid Used for OS Mapping Storage volumes for the Virtual Grid blade_data4 400GB 10 VM Repositories /OVS blade_data6 500GB 10 VM Repositories /OVS/9A87460A7EDE4 3EE92201B8B7989DBA 6 vmcor1-5 1GB each 10 OCR/vogtingdisk 2XOCRs/3Xvotingdisks vmracdb1 50GB 10 Data for RAC DB ASM diskgroup1 vmracdb2 50GB 10 Data for RAC DB ASM diskgroup2

17 EqualLogic storage management console

18 Physical Grid Configuration : 8-node 11g RAC Network configuration Network Interface IO modules Connections IP address eth0 A1 Public Network eth2 B1 iscsi connection (odd) eth3 B2 iscsi connection (odd) eth4,eth5 C1,C2 Bonded to bond VIP Virtual IP for 11g RAC iscsi storage connections Configure host access to storage: eth2 and eth3, /etc/iscsi/iscsid.config iscsi interfaces: i-eth2 on eth2, i-eth3 on eth3: iscsiadm -m iface -I i-eth2 --op=new iscsiadm -m iface -I i-eth2 --op=update -n iface.hwaddress -v <hwadd> iscsiadm -m discovery -t st -p interface=i-eth2 --interface=i-eth3 iscsiadm -m node -p interface i-eth3 --login iscsiadm -m node -p interface i-eth2 login Check Partitions : more /proc/partitions:

19 Find iscsi ID: /sbin/scsi_id -gus /block/sdd Linux Device Mapper to establish mutipath devices to storage alias: /etc/multipath.conf. multipaths { }. } multipath { wwid 36090a c45a2941c7fc76340 alias ocr_crsp14 service multipathd restart Check multipath devices in /dev/mapper: ocr-crsp1, data1, data2, data3, data5 Make partitions: fdisk

20 Use block devices for 11g Clusterware and RAC databases: OCRs, Votingdisks and ASM diskgroups. Configure 11g RAC database Infrastructure 11g RAC clusterware configuration: private interconnect : bond0 OCRs and votingdisks on mutipath devices 11g ASM configuration: ASM instances provide the storage virtualization 11g RAC software to provide the database services Grid control agent to connect to OMS server

21 Consolidate Databases Sizing the DB: Storage, # of DB instances, which nodes to run Provision the storage/asm diskgroup Create the database For some ERP applications, convert the installed DB to RAC

22 Scale out the physical Grid Infrastructure Add more databases to the Grid Increase capability of the existing databases Scale out the storage: add additional EqualLogic array expand existing or create new volumes make new volume accessible to the servers add new ASM diskgroup or add disk to the existing diskgroup or resize the disk of an diskgroup Add new node to the cluster Dynamically move the database to less busy node:

23 Implementation Tasks Overview Virtual servers Installation Virtual server network and storage configuration. Connect VM servers to the Grid Control Create guest VM using VM template Configure the VMs using VM Management Pack with Oracle Enterprise Manager Virtual Server Installation Prepare local disk and enable virtualization on BIOS Install Oracle VM server Change Dom0 memory : /boot/grub/menu.lst: edit line: kernel /xen-64bit.gz dom0_mem=1024m Ensure VM agent working: #service ovs-agent status

24 Oracle VM server network configuration

25 Configure shared storages on dom0 OVM repositories : blade_data4: 400GB, Shared disks for Oracle RAC on VMs Configure iscsi and multipath devices on dom0 /dev/mapper/ovs_data4p1 Put OVM repository on blade_data4p1 (OVM 2.1) mkfs.ocfs2 -b 4k -C 64k -L ovs /dev/mapper/ovs_data4p1 unmount /OVS, moun new OVS: mount -a -t ocfs2 edit /etc/fstab to include the new /OVS partiotion create directories : iso_pool, proxy, running_pool, seed_pool, shareddisk under /OVS

26 Put OVM repository on blade_data4p1 (OVM 2.2) service iptables stop; service o2cb configure, mkfs.ocfs2 -Tdatafiles -N8 /dev/mapper/p1ovsdisk1p1 remove the local /OVS: /opt/ovs-agent-2.3/utils/repos.py -l [ * ] 9e7012d5-69ba-4ac2-a58d-3400cd044efc => /dev/sda3 /opt/ovs-agent-2.3/utils/repos.py -d 9e7012d5-69ba-4ac2-a58d- 3400cd044efc create new repository on shared devices; /opt/ovs-agent-2.3/utils/repos.py -n /dev/mapper/p1ovsdisk1p1 [ NEW ] 567d581b-ec4a-4da8-94df-841dc0c5a390 => /dev/mapper/p1ovsdisk1p1 create the newly created repository the cluster root repository: /opt/ovs-agent-2.3/utils/repos.py -r 567d581b-ec4a-4da8-94df- 841dc0c5a390 [ R ] 567d581b-ec4a-4da8-94df-841dc0c5a390 => /dev/mapper/p1ovsdisk1p1 Reboot all the VM servers

27 VM Management tools: Oracle VM Manager Enterprise Manager: VM Management Pack Connect the VM servers to Enterprise Manager Pre-requisites Steps: oracle user in oinstall group, oracle ssh user equivalence between dome0 and OMS server, Oracle user sudo privileges : /etc/sudoers

28 Create VM server Pool: login to Grid control

29 Add additional virtual servers to the VM server pool

30 Create Guest VMs using VM template Virtual Machine as node of Virtual Grid Create guest VM using VM templates Import a VM template: download template; Discover VM template

31 Create VM using the template

32 Prepare storage for VMs: virtual disks: From the Images files in repository: Local disk: Shared disk Disks as resources for VM:

33 Add local disk to the VM Attach shared disk to the VM s

34 Disks shown in VM as the virtual disk partitions: Attached the storage to the guest VM a) vm.cfg: disk = [''file:/ovs/shareddisk/racdb.img,xvdc,w!', b) vm.cfg: disk = [' phy: /dev/mapper/vmracdbp1, xvdc,w!', Virtual network interface through Xen bridges vm.cfg: vif = ['bridge=xenbr0,mac=00:16:3e:11:8e:ce,type=netfront', 'bridge=xenbr1,mac=00:16:3e:50:63:25,type=netfront',] xenbr0 eth0 in the guest VM xenbr1 eth1 in the guest VM

35 Storage/Network configuration for 11g RAC on VMs:

36 Consolidate Applications Consolidate enterprise applications on the Grid Applications and middleware on the virtual Grid VMs Applications Application VM templates->new VMs Application template VMs Modification New application template Import: existing VM: Copy VM image Import VM image Deploy database service on the physical Grid Provisiong Storage Attach storage to Nodes ASM diskgroup Database create Schema->connect to Database Deploy DEV/Test Application suite on the virtual Grid Multi-tier nodes are on the VMs Fast deployment based on templates

37 Consolidate Applications Grid Architecture to host applications Grid Structure and Applications Physical Grid Virtual Grid VM server Pool 11g RAC Databases Grid Big_5 2vm2 11g RAC nvmr ac4 ebs2a pp2 EBS R12 Test Suite on VMs Banner RAC DB Bann 1 Bann 2 ebs2d b1 Other Database Apps 1 Apps 2 ebsap p1 EBS R12 Apps Tier on VMs EBS RAC DB Other Database Oracle VM EqualLogic SAN storage

38 Consolidate Applications Oracle E-Business Suite Applications:

39 Consolidate Applications Virtual Grid shown on Grid Control Virtual Central

40 SUMMARY Dell Grid POC: Oracle Grid on Dell Blades and EqualLogic Oracle Enterprise Manager Grid control for management Provides the infrastructure to consolidate the enterprise applications and databases Acknowledgement: David Mar, Roger Lopez (Dell Oracle Solutions Engineering), John Tao(Dell IT), Rajat Nigam(Oracle) Related Dell Oracle Whitepapers: 1) Building an Oracle Grid with Oracle VM on Dell Blade servers and Dell EqualLogic iscsi storage 2) Implementing Oracle E-Business Suite in a Scalable and Reliable Virtual System Architecture 3) Efficient Deployment of Oracle Real Application Clusters (RAC) in a Virtualized Environment Using Enterprise Manager Provisioning Pack

41 References 1. Best Practices of Deployment of Oracle ASM with Dell EqualLogic PS iscsi Storage System, Dell White Paper 2. Oracle Press Release: Oracle Enterprise Manager Extends Management to Oracle VM Server Virtualization 3. Oracle Enterprise Manager Concepts 10g Release 5 ( ) Part Number B Oracle Enterprise Manager Grid Control ReadMe for Linux x g Release 5 ( ) April How to Enable Oracle VM Management Pack in EM Grid Control , Metalink Note #: Oracle VM: Converting from a Local to Shared OVS Repository, Metalink note # How to Add Shared Repositories to Oracle VM Pools with Multiple Servers, Metalink Note # Implementing Oracle Grid: A successful Customer Case Study, Kai Yu, Dan Brint, IOUG Select Journal Volume 16, Number 2, Second Quarter Deploying Oracle VM Release 2.1 on Dell PowewrEdge servers and Dell/EMC storage Dell white paper 10. Dell Reference Configuration Deploying Oracle Database on Dell EqualLogic PS5000XV iscsi Storage A Dell Technical White Paper 11. Technical Best Practices for Virtualization & RAC Oracle RAC SIG webseminar slides, Michael Timpanaro-Perota & Daniel Dibbets 12. Kai Yu s Oracle Blog

42 QA

43 Thank You Please Complete the Survey (presentation #705, Building Oracle Grid with Oracle VM on blade servers and iscsi storage, Presenter: Kai Yu) Visit Kai s Oracle Blog: