Oracle Exadata Life Cycle & Services

Transcription

1 Oracle Exadata Life Cycle & Services Best Practises in the Exadata Life Cycle A Whitepaper by avato consulting ag

2 page 2/ avato consulting ag Author: Ronny Egner Lutz Fröhlich Torsten Loof Stefan Panek Frank Schneede (Oracle Germany) Siemensstr , Alzenau/Germany Phone:

3 page 3/69 Contents 1 Exadata Projects on Every Corner - Time for a First Conclusion System Design and Project Planning Analysis and Documentation of the Existing System Landscape Workshop with Oracle Proof of Concept (PoC) Model Selection Preparation Data Centre Detailed Design Detailed Configuration by Means of the Exadata Deployment Assistant High Availability Backup/Restore Installation and Migration Planning of the Installation Process Installation of the Hardware Installation of the Systems Database and Application Migration Migration via Export/Import Migration via Transportable Tablespace Migration with Golden Gate Fallback Scenario Backup/Recovery Exadata Operations Integration into the Support Organisation Horizontal Versus Vertical Support Organisation Oracle DBA or Oracle DMA? Oracle Service & Support Offers Oracle Premier Support & Platinum Support Oracle ASR Further Oracle ACS Services Support by Oracle Partners Exadata Patch Management: Mystery or Predictable Service? What is Patched When and How? Bundle Patch, PSU, CPU, QDPE and QFSDP Time Involved for the Patch Transactions...34

4 page 4/ Patch Cycles in Practise Recommendations for Successful Patch Installation Management- & Support Tools Provsioning/Rapid Provsioning Provisioning Rapid Provisioning Configuration Monitoring Patching Optimisation in the Life Cycle Cell Offloading/Smart Scanning Storage Index Column Projection Predicate Filtering HCC (Hybrid Columnar Compression) Architecture Encryption/Decryption Virtual Columns Parallelisation I/O Ressource Manager Flash Cache Conclusion Appendix Exadata Deployment Assistant Step by Step References...69

5 page 5/69 1 Exadata Projects on Every Corner - Time for a First Conclusion Whether as a basis for large and transaction-rich OLTP, as a Data Warehouse or also as a consolidation platform - more and more often, Exadata is being used as a central building block of the IT infrastructure and many Exadata projects have already been completed. In the foreground of the platform development were scalability, high performance databases and mass data processing, in order to be able to use the system for data warehousing as well as for OLTP applications. Oracle positions Exadata, besides, as an excellent possibility to speed up database consolidation on a platform that is easy to implement. From an Oracle point of view, the systems are Engineered Systems. With the Exadata Database Machine, an entire Engineered IT infrastructure is being supplied. The Oracle Database Machine fascinates not only by excellent performance. Among the rest, it is designed to clearly optimise the deployment and the operation of Oracle databases. It is, so to speak, an all-round talent and a solution for very different challenges. It is time to draw a first Conclusion from a project and operational perspective. What are the requirements to ensure that applications use the platform in the best possible? How is Exadata optimally adapted into existing operating environments? Besides, the following questions are being answered: Which activities result and what do Best Practise approaches look like here? Where does Oracle take on tasks and responsibility in the Life Cycle? At what points, on the other hand, are customers and Oracle partners being called for? In which areas does an optimisation of services require a rethinking in the IT organisation to bring about the greatest possible benefit? Of course, there are standard approaches - but at which points can and should one deviate from these? In this document, the entire Life Cycle of an Oracle Exadata Database Machine is being examined with the help of the essential phases. This Whitepaper, accordingly, presents experience made during the introduction, implementation, operation and optimisation of Oracle Exadata Systems. In addition, the Whitepaper points out different application scenarios and looks at subjects that constitute special challenges on account of the technologies used.

6 page 6/69 2 System Design and Project Planning and afterwards installed via script by Oracle ACS ( Advanced Customer Support ) and are configured. The basic configuration comes from Oracle Engineering. All individual parameters are uploaded via Sheets or the Exadata Deployment Assistant. When filling out the Parameter-Sheets, the customer gets support from Oracle ACS. In the next step, Oracle ACS automatically executes all required scripts and the Post-Check. Oracle advertises that this method ensures an optimal configuration for applications with the ideal mix of high performance, high availability and exceptional scalability. Furthermore, Oracle actively supports with the design by consultation on site. This support can extend up to a Proof of Concept. Because an absolutely new IT infrastructure is being introduced here, numerous tasks still arise from the customer side so that an optimal result can be achieved. 2.1 Analysis and Documentation of the Existing System Landscape At the beginning of every project which deals with the introduction of a new IT infrastructure, there is an extensive requirements analysis. One possible approach is to record the actual architecture and to prepare this for a common Workshop with Oracle. The following key figures are to be considered here: Processor requirements Oracle Databases Server Memory and SGA Planning Network Planning Disk Input/Output Operation Storage capacity planning including growth Consideration of the ASM redundancy Disaster Recovery Planning Fast Recovery Area size and planning Number and size of the databases transaction characteristics User numbers Current bottlenecks (basis can be AWR reports) In most cases, there are monitoring or capacity planning tools in order to simplify the provision of these key figures.

7 page 7/ Workshop with Oracle After conclusion of the actual review and a sufficient Capacity planning where the growth during the next years is considered, a Workshop is coordinated with Oracle. In the Workshop, the key figures are validated, checked and completed if necessary. Oracle can answer many questions from experience with comparable Exadata customer projects. Finally, a decision template is worked out. On the one hand, this serves as a basis for economic calculations (TCO/ROI considerations) and on the other hand as a basis for a possible Proof of Concept. 2.3 Proof of Concept (PoC) A Proof of Concept (PoC) is hard to avoid in the decision for a new IT infrastructure because the risk is substantially minimized through this. Is the expected performance actually being achieved? What is the procedure for a PoC for an Exadata project and what has to be especially considered here? In advance, the objectives of the Proof of Concept are defined. In the course of this, runtimes of Load processes or Batch Jobs and the response time behaviour of an OLTP application can play a part. The execution of the tests is carried out either in a test centre or on a provided machine. Central points for the PoC, beside the performance tests above-mentioned, are above all also compression features as well as a possible migration of the application to Oracle The installations are always based on the latest release. Currently it is possible to also use 12.1 as a DB version, besides With the Storage Server, there is a choice between the versions and Oracle provides the necessary Exadata infrastructure and support, for example in the construction of the database and the application environment. Detailed test series are carried out only if all prerequisites and requirements are fulfilled on the part of the customer. The results provide the basis for the target architecture.

8 page 8/ Model Selection The next step is the target configuration. As already mentioned, a resilient Capacity planning and the consideration of possible reserves are necessary. If, for example, the Exadata is too scarcely calculated in the Storage, this could mean that bottlenecks might result already after a short time. Or, in case the Flash Recovery area was calculated as being too small, the backup concept must be reworked. Experience shows that a reserve of at least 25% should be included in the plan. Target architecture (figures of the current X4 model Series): Computer system Eighth Rack Quarter Rack Half Rack Full Rack Number of Computer Nodes Number of Processor Cores Total Memory in GB Number of Storage Servers Number of Disks Capacity with High Performance Disks in TB Capacity with High Capacity Disks in TB InfiniBand Switches Preparation Data Centre Besides the planning of the Exadata model (or the Exadata models), a thorough planning must be carried out in the Data Centre. An Oracle Support employee provides support, on this occasion, on site. The access to the Data Centre, electricity supply, air conditioning and other parameters are tested.

9 page 9/ Detailed Design For the conclusion of the configuration, the following areas should be looked at in more detail: Network: Domain-Name, IP-Address of the Name server and Network time server; Management Network Addresses; Client Network Addresses and Client Gateway Network Addresses; Additional network addresses for Bonding; All preparations in coordination with the network team within the company (for example to avoid double allocation of addresses); Storage/ASM: There are two kinds of hard disks: * High Performance with 1.2 TB gross capacity; * High Capacity with 4 TB gross capacity; (Note: In an Exadata system, no different disk types can be installed. A mixed operation with 1.2 TB and 4 TB Disks within a system is not possible. Storage Extensions with other Disks can be used.) * Flashcache Storage: Beside the hard disks, more than 44 TB are available for performance optimisation, in addition. Hence, bottlenecks are almost excluded in the I/O area. Recommendation: High Capacity disks with 4 TB capacity; The total capacity increases enormously thereby, even though there are I/O performance losses. Here, a comparison of the gross capacity 1.2 TB versus 4 TB Disk (It is important to note that, after the ASM configuration, only just under 50% of the gross storage space are available.): Computer system Quarter Rack Half Rack Full Rack High Performance Disks in TB High Capacity Disks in TB Oracle ASM: Exadata uses Oracle ASM for Storage Administration. Hence, one should consider whether one works with normal or high Redundancy in the disk groups. This also has effect on the total capacity of the Storage.

10 page 10/69 Disk Groups: * The names of the Disk Groups can be freely chosen - the standard at Oracle ACS is +DATA for user data and +RECO for archives and backup data. * Distribution of the total capacity on the Disk Groups: The distribution must be calculated - the total capacity is mostly implemented in the ratio 40% DATA Disk Group and 60% RECO Disk Group. It has to be noted, for example, whether a backup is written on Disk and which growth rates are to be expected. Database server / operating system: Two possible operating system variants: * Oracle Enterprise Linux x86_64; * Oracle Solaris 11 x86_64; (Note: With a Exadata System X3-8, no Solaris x86 can be installed) The vast majority of installations were carried out on the basis of Oracle Enterprise Linux. However, there are meanwhile also isolated Exadata systems on the basis of Solaris x86. Among the rest, the decision is dependent on which know-how is present within the company in the area of operating systems. Recommendation: Oracle Enterprise Linux; In addition, it should be mentioned that, with Exadata, one gets a Oracle ACS Consulting Package of 20 man days. It is recommended to include this resource actively in the planning. 2.7 Detailed Configuration by Means of the Exadata Deployment Assistant The detailed configuration is carried out in cooperation with the manufacturer. Oracle uses the Exadata Deployment Assistant for it. This tool offers a menu-driven step by step configuration of the overall system. As a rule, a summary with the possibility for the correction is indicated after every configuration step. In the essentials, information is expected in the configuration process in the same manner as is also necessary for standard Oracle databases during the installation and configuration. Nevertheless, at some places, information specific for Exadata is necessary. Though many parameters can still be adapted after the installation, however, this can mostly only be done with considerable effort. Details on the Assistant and a Step by Step example can be found in chapter 8 Appendix - Exadata Deployment Assistant Step by Step

11 page 11/ High Availability Besides a careful planning of the system configuration, the subject Disaster Recovery plays an important part within the scope of an Exadata planning for many customers. However, solution approaches such as the configuration of Stretched Clusters are not feasible with an Exadata architecture. Exadata integrates all hardware as well as software components in a rack as a ViS (Vertical integrated System). Hence, the only Disaster Recovery solution is to establish a second Exadata System in a remote Data Centre and to use Data Guard. With this solution one achieves the maximum in security in the form of a Maximum Availability Architecture MAA. Further information on this subject can be found here: Deploying Oracle Maximum Availability Architecture with Exadata Database Machine [1]. This second system is of double benefit, because it can be used, in addition, for the development and test databases or as a Reporting system - provided that Active Data Guard is used. A cheaper solution is standard x86-hardware on the basis of Oracle Enterprise Linux and as a storage solution either SUN ZFS Storage or a Pillar Axiom Storage System. This configuration can also be used with Data Guard as a Disaster Recovery solution. It is important to note that special features of Exadata, such as Smart Scan, Storage Index, Flash Cache and also IORM are not present. Therefore, this solution is eliminated for performance-critical implementations. 2.9 Backup/Restore The implementation of a backup procedure is not part of the Installation Package of Oracle ACS. Hence, the customer must develop an own concept early on. Basically the backup of Exadata does not differ from the backup of other Oracle databases. In the Exadata environment, too, RMAN (RMAN scripts) is mostly used. Nevertheless, one should pay attention to a few peculiarities: The Compute Nodes must be included in the backup strategy and the procedures. When using an existing Tape Infrastructure, a Media Server for integration of Fibre Channel must be used. Alternatively, one can resort to InfiniBand Fibre Channel Gateways. If it is planned to secure the system via Snapshot and to use HCC, ZFS or Pillar Axiom Storage-System must be used. For more details on this topic, see Chapter 3.6 Backup/Recovery.

12 page 12/69 3 Installation and Migration During the installation of Exadata Systems, many questions arise. Aspects of the vertically integrated technology and the new features are to be considered: Smart Scan Storage Index Flash Cache Hybrid Columnar Compression I/O Resource Manager The installation is divided into different phases that are described in this chapter. 3.1 Planning of the Installation Process The installation process is divided into two areas: 1. Coordination /ODEA: Existing standards in the company are co-ordinated once more and the configuration is prepared with the help of the Oracle Exadata Deployment Assistant (OEDA). The output of the OEDA is an essential factor for the quality of the installation. 2. Installation: The installation is carried out by Oracle ACS. This is an automated process which is controlled via script. 3.2 Installation of the Hardware The delivery of the systems is done through a shipping agency. In the first step, the systems are brought to their correct place in the Data Centre and are erected. However, the commissioning happens only after some days there, since the system must acclimatise. Afterwards the system is commissioned by Oracle and Oracle ACS can begin with the set-up. 3.3 Installation of the System The installation of Exadata is reserved to Oracle ACS and proceeds as follows: Completion of the configuration assistant; Data transfer to the DBM configurator & generation of the parameter files; Upload of the files to the Exadata System; Start of the CheckIP Script to check the network configuration and first Boot; Start of the Onecommand Script;

13 page 13/69 Concerning the installation process, the following should also be noted: Database Software Installation * Oracle ACS installs the at the time current software on the Exadata System ( current with respect to Grid Infrastructure and RDBMS Software). * The software version can be found in MOS Note This MOS Note, beside the Oracle Information Center (Doc ID ), is a good entry point for all information on the subject Exadata. * Besides, the customer should transfer his existing OFA (Oracle Optimal Flexible Architecture) Standards to the Exadata Installation for the purpose of standardisation. Acceptance of the systems * Oracle ACS concludes the installation with an Exachk Performance Test and hands the system over to the customer. It is recommended to repeat the Exachk performance test after configuration changes (Oracle Exadata Database Machine exachk or HealthCheck Doc ID ). * A cluster acceptance test was not carried out by Oracle during the installations accompanied by avato. In the course of the quality assurance, extensive cluster tests are absolutely recommendable. Thus one ensures, for example, that no Single Point of Failure is present and the cluster is fully functional. Documentation * The system documentation handed over by Oracle should be supplemented by company-specific and customer-specific information.

14 page 14/ Database and Application Migration After completion of the installation, the Exadata system is configured and is operational. In the next step, the database is migrated onto the system. The following overview introduces the essential methods for a migration and their boundary conditions: Requirements Methods Export / Datapump Transporta- Data Guard Data Guard Streams GoldenGate Import Export / ble physical Logical Import Standby Standby Oracle Version Table Data Guard Data Guard Streams GoldenGate From 9i independent spaces physical Logical Standby Standby Oracle Edition Standard Standard Standard Enterprise Enterprise Enterprise independent Edition Edition Edition Edition Edition Edition Downtime high high medium low low low low System load (source) high high low low low medium medium medium medium low low medium high high Installation & configuration effort (Database parameters, Script parameters) Complexity low low medium medium high high high Invasive procedures no no no no no If applicable If applicable (engaging in application required) Manual development low low high low medium high high effort Fallback Scenario * effort low low medium low low medium medium Depending on platform / no no no yes ** no no processor architecture Data reorganisation yes yes no no / yes yes Adaptation of Storage yes yes no yes yes Features / parameters Data type Support very good very good very good very good good good good Additional licences no no no If applicable If applicable If applicable yes EE EE EE * Fallback scenario that goes beyond the time of migration (e.g. 1 week) ** MOS Artice ID:

15 page 15/69 The subject of Migration will not be discussed to the full extent in this Whitepaper on account of the high individuality. In the table, for example, the migration from an external system was not considered. In practise, the following central aspects of migration have emerged: Handling of Downtimes; Migration to a x86- architecture; Release (Exadata is always based on the current release); Our experience has shown that, in more than 90% of the migrations carried out, the database Utility Data Pump is used, although a long Downtime is required here. To minimise the Downtime and to carry out a change to a x86 infrastructure at the same time, the export / import process has proven itself. Further recommendations concerning migration: drafting of detailed check lists according to the Step by Step process; Involvement of the specialist department during the migration; Multiple testing of the migration process & dress rehearsal ; Below, some migration processes will be discussed in more detail Migration via Export/Import In practise, a process has established itself with which the Downtime can be minimised, as well as the change to x86 be carried out. Export/Import (and/or since Oracle 10g Data Pump) is the Allrounder for the migration of data volumes up to approx. 1 TB. Bigger data volumes should be migrated by means of Transportable Tablespace if possible. The advantage of the export / import process is that it can be used up to version 8. Moreover, it is independent of the previous operating system of the database. In addition, it offers the possibility to ex and/or import only certain data and to carry out certain changes to the data during the import (for example, a compression of the data). The relatively low speed is disadvantageous. In particular the necessary new construction of the Indices and Constraints can last longer than the following import without corresponding tuning.

16 page 16/69 In practise, the following possibilities have proved themselves to accelerate the export process: Use of the PARALLEL Degree Parameters: A roughly linear increase of the export speed is achieved in practise by a subdivision of the tables to be exported among several Worker Processes. Nevertheless, this speed increase is possible only up to the physical maximum reading speed of the Storage and/or the maximum writing speed of the Storage. Separation of the export file from the Data Files: It is recommended to separate the export to be written from the Data Files to be read files so that both Storages involved do not need to change repeatedly between reading and writing access. In this manner, a sequential reading and/or writing process is achieved. Depositing the export files on the Exadata System: For a speedy import, it is advisable to deposit the dump file already on the Exadata system so that it does not get restricted on the Exadata system owing to the speed limited to 1 Gbit/s. Alternatively, the dump file can be deposited on a Storage that is integrated by a high-speed network (for example, via InfiniBand or 10 Gigabit Ethernet). Compressed export: If the time necessary for the export is too high in spite of parallelisation and separation of the files, the export should be executed compressed. This requires the Advanced Compression licence or a separate arrangement with Oracle in order to be able to use this feature for the migration. If both are not present, the source server can be equipped with a high number of as fast as possible network connections and the Dump Files generated by the export files can be distributed to these servers. According to experience, most performance problems are to be expected with the export from tables with big LOBs. This is above all because LOBs are exported sequentially - even if a corresponding PARALLEL Degree was stated during export. The export of the table with the LOBs takes longer than the export of all other dates and metadata. This provides for delays. Hence, one should export all data and metadata without tables with LOBs and begin an individual export per table with LOBs in parallel in each case. The export of the Non-LOB tables can already be imported into the new database after termination and the Constraints/Indices can be established. The Separation of the data into important and insignificant also aids to save time. In this case, important data are those without which the application can not function. Insignificant data are, for example, archive data whose migration is less time critical.

17 page 17/69 The scaling of the import process should be tested beforehand via several nodes. An increase in output is not always is achievable. Under the already mentioned requirements, the tables can be imported relatively fast - however, the generation of the Indices and Constraints takes substantially longer. The practise recommends generating the Constraints and Indices manually and not with Data Pump. This step allows for a maximum degree of parallelisation. Caution is required if the tables are generated by the DBA before the import and contain many partitions. This combination provides for long import times on account of additional checks, which the Data Pump must carry out during import, already. For every partition and/or Subpartition to be imported, approx. ten seconds are needed for checking. This is only justifiable for a small number of partitions Migration via Transportable Tablespace If data volume or time is not sufficient, another process offers itself: Transportable Tablespace. You can find further information here: Reduce Transportable Tablespace Downtime using Cross Platform Incremental Backups (MOS Note: ). With this process, the Data files with the data are copied. Differently than export / import, this process has the disadvantage that all data are migrated in one or several Tablespace(s) without possibility to exclude data from the migration. The advantage is in the higher speed since, for example, the time for the new construction of the Indices and Constraints is longer needed. Transportable Tablespaces are subject to restrictions, however - for example, to the following ones: The database version of the target database may not be more current than that of the source database. The source database must be at least Version 10g Release 2. Provided that the platforms of source and target differ, the Data files must be converted with RMAN before import. Certain object types, such as for example Indices, which are based on user-defined functions, are not supported and must be deleted before the transport and afterwards be created anew. The data in the Tablespace(s) must be self-contained i.e. all properties of the diagram must be registered by the Tablespaces to be migrated.

18 page 18/69 In spite of the restrictions listed, this process is well suited to the quick migration of big databases. The process of the migration is as follows: 1. At the source: a. Set Respective Tablespace(s) to READ ONLY; b. Export with the Parameter TRANSPORTABLE=y and the specification of the Tablespace; c. Copy the Data files to target; d. After conclusion of the copy process, set the Tablespaces to READ-WRITE ; 2. At the target: a. Provided that necessary: Convert the Data files with the RMAN (and depositing in the ASM); b. Import the Data by means of Data Pump; c. creation of previously deleted, incompatible objects; The longest time is needed for copying the Data files. This time, nevertheless, can be reduced to a few minutes, regardless of their size. To this end, the Data files are secured during operation as RMAN Image Copy. An image Copy is a 1:1 copy of the Data files which can be used directly by the database. The changes can be transferred incrementally into the image Copy by the RMAN and hence very fast. If one stores this copy via NFS on the target system, the duration of the copy process is reduced to a minimum. However, the process described must be supplemented around another incremental backup after the activation of the status READ-ONLY of the Tablespace. The time for converting of the Data files cannot be reduced with the version 11 g of release 2 of the database. However, since the data to be read are already present locally or on broadband integrated Storage (InfiniBand or 10 - however, not Gigabit Ethernet) the conversion on the Exadata platform is very quick thanks to parallelisation. With this process, the authors have migrated a 20 TB database within five hours. A comparable speed is scarcely achievable with export / import.

19 page 19/ Migration with Golden Gate Golden Gate can be used as the third method of migration. Golden Gate is a replication technology by Oracle, which allows migrations, independent of the data volume, almost without Downtime. With this technology, the data are continuously replicated from the source to the target. Provided that the table is not yet present, it is copied and is approximately synchronously replicated after completion. Golden Gate is also capable of not only carrying out a replication between different Oracle versions but also between different database systems (for instance MS SQL Oracle or Oracle MySQL or MS SQL DB2). On account of this feature, a migration can be implemented after initial synchronisation with minimum Downtime - regardless of the size of the database. However, it is a disadvantage that training in Golden Gate takes a lot of time and a license is required for using it. For more information about Golden Gate in the Exadata environment can be found here: Oracle GoldenGate on Oracle Exadata Database Machine [2]. 3.5 Fallback Scenario Even the best preparation is no guarantee for it that at the end, there will be no errors. Hence, one should work out a Fallback Scenario in advance. If, on the production database, they are already working with Oracle Flashback technology, one could set a so-called Restore Point at the beginning of the migration and in the error case recover back to this. Another method could be to activate a Standby database with the initial state. 3.6 Backup/Recovery The backup of Exadata does not differ from the backup of an Oracle RAC database. The difference lies rather in the Location of the backup. There are the following possibilities: Within the Exadata Rack * In the Fast Recovery Area; * In a ASM Disk Group (Non-Fast Recovery Area); Outside the Exadata Rack * In an ASM Disk Group; * On a NFS Mount Point; * Directly from the database to Tape; * from Disk Backup to Tape;

20 page 20/69 Beside the abovementioned boundary conditions and depending on the specific RPO/RTO requirements of the customer, a backup concept can be created and be implemented. For an effective backup, there are two scenarios: Scenario 1: DB with physical Standby; Backups are drawn by the Primary in the form of an Image Copy very quick recovery by Switch of the Data file in the error case; The Standby NOT running on the Exadata Storage but on the ZBA (ZFS Backup Appliance) this is possible also with HCC. Backups are Snapshots of the Standby DB. These can be used * to generate UAT- and DEV-DBs; * for backing up the database to tape; Disadvantage: A ZBA is necessary; Advantage: Very small RTO and the construction of test databases is very quick; the individual test DB hardly needs any space, since Snapshots are being used; Scenario 2: DB with physical Standby; Pure backups of the Standby DB on tape with the RMAN; Disadvantage: Provides a less large RTO (depends very strongly on the DB size and the speed of the network and the tape drives). Besides, the construction of test databases is substantially slower and, accordingly, needs a lot of space (2 DBs = double space 3 DBs = triple space etc.); You can find further information about here: Backup and Recovery Performance and Best Practices for Exadata Cell and the Oracle Exadata Database Machine [3] and Backup and Recovery Performance and Best Practices using Oracle Sun ZFS Storage Appliance and Oracle Exadata Database Machine [4].

21 page 21/69 4 Exadata Operations Exadata is a Vertical Integrated System with integrated Storage and a coupling of the components via Infini- Band. Thus, new challenges arise in many areas for the Operations. This applies particularly for the Hardware Support, however, also for the organisation of the Operations Teams. Besides, questions arise with regard to Support effort and service offers by the manufacturer. Oracle advertises the platform with the argument that the Support efforts are clearly lower. This, according to manufacturer, is not only due to the standardiation of databases but above all due to the fact that the efforts in the areas Storage, OS, Hardware and network are reduced. Hereinafter, this statement by Oracle will be examined in more detail. 4.1 Integration into the Support-Organisation As a rule, the classical Operations and Support organisation knows four to five areas that carry out the Support specializing across applications: Network, Storage, Operating System, Database (Backup) and Application. Empirical values have shown that one can well orientate oneself by the values stated by Oracle: 60% of the activities relate to the database, 20% to the Administration of the Storage cells and approx. 20% to the Linux Administration as well as other components (like InfiniBand and Network). With the exception of the Application Support, Exadata integrates all levels with special technologies in one system and hence requires an extensive coordination and an intensive co-operating of different Support units. At first glance, it looks aggravating for some Support areas that, besides new technology, basically also the use of the Oracle Enterprise Managers CC is necessary. In addition to the support provided by Oracle and/or Oracle partners, two approaches to come up for the Support during the introduction of Exadata: Oracle DMA Team (Database Machine Administrator) Instead of to the classical horizontal Support organisation, one goes over in subareas to a vertical structure. In the process, task areas are transferred to the DBA team and this team is extended into an Oracle DMA team. Beside the standard subjects, the Oracle DMA team deals in future also with the Storage (ASM), operating system as well as InfiniBand network. Tool- introduction and Skill construction In the classical horizontal Support structure, Enterprise Manager Cloud Control is introduced across teams. Besides, the construction of the corresponding know-how with regard to the system and to the tool takes place in all teams.

22 page 22/ Horizontal Versus Vertical Support Organisation As already described, the introduction of Exadata requires a clear operating concept. Which services are obtained by Oracle and/or Oracle partners? How is the own organisation structured? Quite essential points beside the specifics of a Vertical integrated System are the Software Stack used as well as the management and monitoring tool. Specifics of integrated systems: Requirements originate for the operation that, on the basis of the current structure and the service companies used, may not be able to be resolved. These include, for example: * Setup and Installation: Is extensively performed by Oracle and/or Oracle partners (Further information, see Chapter 3 Installation and Migration ); * Hardware Support (Storage, InfiniBand), ASR (Auto Service Request): Is extensively performed by Oracle (in isolated cases also by Oracle Partners); * Requirements for tools and processes on account of the special hardware (for example Disk Replacement, InfiniBand); Software Stack: DMS & OS With Exadata, Oracle technologies are being used that were perhaps not used in this form up to now. Hence, a Support organisation must bring along extensive know-how and sufficient practical experience in the fields of Oracle Clusterware, ASM, RAC and Data Guard. Since OEL is used for almost all installations, it is advantageous if the support is familiar with OEL. Besides, experience with InfiniBand is also helpful for the error analysis (read and interpret protocol files). Management Tool: The Enterprise Manager Cloud Control is the central tool for Management and Monitoring of the platform. In the event that this is not yet being used, it should be present at the latest at the time of Exadata. In which areas do essential changes arise for a Support organisation and which models can be derived out of this? Basically, there is a second model as an alternative to the established horizontal model (with clear separation between Application, Database, System and Storage): A vertical Support structure, in which the areas Database, System and Storage are summarised into a DMA team (Database Machine Administrator).

23 page 23/69 The following table describes the main differences and the pros and cons of both models: Support Organisation Coordination efforts in Appliance Support Management Tools Team Skill Horizontal Support Structure Database Administration (DBA) System Administration (SA) Storage Administration High: Many activities require the participation of several groups. Different teams have interfaces to Oracle Support. Variety of tools or central tool application (SA Team uses Oracle EM) SA Team must build up know-how about: Grid, ASM Oracle Enterprise Linux Basis Database InfiniBand Oracle Storage Vertical Support Structure New support group for Appliance: ODM Team Low: One team can (almost) cover all activities and there is just one interface to Oracle. Complete management of the systems through a console (incl. storage) The primary know-how is database technology. Except for OEL, the required know-how is usually present. Patching Several teams are involved jointly. Instead of several teams (DBA / SA / Storage) there is only one team (introduction of a four-eye principle). Incident diagnosis Cross-Team Diagnosis error diagnosis team-internally (in > 90% of cases) Database Provisioning Cross-Team activity Complete via EM

24 page 24/ Oracle DBA or Oracle DMA? If one decides in favour of the vertical Support structure a new role is introduced: The DBA (Database Administrator) becomes the DMA (Database Machine Administrator). What kind of know-how and experience is mandatory for operation? Oracle Enterprise Linux (in addition, if necessary, Oracle Solaris x86) Database experience with the current Version 11.2 and / or 12c RAC, possibly Data Guard Enterprise Manager Cloud Control 12c Backup Software What kind of Exadata specialist knowledge must be present? Exadata Architecture Special Exadata Software Features (further information in Chapter 6 Optimisation in the Life Cycle ) InfiniBand, Storage Cells, Flashdisks, KVM Switches, etc. In order to administer an Exadata system, the know-how should approximately be made up as follows: 60% DBA, 20% System administration and 20% Exadata. A DBA with RAC, Clusterware (and possibly Data Guard) Know-how is an excellent basis. 4.2 Oracle Service & Support Offers The Oracle offers differ depending on the phase in the Life Cycle and supporting Organisation. Design Installation Migration Operations/Support Oracle Pre-Sales consults during the planning phase and with the Design (Further information, see Chapter 2 System Design and Project Planning ) Oracle ACS installs the systems (Further information, see Chapter 3 Installation and Migration ) Oracle Partner/Oracle Consulting Oracle has two standard support options: * Premier Support * Platinum Services These can be expanded by further Oracle ACS options: * Oracle Exadata Start-up Pack * Oracle Solution Support Center * Oracle Advanced Monitoring and Resolution * Additional Services

25 page 25/ Oracle Premier Support & Platinum Support Oracle offers Premier Support as well as Platinum Support in the standard, for support of operations teams. The main element is 24/7 Hardware and Software Support. What is the difference between Premium and Platinum Support and for whom / which the application scenario is the complimentary Platinum Support worthwhile? What requirements must be met for the Platinum Service? Source: Oracle Oracle advertises the free Platinum Service with the following services and benefits (source: Oracle): Fault Monitoring * 24/7 Fault monitoring; * Event filtering and qualification; * Reporting on event management; * A single global knowledge base, tool set and client portal; Respond and Restore * 24/7 Response Times: - 5 min fault notification - 15 min restoration or escalation to development - 30 min joint debugging * Escalation process and hotline with dedicated escalation managers; Expert support staff available 24/7; Update and Patch * Assess and analyze: Produce quarterly patch plan; * Plan and deploy: Proactively plan and deploy recommended patches every quarter across all systems and software components;

26 page 26/69 When is the Platinum service helpful and for which scenarios is the service less useful? To decide this, it should be precisely considered what the service can do and what it presupposes. Here, the Patch Service is in the foreground. To be able to offer such a service, Oracle needs on the one hand direct access to the systems via the Oracle Advanced Support Gateway. On the other hand, Oracle must be sure to keep systems on the customer side on a uniform Patch-Level. Besides, no individual Maintenance windows can be agreed within the scope of such a service. If one now looks at three potential application scenarios for Exadata, the following differences concerning the Platinum Service arise: Consolidation platform The nature of every consolidation and hence also of every consolidation platform consists of a high standardisation degree - combined with a strictly regulated process for maintenance and Patching. In such an environment, the Oracle Platinum Service can absolutely make sense. OLTP, Data warehouse Applications with high transaction volume as well as Data warehouses are complex surroundings in general, with individual maintenance agreements and application landscapes. Elaborate test procedures and coordinations for maintenance / changes are generally necessary here. These costs are mostly the downside of a highly standardized service. Characteristics and prerequisites: Hardware Oracle Advanced Support Gateway Recomm. 8 Cores, 48 GB RAM, 6 * 300 GB Disk, Firewall Port enabled Certified Configuration Exadata X2, X3 and X4 with current patch level (OS, DB, Exadata SW) (Further information can be found here: Certified Platinum Configurations [5]) Number of DB Homes and databases for Patching/Resolution: Maximally 4 databases, 2 DB Homes for Eighth, Quarter, Half Rack as well as a maximum of 8 databases, 2 DB Homes for Full Rack Further information on the Premium / Platinum Service can be found here: Oracle Platinum Services Support Policies [6] or Oracle Website - Overview Platinum Services [7].

27 page 27/ Oracle ASR Oracle ASR is an essential component of a comprehensive Support concept by Oracle ACS. ASR is the component that summarises the hardware-sided events of a proactive monitoring and passes them on to Oracle ACS. There, the analysis takes place and the Oracle Field Support is steered by Oracle ACS. According to Oracle, the greatest possible availability as well as a minimised risk are thereby ensured. Source: Oracle The Oracle ASR manager should be operated on a dedicated server under Linux or Solaris. Besides servers (CPU, Memory, System Boards, power supply as well as fan) and Storage (Disk Controller, Disks, Flash Cards and Flash Modules), network components (InfiniBand modules, Switches) are also monitored. Further details on installation and configuration can be found here: Oracle Auto Service Request Exadata Database Machine Quick Installation Guide [8].

28 page 28/ Further Oracle ACS Services Oracle Advanced Customer Support Services offers customers a graded set of services that are not limited to the services during installation. In addition, they can be combined with other Oracle services. The manufacturer s offer also extends to the phase after handover to Operations and includes optimisations, monitoring and support. Source: Oracle According to Oracle, the proactive services that are liable to costs are rendered by a dedicated Support team. Besides, Oracle makes sure that Incidents are immediately detected, are taken on by the team and are solved as quickly as possible with the support of Oracle s extensive Knowledge Base. As already mentioned, the Premium / Platinum Support can be upgraded by Service Sets (for example Oracle Exadata Start-up Pack, Oracle Solution Support Center und Oracle Advanced Monitoring and Resolution ). In addition, Oracle Consulting also offers project-related services - such as for example Oracle Migration Factory and Implementation Services Oracle Exadata Start-up Pack Oracle Exadata Start-up Pack is a bundle of services offered by Oracle Advanced Customer Support Services together with Oracle Consulting. Objective is to especially design the deployment process, in addition to the installation, in an effective and fast manner and to ensure a high quality of the installed systems. Essential services: Advisory Service Assessments and consultation on the basis of Best Practise- solutions throughout the whole Deployment-Process; Installation and Configuration Standardised system installation and system configuration of all components;

29 page 29/69 Production Readiness Technical Reviews and instruction by Advanced Support Engineers, in order to optimally prepare systems for production; Quarterly Patch Service Proactive, quarterly patch service for one year (already included in Oracle Platinum Service); Oracle Solution Support Center A Technical Account Manager and a team of Advanced Support Engineers are closely working together with the customer, either onsite or remote and hence offer 24/7 Support. This dedicated team knows the IT environment on site and the business requirements, important projects, central technologies, as well as operating requirements and processes. Essential services: Critical questions are passed on immediately to the team via a 24/7 hotline. The team closely co-operates with Oracle Support and product Development and in this manner can substantially accelerate problem solving and clearly reduce recovery time. The customer receives regular Feedback and Input on Patch-level, configuration and Performance-questions Oracle Advanced Monitoring and Resolution Oracle Advanced Monitoring and Resolution Services offers an extensive integrated monitoring across the entire environment and the IT stack - from application and database up to server, Storage and network components. Oracle offers proactive monitoring here and supplies solution approaches for performance problems to the customer. Essential services: Oracle promises that Incidents are automatically detected on the basis of a Knowledge Base grown over decades. According to Oracle, Advanced Support Engineers use special diagnosis tools in order to be able to find faults faster and to analyse Root Causes within a short time. Regular reviews of the environment are carried out, including the patch level and the configuration Additional Services This concerns individual services for installation and configuration for Oracle Exadata and for Oracle Exadata Storage Expansion Rack.

30 page 30/ Support by Oracle Partners Which performances are rendered by Oracle partners and how can they provide support in the Life Cycle? Design Installation Migration Consulting: PoC during the planning phase Design of High Availability Integration into Backup-Infrastructure The installation is carried out by Oracle ACS As already described, essential standards and naming conventions of the customer should be considered for the installation. Here the support by Oracle ACS by consultants who precisely know the environment and the standards of the customer is an advantage. Support by Oracle Partners: Exadata is always installed with the newest release version. Exadata is a x86-based System. Exadata offers many features that are not available in this form on other platforms. Even though many applications run on account of highly competitive hardware performanters, experience shows that the performance yield may be clearly higher owing to some adaptations and optimisation Further information, see Chapter 6 Optimisation in the Life Cycle. Operations/Support Exadata is an integrated system. For many customers, the question arises whether Exadata should be integrated in the Standard Support or whether to resort to offers by specialised partners. Further information, see Chapter 4 Exadata Operations. Patch Management Further information, see the following chapter. 4.4 Exadata Patch Management: Mystery or Predictable Service? The following section describes the Exadata Patch Management and provides an insight into the main functions and dependencies. An always current overview of the available patches for Exadata including any critical bugs can be found in MOS (My Oracle Support formerly Metalink ) in Note ID , the 14-day reading of which is recommended to the Exadata DBA.

31 page 31/ What is Patched When and How? For better understanding of Patch Management, one should get an overview about which components of Exadata are patched how and in which intervals. This is listed in the following table: Component Patch through Patch cycle Depends on Exadata PDU Separate Patch If required (hitherto 1 Patch in 3 years) Infiniband Switches Separate Patch If required (hitherto 2 Patches in 3 years) Ethernet Switches Separate Patch If required (hitherto no Patch) Exacheck Separate Patch If required (1-2x a year) - Exadata Cell Node Cell Patch Approx. every 3 months, - Exadata Compute Node there is a new version of the Cell Software Cell Patch Approx. every 3 months, Cell Patch Approx. every 3 months, there is a new version of there is a new version of the Cell Software the Cell Software Oracle Database Bundle Patch every 3 months, together with the Cell Software Cell Patch and Grid Infrastructure As can be seen, the power supply and the Ethernet and/or InfiniBand Switches are used relatively rarely and only when required. This is done, if necessary, within the scope of a new Cell Version. The Exacheck appears at regular intervals and should always be launched and tested before the installation of patches on Exadata in order to remove known problems before the patch. The patch of the Exadata Cell and Compute Nodes is done by means of the Cell patches. These appear within the scope of the QDPE (Quarterly Database Patch for Exadata) about every three months, and the following on the Cell and Compute Nodes: The operating system, including the Packages; The firmware of all the components (e.g. BIOS, Firmware of HBAs, Firmware of the hard disks, etc.); Settings in / on the operating system;

32 page 32/69 The installation of the Cell patch on the Cell and Compute Nodes should be done regularly and always at the same time on the Cell and Compute Nodes. At least once a year, a current version of the Cell software should be installed, so that s to new versions are possible without interim steps. Mixed operation of different Cell versions during a Rolling Upgrade is possible, nevertheless, should be kept as short as possible. The background to this is that features perhaps, can only be used after entire execution of the patch, owing to new Cell versions. The next component in the software Stack is the Grid infrastructure which is the basis for all other services. The Grid infrastructure (GI) is d by means of so-called Bundle patches. The Bundle patches work cumulatively (i.e. the installation of the current Bundle patch is sufficient in order to install the bug fixes of all previous patches in parallel). They appear approx. every three months together with a new version of the software for the Cell and Compute Nodes. For the installation of the Bundle patches, a minimum version on the Cell and the Compute Nodes is assumed, so that a regular Update makes sense. The minimum required version, from experience, dates back at least one year, so that Cell versions with an age of one year are sufficient as requirements for the installation of the Bundle patch. Should the patches be installed in rolling mode, however, the versions must not date back as far as is the case with Non-Rolling installations. The database works by directly building up on the Grid infrastructure. It is patched analogously to the Grid infrastructure with the quarterly appearing Bundle patches. For patching of the individual components, different tools are used. These are listed in the following table: Component Software Tool Cell Node Cell Patch patchmgr Compute Node Cell Patch dbnodeupgrade.sh (see MOS Note ) and/ or YUM in older Versions Grid Infrastructure Database Bundle Patch OPatch Grid Infrastructure Database Minor ormajor Release (z. B oder ) Oracle Universal Installer (OUI) Switch and PDU Separate Patches ILOM

33 page 33/ Bundle Patch, PSU, CPU, QDPE and QFSDP Over time, Oracle has developed a variety of terms that describe the patches (among others Bundle Patch, PSU, CPU, QDPE und QFSDP ). The following table maps and systematises the concepts: Patch Type For System For Component Appearance interval Comment CPU Non-Exadata Grid Infrastructure Database Approx. every 3 months - PSU Non-Exadata Grid Infrastructure Database Approx. every 3 months includes the CPU Bundle Patch (BP) Exadata Grid Infrastructure Database Approx. every 3 months includes the PSU (and thereby implicitly the CPU) Quarterly Databa- Exadata Cell Nodes Approx. every 3 months includes the current se Patch for Compute Nodes Version of the Cell Software Exadata (QDPE) Grid Infrastructure Database (for the Cell und Compute Nodes) and the current Bundle Patch (incl. PSU and CPU) Quarterly Full Stack Exadata (PSU and Switches, if Approx. every 3 months includes not only Patches Download Patch necessary) but the complete Software (QFSDP) Cell Nodes (incl. Installation Binaries) Compute Nodes Grid Infrastructure Database As one can see, the concepts PSU and CPU come from the Non-Exadata world and describe patches for problems relevant for safety (CPU) or safety-critical and operation-critical problems (PSU). The PSUs discussed above - supplemented by Bug fixes for Exadata-specific issues - become part of the Bundle patches with which the Grid infrastructure and the database is patched. These, in turn, are a component of the quarterly appearing Quarterly Database Patches for Exadata (QDPE). For the case of a complete new installation, Oracle provides a possibility, with the Quarterly Full Stack Download Patch (QFSDP) to install the system including all s that have appeared up to now. Thus it can be avoided that the basic Installation is followed by a very complex patch cycle. In the above-mentioned Metalink Note , there are notes on critical bugs now discovered and their Workarounds - and/or patches that may need an additional patch installation.

34 page 34/ Time Involved for the Patch Transactions The necessary time involved for the patching of the system is determined by the following two core aspects: 1. Should the installation be done Rolling or Non-Rolling? 2. Is an Upgrade of the Grid infrastructure and/or the database desired (e.g. from Version or )? From both criteria listed above, the question arises concerning Upgrades, with Minor Upgrades only approx. once in 18 to 24 months (as for example from ) and/or with major Upgrades approx. once in three years (11 12). Also in this case, an approach is possible nearly without Downtime. However, this requires very experienced DBAs and an intensive preparatory phase. The case that is much more frequent in practise is the installation of scheduled s that can take place about every three months - if one follows the patch cycle of Oracle. These s are installable almost always Rolling as well as Non-Rolling. The following table shows approximate values for the installation duration to be expected (depending on the selected process), but includes no buffer to find and fix any problems: Non-Rolling (Full Downtime) Component Effort in hours Depending upon Comment Cell Node 4 hours for all Cell Nodes - Cell Nodes are patched Non-Rolling in parallel; Compute Nodes 4 hours for all Compute Nodes Patch of Cell Nodes Parallel Patching possible; thereafter parallelisation by the DBA necessary; Grid infrastructure on the respective Compute Node Database on the respective Compute Node IN-PLACE 4 hours for all Compute Nodes Patch of Cell and Compute Nodes 2 hours for all Compute Nodes Patch of Cell and Compute Nodes as well as the Grid infrastructure Parallel Patching possible; thereafter parallelisation by the DBA necessary; Parallel Patching possible; thereafter parallelisation by the DBA necessary;

35 page 35/69 Database on the respective Compute Node OUT-OF- PLACE No time during the patch required, because the patch can be prepared completely; then time for removing old binaries necessary; - After the patch time for removing old Binaries necessary; Database (Data Dictionary) 30 minutes per database for importing changed objects, plus time for recompilation (depends on the number of objects) Patch of Cell and Compute Nodes as well as the Grid infrastructure and Database Binaries Rolling (Minimal to no Downtime) Component Effort in hours Depending upon Comment Cell Node Depending on the redundancy of the - - Disk group, between 0-24 hours 4 hours per Cell Node - If space is available, more than one cell node may be removed simultaneously if necessary, and thus the number of patch cycles can be reduced. Compute Nodes 4 hours per Compute Node Can take place in parallel with the patch of Cell nodes Parallel Patching possible; thereafter parallelisation by the DBA necessary; Grid infrastructure on the respective Compute Node Database on the respective Compute Node IN-PLACE 2 hours per Compute Node Patch of Cell and Compute Nodes 1 hour for all Compute Nodes Patch of Cell and Compute Nodes as well as the Grid infrastructure Parallel Patching possible; thereafter parallelisation by the DBA necessary: Parallel Patching possible; thereafter parallelisation by the DBA necessary; Database on the respective Compute Node OUT-OF-PLA- CE No time during the patch required, because the patch can be prepared completely; - Nach dem Patch Zeit für das Entfernen alter Binaries notwendig; Database (Data 30 minutes per Database for importing Patch of Cell and - Dictionary) changed objects, plus time for Compute Nodes as well recompilation (depends on the as the Grid infrastructure number of objects) and Database Binaries

36 page 36/ Rolling Patch-Installation With the Rolling patch installation, only individual components are always switched off for the and are activated again after installation. In the normal case, the entire system is available - albeit with decreased efficiency - and can be used by the users / applications. So that an installation can be done Rolling, certain patches must have been installed successfully. The details vary from version to version and are always taken from the respective documentation. The Rolling process can be used for nearly all patch activities of the components: Cell Nodes Compute Nodes Grid Infrastructure Database The only exceptions are necessary code adaptations due to modified internal database packages requiring recompilation of certain database objects. With a Rolling patch the longest time is taken up by the (possibly) necessary Rebalance in the ASM before the patch and by removing a Cell Node from the ASM. A patch process can last from two to four hours With NORMAL Redundancy, the data are present in duplicate on different Cell Nodes. If a Cell Node is now switched off without prior clean removal from the ASM (where the data on the Cell Node hard disks are distributed among all remaining Cell Nodes), certain data are still present only once. If another hard disk from a Cell Node now fails during the patch process, data loss originates, in any case. That is why it is recommended, for Disk Groups with NORMAL Redundancy, to drop the Cell Node from the ASM before beginning and to wait for the Rebalance to be complete (the distribution of the data from the Cell Node to be patched onto the remaining Cell Nodes). According to the scope of the stored data and the size of the individual hard disk, this can take between six and twelve hours (High speed hard disks) and/or between six and 24 hours (High Capacity hard disks) (per Cell Node). The installation of the patch on individual/several Cell Nodes takes approx. four hours.

37 page 37/69 However, the situation is different for Disk Groups with HIGH redundancy. Here, after switching off a Cell Node, two copies of the data are still present so that a failure of a hard disk on one of the remaining Cell Nodes does not lead to the loss of the Disk Group. The patch of the individual Compute Nodes is independent of the model and the number of the databases. It lasts between two and four hours. Requirements are that the components running on the node (such as instance or application) are available on another node than the node to be patched. The patch of the Compute Node can be done in parallel with the already running patch of the Cell Node(s), in order to save time. Immediately after the of the Compute Node, the patch of Grid infrastructure and Database binaries should be done. Only after this should the Compute Node be added to the cluster again and one can then proceed with the next Compute Node. The Database binaries can be optionally patched In-Place or Out-of-Place. In-Place means that the existing Binaries are patched. Out-of-Place means that to begin with, new Binaries are installed in parallel and these are afterwards patched. Afterwards, the Database instance is launched with the new Binaries. The advantage with Out-of-Place lies in the saving of time - in particular if patching takes place with a complete Downtime instead of Rolling. This is because the Database binaries can be prepared in advance and the process takes place faster. However, the total time required is greater because the new Binaries are rolled out in the preliminary work and the old Binaries must be removed again after the patch Non-Rolling Patch Installation This method is suitable in particular if the further availability of the database is ensured via a Standby or replicated database (e.g. by means of Data Guard or Golden Gate) and the time slot for the installation of the patches should be as brief as possible. If no Standby database is available, the Non-Rolling method is still the method with the shortest time for the installation - but also the method with the longest Downtime.

38 page 38/ Patch Cycles in Practise Experience shows that the non-queried installation of all the available patches is as unfavourable as no installation of patches. The following table shows a patch cycle proven in practise, where the Cell and Compute Nodes are patched once a year and the Grid infrastructure and database approx. every six to twelve months. Patch Cycle Planned Maintenance 6-12 months Grid Infrastructure/Database 12 months Cell and Compute Nodes 2-4 years Grid Infrastructure/Database Major Release ( or ) The patch frequency of the Database and Grid infrastructure decisively depends on two components in this case: 1. Stability of the Database and availability of any required One-Offs 2. Known safety gaps If the application running on the database is stable (i.e. no data corruption and losses, unplanned crashes or incorrect results occur), a patch cycle of twelve months may be sufficient. If serious problems limiting the application are solved in current Bundle patches and Backports in the form of One-Offs are not available, a shorter patch cycle should be selected. For the use of the Platinum Service, shorter patch cycles also apply. Here, attention has to be paid to the conformity with the recommended versions according to MOS Note , since, otherwise, the Premium Support may lapse. The same approach is valid for known safety gaps. These should be judged according to gravity and criticality and should be closed according to their importance through the installation of current Bundle patches. An Upgrade of the database version is necessary in each case after two to four years - at the latest, however, if the Support has expired or the currently used version is more than one version behind the present current version. Example: At present, version is current. Versions and receive s within the scope of the Support from Oracle. Version dates back more than one version and receives no more s. Customers with , hence, should upgrade either to or (better) to

39 page 39/ Recommendations for Successful Patch Installation The following section lists in bullet points Form Tips for a successful patch installation - without claim to completeness: General recommendations * using the Exacheck Tool (MOS Note: ) - using the current version - repair warnings and alerts of the tool before the Patching * using OPlan (better understanding of installation steps) * work through documentation and MOS Notes intensively in advance * preferably, patch Standby or test system before Primary or production Exadata Storage Server * do not make any unsupported changes * do not install any additional software Grid Infrastructure * Software Release should be at least equal to (better: higher than) Database, so that GI >= DB Release Database Software * Recommendation: patching Out-of Place * Installation of One-Offs as Online Patch * Management of the load by means of load services

40 page 40/69 5 Management & Support Tools Beside the known Oracle tools, Enterprise Manager Cloud Control is the most important tool for management and monitoring of the platform (regardless of individual systems / variety of systems / OLTP system / DWH / consolidation platform). Besides, there are some new tools that are used exclusively in the environment of Exadata. These include: Auto Service Request (ASR) SRs are automatically generated for HW defects and are transmitted to Oracle. These include: CPU, Disk Controller and Disks, Flash cards and Modules, InfiniBand, Cards, Memory, System Boards, Power supply as well as fan; Command Line Interface (CLI) For local control of the Exadata Storage Server; Distributed Command Line Interface (DCLI) For concurrent execution on several Cell Nodes; Integrated Lights Out Manager (ILOM) For remote monitoring and control of the Hardware; In the following table, there is an overview of tasks and standard tools. Provisioning Configuration Monitoring Patching Backup Database EM EM EM EM EM/RMAN Grid EM EM EM EM/RMAN ASM EM EM EM n/a Compute Node EM EM OC e. g. TSM Storage Cell EM EM CL n/a InfiniBand Switch EM EM CL n/a

41 page 41/ Provsioning/Rapid Provsioning Provisioning Oracle Enterprise manager is to be recommended for the Provisioning of new databases. The method does not differ from other RAC databases. The following models are distinguished and supported by the Enterprise manager: Database Machine, Database Instance and Database Schema. Advantages and disadvantages of these models: Database Machine (Dedicated VM/IaaS) Database Instance (Dedicated DB/DBaaS) Database Schema (Dedicated Schema/SaaS) Implementation/ Onboarding Easy Easy Intricate (Standardi-sation, Versions) Maintenance Complex Easy Complex Isolation High Medium Low Degree of Consolidation Low Medium High ROI Low (Server level) High (Server, OS) Very high (Server, OS, Database) In particular where Exadata should be used as a consolidation platform, the features of Enterprise manager are in demand for Rapid Provisioning (up to Self Provisioning) Rapid Provisioning Standard Provisioning, as is still carried out today in many enterprises, has a number of disadvantages. Time-consuming processes that are established via different departments must be coordinated, owing to which Transition management becomes complex. This makes the overall process, including Life Cycle Management, personnel-consuming and expensive. Rapid Provisioning on the Exadata platform offers numerous advantages: Required resources are available (within the scope of the Quotas); Provisioning in a few hours; Automated process with little manual resources required; High degree of standardisation;

42 page 42/69 The technical implementation of the Provisioning can be done with the RMAN Clone and Snap Clone methods. DBaaS Snap Clone: Provisioning of big databases within a few minutes; Currently supported: NAS Netapp, ZFS (EMC, HDS planned); Features: * Storage Technology; * Use and throw - Databases (UAT, time travel), short-lived Databases; * Fully integrated into 12c CC Life Cycle Management; Advantages * Saves memory because only the space for the deltas is required (copy-on-write Snapshot), few megabytes for a 1 TB DB; * Because the Snapshot only consists of pointers, merely the pointers must be replaced. Cloning 1 TB DB within a few minutes; DBaaS RMAN Clone: Provisioning via RMAN Clone; Currently supported: All platforms and Storage models; Features * Oracle Technology; * Databases with longer life cycle; * Fully integrated into 12c CC Life Cycle Management; Advantages * Storage neutral; * Fits into the existing infrastructure;

43 page 43/69 Self-Service Provisioning with the EM 12c: Out-of-the-box Console, can be customised, API Provisioning: PaaS, DBaaS Clone Methods: RMAN Clone (copy-on-write), Snap Clone, Export Scheme Integrated: Monitoring, Backup, Patching Self Service user gets a service catalogue presented Is integrated into the complete CC Life Cycle Self Service users can * start and stop Databases; * perform Backup and restore operations; * monitor important metrics; * trigger the Retirement Process;

44 page 44/ Configuration Details are in Chapter 3 Installation and Migration, as well as 8 Appendix - Exadata Deployment Assistant Step by Step. 5.3 Monitoring The Exadata system, as an Engineered system, makes special demands on monitoring. The availability can only be guaranteed if all hardware and software components are included in the monitoring. Only in the Oracle Enterprise Manager can all components be defined as a Target and be provided with metrics. Integration is done with the help of a defined Discovery process. Further information is here: Oracle Enterprise Manager 12c: Oracle Exadata Discovery Cookbook [9]. Verification should be done before the Discovery process can begin. For this, Oracle makes available the following script: $ORACLE_HOME/perl/bin/perl exadatadiscoveryprecheck.pl. If the Agent is installed, the Discovery Process can begin via the console of the Enterprise Manager. This is done via the menu items Add Targets and Add Targets Manually.

45 page 45/69 After the Hosts have been included as a Target, the Oracle Enterprise Manager Setup Automation Kit for Exadata is installed. Afterwards a guided discovery process is carried out via the console of the Enterprise manager. At the end of the process, all Targets are available in the Enterprise manager. Now the individual metrics can be defined for the components and the threshold values for the monitoring can be set. At the conclusion, the inclusion of the software components such as Cluster, Cluster database, Listener and ASM is done. This process is identical to the normal Target Discovery for Oracle Real Application Clusters. Via open interfaces, SNMP-Traps can be sent to other monitoring systems.

46 page 46/ Patching Although the Patching of individual software components is possible via the Enterprise Manager, practical experience shows that control via command line provides more transparency and security. This is largely due to the relationships of the individual components and versions with one another, as well as with the provided Utilities. Further information, see Chapter 4.4 Exadata Patch Management: Mystery or Predictable Service?. 6 Optimisation in the Life Cycle Exadata has numerous features that justify extreme performance. These technologies in particular include Intelligent Storage, PCI Flash Cache and also (E)HCC ((Exadata) Hybrid Columnar Compression). Many of these features are automatically used after the migration to Exadata. Others need an adaptation of the application to the platform as such. The following features are directly usable: (Partial) Cell Offloading and hence features such as for example: * Column Projection * Predicate Filtering * Storage Indices Flash Cache Optimisation and/or adaptations of the application to make full use of the possibilities of the platform require the following features: Cell Offloading * Storage Indices * Predicate Filtering Parallelisation Hybrid Columnar Compression (HCC) Flash Cache

47 page 47/69 Some of the features were listed twice. This is because these are active without optimisation by Default, but cannot completely exhaust their potential. An example of necessary optimisation is the Flash Cache: This is active by Default and tries to cache frequently used data in the quick Flash memory and virtually serves as an Expansion of the Database Buffer Cache. By setting special attributes in the tables, one is able to specifically control the caching of properties in the Flash Cache so as to optimise the performance. For Cell Offloading and in particular for the efficient use of Storage Indices, however, adaptations of the application are necessary. Exadata is able to ensure very fast data access solely through Storage Indices and Cell Offloading. Therefore, it should be checked how the implementation plans of the statements change if one removes the Indices of the Non-Exadata systems necessary for quick data access. The situation is similar with parallelisation, which contributes significantly to the increase in performance. The Key Features of Exadata will be presented and briefly explained in the following section. 6.1 Cell Offloading/Smart Scanning A lot of technologies are subsumed under the concept Cell Offloading or also Smart Scanning. Here, Smart Scanning refers more to the optimisation of SQL Statements by Exadata, while Cell Offloading is a rather generic concept. However, all these technologies have the purpose to shift the work away from the database and towards Storage (the cell). Thus the number of data records being considered should already be minimised on the Storage. The objective is to transmit as few as possible data records and columns to the database for further processing and/or to have the necessary process steps done directly by the Cell Node. Under the umbrella term Cell Offloading, the following technologies are summarised: Column Projection Predicate Filtering Storage Indices Bloom Filter Function Offloading Compression/Decompression Encryption/decryption Virtual Columns

48 page 48/ Storage Index The Storage index ensures the avoidance of superfluous I/Os. It is present in the Memory of every cell, is automatically maintained, is transparent for the database and functions with compressed and not compressed tables. Because the Storage Indices are held only in the RAM they must be newly built up with each relaunch of a cell. Storage Indices can substantially reduce the IOs required for the implementation of the query, since only the blocks must Source: Oracle be read that contain the queried values. Afterwards, in connection with Column Projection and Prodicate Filtering, the data volume to be read and transmitted to the Compute Node is further reduced. Through this, the work on the Cell Nodes and on the Compute Node (where the database runs) is minimised considerably. By this strategy, for example, FULL TABLE SCANs that run on Non-Exadata systems for a very long time, can also deliver very fast results without Indices. The following example should show this briefly with the help of a Select to a big table without Indices: Storage Index switched off: SQL> ALTER SYSTEM SET _KCFIS_STORAGEIDX_DISABLED =TRUE; System altered. Elapsed: 00:00:00:20 SQL> SELECT COUNT(SPALTE1) FROM GROSSE_TABELLE WHERE SPALTE1=0; COUNT(SPALTE1) Elapsed: 00:00:10:00

49 page 49/69 Exadata needs approx. 10 seconds to deliver the result - in spite of activated Predicate Offloading and Column Projection (see below). Without these two features, the required time would still be much more since all data and columns would have to be transmitted from the cell to the database. To show the effect of Storage Indices, these are now activated again: SQL> ALTER SYSTEM SET _KCFIS_STORAGEIDX_DISABLED =FALSE; System altered. Elapsed: 00:00:00:15 SQL> SELECT COUNT(SPALTE1) FROM GROSSE_TABELLE WHERE SPALTE1=0; COUNT(SPALTE1) Elapsed: 00:00:00:05 Through the use of Storage Indices, the processing time has decreased for the Select statement from ten seconds to five milliseconds - without creating a classic index Column Projection With Column Projection, only those columns are returned to the Database that are actually also requested by the query or the Join condition. The result: The execution time of the query is reduced by more than 50%. Example: Only five columns of a table with 100 gaps are queried. As a result, the transmitted data volume reduces to approx. five percent. To demonstrate this we create a table with approx. 250 million rows and query these: SQL> alter session set cell_offload_processing=false; Session altered. Elapsed: 00:00:00:01 SQL> COUNT(SPALTE1) FROM GROSSE_TABELLE COUNT(SPALTE1) Elapsed: 00:00:38:03

50 page 50/69 SQL> alter session set cell_offload_processing=true; Session altered. Elapsed: 00:00:00:01 SQL> COUNT(SPALTE1) FROM GROSSE_TABELLE COUNT(SPALTE1) Elapsed: 00:00:22:55 The example shows that, through the use of Column Projection, already considerable speed advantages result, since less data must be transmitted between Compute Nodes and Cell Nodes Predicate Filtering With the technology of the Predicate Filtering, only those rows are returned to the Compute Node, and hence to the Database, that also correspond to the query. Through this, the amounts of data that are processed by the Database and must be transmitted to the Database are clearly reduced. With Non-Exadata systems, all rows are always transferred and afterwards filtered by the Database. With Exadata, this step is largely already done on the cells. SQL> alter system set _kcfis_storageidx_disabled =true; Session altered. SQL> COUNT(SPALTE1) FROM GROSSE_TABELLE WHERE SPALTE1=0; COUNT(SPALTE1) Elapsed: 00:00:04:48 This query shows how efficiently Exadata queries can be processed by Predicate Filtering. To make sure that the Storage Indices do not have a positive effect, these were first switched off. The resulting running time is only the result of Column Projection and Predicate Filtering.

51 page 51/ HCC (Hybrid Columnar Compression) Architecture The Hybrid Columnar Compression (HCC) was originally reserved for the Exadata systems under the name EHCC and describes a column-wise compression of the data. By the choice of column-wise rather than rowwise compression, Exadata achieves compression factors of 1:5 to approx. 1:15. HCC, however, is only suitable for static data, because changes to HCC compressed data decompresses the affected rows again. Moreover, the data for compression must be loaded by HCC Direct Path (i.e. via Create Table as Select, INSERT /*APPEND */, the SQL Loader or similar). Because of the architecture of Exadata, the data is compressed on the Compute Nodes - but can be decompressed on the Cell Nodes. Thus, the computational burden is shifted from the Compute Nodes to the Cell Nodes. For pure Filtering, the data must not be decompressed. That is only necessary on return to the Database in the form of a rowset. It should be noted that HCC is only supported on Exadata, ZFS Appliances and Pillarstore systems from Oracle. If several similar type Records are compressed, there is a better compression factor than with a row-wise approach (a constant change of text and number cannot be well compressed). In addition, the compression in HCC occurs in units larger than a block (8-32 KB). Thereby the compression ratio is also increased. Source: Oracle

52 page 52/69 In HCC, the data to be compressed are divided into so-called Compression Units of 1 MB each and then compressed column-wise. Source: Oracle As already described, there are various compression methods for HCC, which are listed in the following table. Depending on the selected compression level, different compression methods are used. In practise, the best compromise in compression ratio and speed is often found with ARCHIVE LOW and/or QUERY HIGH. However, this is dependent on the data. Frequently, queries on compressed data are even faster than those on uncompressed data. ARCHIVE HIGH is, however, only to be used if data is kept as long as possible, because the compression of the data takes a long time to complete. Type Format Exp. ratio QUERY LOW LZO 4x QUERY HIGH ZLIB (gzip) 6x ARCHIVE LOW ZLIB (gzip) 7x ARCHIVE HIGH BZIP2 15x

53 page 53/69 With Compression Advisor, the expected compression factors can be estimated: Exec dbms_compression.get_compression_ratio: Compression Advisor self-check validation successful. select count(*) on both Uncompressed and EHCC Compressed format = Blocks, Compressed=4329 Blocks, Uncompressed=17980 Rows, Compressed=231 Rows, Uncompressed=55 Compression Ratio=4.1 Compression Type= Compress For Query Low rows 6.3 Encryption/Decryption Encryption and decryption function similar to the HCC, where the encryption is done on the Compute Nodes and the decryption mostly on the Cell Nodes. The CPUs used can deal with the encryption and decryption directly through the hardware. 6.4 Virtual Columns Virtual columns are a new feature of the Oracle 11 g. With virtual columns, the content is calculated from columns of other tables. Thus, it is possible to create, for example, a virtual column Gross whose values arise from the multiplication of the values of the Net column with the percentages of the percentage column. This saves redundant information and storage - however, the values must be calculated once more with every query, which is done by the Cell Nodes. 6.5 Parallelisation Due to the high number of hard disks, Cell Nodes and Compute Nodes, an Exadata system pays off particularly if the tasks are distributed over as many nodes as possible and are executed in parallel. This fact should be noted for the optimisation in the Life Cycle to be able to be to optimally use the platform Because the parallelisation of an existing environment is very complex, Oracle offers help in the form of the feature Auto DOP. With Auto DOP, Oracle independently selects the optimum number of processes to be executed in parallel. The feature is activated by setting PARALLEL_DEGREE_POLICY=AUTO and controls henceforth the number of the processes working in parallel; resource utilization (CPU and IO) up to a defined maximum; the avoidance of overload;

54 page 54/ I/O Ressource Manager The I/O resource manager offers the possibility to distribute the I/O between several databases running on an Exadata platform, based on different criteria. Though this distribution with CPU resources was possible already in the past, nevertheless, for I/O, this represents a novelty and unique selling point for Exadata. The available I/O- bandwidth can be fairly distributed, guaranteed or limited (similar to what you would expect from the CPU resource manager), for instance to be able to hold SLAs agreed during a consolidation. 6.7 Flash Cache While hard drives take up large amounts of data with low I / O throughput, exactly the opposite is true for Flash cards: They store only low data volumes in proportion to the overall capacity, at very high I/O throughput (currently 3.2 TB per Cell!) that, besides, is not limited by controllers. Here, Exadata is useful to accelerate the data access: Automatic caching of data in Flash; Utilisation of PCI cards to bypass the bottleneck Disk-Controller ; The essential share of the data remains on the hard disks; Depending on the model and expansion stage, up to 44,8 TB Flash Cache (in FullRack) sufficient for most databases; The Flash Cache of Exadata can be operated in two modes: Write Through or Write Back. The Default Write Through accelerates the reading operations. Nevertheless, writing operations still go directly on the hard disk and can constitute, perhaps, a bottleneck.

55 page 55/69 Write Through Write Back The hitherto known Write-Through mode is the Default. The Cache is not persistent. Random Reads are accelerated. New: with Exadata SW , the Write-Back Mode was introduced. Now, the Cache is persistent and Random Writes are accelerated. This leds to the avoidance of free buffer waits in the AWR. To what extent the application benefits from the use of Write Back Flash Cache has to be tested individually for every application. Here, there are no recommendations, since every application works individually. Applications with many small I/Os have tended to benefit more than applications with big sequential I/Os. The AWR serves as the first starting point of the analysis. The conversion between these two concepts can be done online and requires a deletion and the new installation of the existing Flash Cache in the Cell Node.

56 page 56/69 When using the Default Write Through, one can define for certain tables that these are held in Flash Cache. For this, the tables have the attribute CELL_FLASH_CACHE. Choose from the following options: NONE: Never cache this object; DEFAULT: The Automated cache mechanism (This is also, at the same time, the Default); KEEP: The object should preferably be kept in the cache, so that a very fast access is possible. Alternatively, the memory of the Flash Cache can be used as a Disk Group in the ASM for the storage of Data Files of all kinds - however, this is not used in practise. 7 Conclusion To close the arc, the introductory questions will now be considered once more. To what extent have answers and solutions been found? What is the first conclusion from a project and operating perspective? The introduction of Exadata is - from a project perspective - absolutely comparable with other introduction projects of big systems. Oracle and Oracle Partner provide adequate assistance in all areas and the essential steps are standardised. An essential advantage, however, arises from the saving of time. The systems are extremely quickly installed and ready for use on account of the structured approach. Besides, almost all applications already profit from the features of the system in the standard configuration. Custom Configurations can still bring huge benefits but are not always mandatory so that Exadata can very rapidly be used productively in most cases. The customer needs no own Exadata expert knowledge during the project phase assessment, provided that the essential concepts and specifications of Oracle are taken into consideration. What are the requirements for it that applications can use the platform in the best way possible? Here, the situation is exactly like that with numerous other projects in which databases are migrated to a new more efficient infrastructure. Basically, almost every application profits from the new Out-of-the-Box platform without adaptation. However, in many cases, it has been shown that the effort for further optimisations very worthwhile. An optimisation under inclusion of the Exadata-specific features as well as resources goes along with huge improvements in many cases, above all concerning performance

57 page 57/69 What about the operating perspective? How is Exadata optimally integrated into existing operating environments, in which areas is a rethinking of the IT organiation required? Again, it can be said that Oracle and Oracle Partner, in Exadata-specific areas, offer an entire palette of services, ranging from Hardware-Support up to the full operation of the platform. Customers who operate several systems and are thinking about own Support here, should consider a vertically structured platform support, in addition. 8 Appendix Exadata Deployment Assistant Step by Step The Oracle Exadata Deployment Assistant Version 2 can be downloaded at MOS as a patch for Linux, Windows and Solaris. The Deployment Assistant is based on Java and part of the OneCommand Utlility. Start: Instructions for the OEDA (Oracle Exadata Deployment Assistant) using the configuration of the X3-8 hardware as an example.

58 page 58/69 Customer Details Required Information: Name Prefix: Every component name will derive from this Name Prefix. NTP Server/DNS: NTP Server should be on the same hierarchy level. NTP/DNS-entries can be changed after installation process. Hardware Selection The available types of hardware are offered in a list - (e. g. the SuperCluster). There are different consequences if you choose HP or HC Storage: HP (High Performance Less Capacity); HC (High Capacity Less Performance); (Example: Exadata X3-8)

59 page 59/69 Define Customer Networks Admin Network; Client Network; InfiniBand Interconnect (for RAC and access to Storage Cells); Optional: Network for Backup; Exadata seperates the networks for Client and Interconnect as does RAC. In addition, the Admin Network and an optional Backup Network is configurated. Overview of Networks Details of the networks: Name; Subnet Mask; Gateway; Physics of the network (copper/optical); Network application servers (e. g. Exalogic) and/or NFS storage may be integrated in the Infiniband.

60 page 60/69 Admin Network Details about Exadata-spezific Admin Netzwork: The Pool Size and the highest IP Address is derived from the first IP Address and the defined Subnet Mask. Here one can define if the Default Gateway of the database servers is in this Network (not in this case) and if the database server admin name defines the physical host name (as entered in this example). In this part of the net there are also the ILOMs of the servers integrated. Client Network The Client-Network is the network where the application servers are located usually. The Pool Size and the highest IP Address is derived from the first IP Address and the defined Subnet Mask. Here one can define if the Default Gateway of the database servers is in this Network (not in this case) and if the database server admin name defines the physical host name (as entered in this example).

61 page 61/69 InfiniBand Network The Pool Size and the highest IP Address is derived from the first IP Address and the defined Subnet Mask. The Infiniband Network does not have a default gateway. Network application servers (e. g. Exalogic) and/or NFS storage may be integrated in the Infiniband. Backup Network Definition of: Backup Network (as far as defined before); Additional network for Data Guard Network Traffic (is a DR system is in use);

62 page 62/69 Identify Operating System You can choose either Linux or Solaris (usually Linux is chosen); Review and Edit Oracle recommends using connected IP addresses for the configuration. The host names with increasing IP addresses derive from these addresses. If there are no connected IP addresses available the hosts can have other IP addresses.

63 page 63/69 Define Clusters Several RACs can be configurated on an Exadata system. On up to four 2-Node-Clusters there can be cobfigurated up to eight database nodes; Cell Nodes are assigned. This can be changed afterwards - but only with high effort. On a X3-2 Full Rack (eight Compute Nodes and 14 Cell Nodes) there could be four RACs, each with two database servers. On a X3-2 Half Rack (four Compute Nodes und seven Cell Nodes) there could be two RACs, each with two database servers. This is defined now.

64 page 64/69 Overview: Cluster Definition Definition of separation of roles (as RAC)between database and Grid Infrastructure; Disk Group details; Distribution 80% to 20% (if no backup as copy); Keep in mind company-specific installation standards of Oracle software; DBFS Disk Group: Information about database file system: The Oracle Database File System (DBFS) [10] Cluster Definition These aspects are also defined: ORACLE_HOME for Grid Infrastructure, database and the software version; database name and default size of the database; Client Network shows the pool size 7 : * two database servers * two VIP Addresses * three SCAN Addresses

65 page 65/69 Review Cluster Definition This is the overview and last chance to make changes. Definition Monitoring Cell Nodes The Cell Nodes can be configurated for and SNMP Alarm. and/or SNMP server details are shown here.

66 page 66/69 Definition OCM As any other database Exadata can be configurated by using the OCM. This enables to download s via MOS and to enter the configuration there, so that Oracle recognizes the configuration in case of need for support. (also via Proxy Server) Configuration ASR ASR serves to send a service request automatically to Oracle in case of defective hardware. The Auto Service Request (ASR) Process requires an ASR Manager. This is a software, that is in the client network and serves as agent between Exadata and Oracle Support. The ASR Manager receives SNMP Traps of the Exadata System and forwards them via HTTPS to the Oracle support (also possible: via Proxy Server). Moreover, the name, and MOS account name of the local client contact is required.

67 page 67/69 Grid Control Agent Cloud Control is available or can be installed afterwards. The ORACLE_BASE of the agent to be installed, the host name and the Upload Port of the OMS are indicated here. Comments These are comments addressing Oracle ACS.

68 page 68/69 XML File XML File is created.