I-Motion SQL Server admin concerns

I-Motion SQL Server admin concerns I-Motion SQL Server admin concerns Version Date Author Comments 4 2014-04-29 Rebrand 3 2011-07-12 Vincent MORIAUX Add Maintenance Plan tutorial appendix Add Recommended SQL edition 2 2011-07-05 Vincent MORIAUX Add Setup hints Add general information about the I- Motion DB 1 2011-06-29 Vincent MORIAUX First version Contents I-Motion SQL Server admin concerns... 1 Introduction... 3 Document goals and target... 4 SQL Server generalities... 5 Database schema... 5 Memory... 5 Maintenance plan... 5 Hard drive... 5 The hardware... 6 Recommended hardware... 6 32 bits platform concern... 6 Memory sizing... 6 How to know that the server need more memory... 6 Hard drive considerations... 7 The data file... 7 The log file... 7 Fragmentation... 7 Virtual servers... 7 Operating system & SQL Server... 8 Dedicate the server to SQL Server only... 8

I-Motion SQL Server admin concerns 32 bits system tweak for system with more than 3GB of RAM... 8 Force the OS to use only 1GB of memory... 8 Use more than 4GB of memory... 9 SQL Server setup... 10 Recommended Edition... 10 Setup options... 10 Components... 10 Instance ID... 10 Services accounts... 10 Database maintenance... 11 Backup plan... 11 The data file backup... 11 The transaction log file backup... 11 Backuping to a network share... 12 Files size and shrinking... 12 Indexes maintenance... 12 I- Motion DB basics... 13 Major tables... 13 Table DBInfo... 13 Table Engines... 13 Table DEVICE_STORAGE_INFORMATIONS... 13 Table MEDIA_INFORMATIONS... 13 Table chprocesses... 13 Table chenginesprocesses... 14 Table chrequests... 14 Table chrequestvariables... 14 Appendix A Maintenance Plan tutorial... 15

I-Motion SQL Server admin concerns Introduction I- Motion software suite is about maintaining up to date information about media and applying customized workflows to these media. As you may know, since I-Motion v3, all the data about the media and the workflow execution are stored in the I-Motion database hosted on a SQL Server. Due to the increasing number of media the system has to handle, the database size grows fast. Has the database become the heart of the I-Motion system, it now requires careful setup and maintenance to keep its performance optimal.

Document goals and target This document tries to give some guidelines not only about the I-Motion database but also about the SQL server itself that may host other database (Invenio and Automation for instance). This document should be read and understood by all field engineers installing SQL Server database on customer site. All comments are welcome to improve this document. Please contact the I-Motion team if you think of a way to improve this document that will then help your coworkers. Any database (and in general I-Motion) questions can be posted on the I-Motion SharePoint discussion section.

SQL Server generalities SQL Server isn t a classic application. It has to handle lots of data. Data are stored on the hard drive but SQL Server will try to keep as much data it can in memory because it s much faster. It also needs some CPU power to serve these data. Generally speaking we can list, by order of performance impact, several items: Database schema The database schema (tables, columns, fields, etc.) is managed by developers. You can t act a lot on this item. If you see something that is wrong with this schema, please report to the I-Motion team. We will work on the issue. Especially Indexes have a huge impact on the database performance. Without team approval, it s strongly unadvised to modify the database schema because it could impact the whole system. Memory Yes SQL server needs memory, a lot. It has a great impact on server performance. The less SQL Server has enough memory, the more it will use the hard drive which is very slow compared to memory. Maintenance plan Maintenance can improve performance. But done improperly, it can also decrease the performances. Hard drive Unless you don t have enough memory, the hard drive (or other file systems using RAID or SAN) performance doesn t really matters. By the way there is very little ways to improve hard drive performance and yet none of the solution will compete with the memory performance. We will still see in this document how we can improve the performance of this item. The following section will detail how to make the right choices when installing or maintaining a SQL Server for I-Motion.

The hardware For new installation, customer will have to buy servers for Harris services. This section gives some hints about the hardware to purchase. It can also be used if it plans to upgrade an existing server. Recommended hardware CPU Memory HDD 64 bit with at least 2 or 4 cores at least 8 GB of memory RAID 1 hard drive array for the Operating system Either RAID 0 or RAID 1+0 for database files Or SAN to store the database files. (Mandatory for clustered installations) 32 bits platform concern Most new servers are using 64bits CPU. But you may have to reuse an older server with a 32bits CPU. As you may know this limits addressable memory to 4GB. Well that s not true for SQL Server. So you can still add some memory to an old server to boost it. Just know you ll need to change few settings in the OS and the SQL Server to take benefit of this additional memory. Memory sizing For I-Motion database 4 to 8GB of memory should be enough for most customers. The goal is to fit all the database data into the memory. So Memory size should be sized taking into account: Operating system use 2GB of memory Database data size (including all databases on this server) Other services running on this computer also need memory Unfortunately, there are a lot of factors, and you don t exactly know the database data size. But nowadays memory is cheap. So if you have several databases on the same server, and a big system, ask for 16GB or more memory and you should be safe. How to know that the server need more memory When the server is in production, and you think it s too slow you can use the Performance Monitor to check if you need to add memory or not. You need to monitor the SQLServer:Buffer Manager Buffer cache hit ratio counter. It tells you (in percent) how many time the SQL Service got the data from the memory instead of from the files since the service started. So the closer you are to10 the better. If it s below 98% your server probably need more

memeory. Before buying more memory, you can try to stop all unnecessary service, move other application to another server, etc. to free up memory. Hard drive considerations When you can t add memory, the hard drive becomes the bottle neck of your server. It s then a good thing to have fast physical units but you first need to understand few things. Databases are stored in two files: the data file (MDF) and the log file (LDF). The data file Should be at least on a separate partition to prevent fragmentation. This file is only critical (in term of performance) when you don t have enough memory. It access randomly. You can then dedicate a fast drive to this file or better, a flash drive. The log file Should be on a dedicated hard drive if the database activity is high, even if you have enough memory. This file is accessed for every client request but is only accessed sequentially. So a fast drive isn t required there. Fragmentation Even if the database files are on dedicated partitions, they may appear fragmented for the OS. Avoid defragmenting these partitions. SQL server directly manages the files so defragmenting them can be counterproductive. Virtual servers Ok virtual server can be very convenient but it also has a performance impact of at least 10%. It can be worst if the host server doesn t have enough memory or if the hard drive is shared with other busy virtual machines.

Operating system & SQL Server You should choose the operating system based on the hardware and your SQL Server edition. A 32 bits system requires a 32 bits OS A 64 bits system requires a 64 bits OS to get benefits from the 64bits architecture. Match the OS edition (Standard, Enterprise, etc.) to the SQL Server edition. Dedicate the server to SQL Server only SQL Server use direct access to resources (memory, hard drive and CPU). But still have to share it with other application running on the same server. Always prefer a dedicated server for SQL Server and stop all unnecessary services. Most of all avoid using IIS on the same server. Do not limit the memory used by SQL server. It s ok if it uses all the system memory. 32 bits system tweak for system with more than 3GB of RAM 32 bits system and OS limits the addressable memory to 4GB. Plus the OS reserves 2GB of memory so SQL Server is limited to 2GB of memory. We can tweak the OS and the SQL Server to out pass these limitations. But this tweak is only recommended on server dedicated to SQL server. Force the OS to use only 1GB of memory If your server memory is between 4GB and 16GB you can add the /3GB switch in the boot.ini file and restart your OS. [operating systems] multi(0)disk(0)rdisk(0)partition(2)\winnt="????" /3GB Don t use this switch if your server has more than 16GB of memory

Use more than 4GB of memory If your server has more than 4GB of RAM, you should add the /PAE switch to the boot.ini file. Then activate the AWE support in SQL Server properties and restart your computer.

SQL Server setup Recommended Edition We recommend at least having the Standard edition. Unlike older version the memory is now limited to 64 GB which is more than we need for I-Motion DB. You can check this page if you want to know the differences between editions: https://www.microsoft.com/sqlserver/en/us/product-info/compare.aspx Setup options Components For I-Motion you only need to select: Database engine services Management tools basic and complete if you need to manage the SQL server directly on the server. But you can install and run these tools from another computer Integration services is required if you want to replicate your database You may need to add other options depending of your needs. For example Full-text search is required for Invenio. Instance ID You then have to select the Instance ID. It s not recommended to run several instances on the same server. So it s preferable to use the default instance option. If you choose to name your instance, be aware that by default it won t listen on the SQL default port. So you need to enable the SQL Browser service in order to be discoverable by client applications. Services accounts Make sure database engine and SQL server agent services startup type is set to Automatic. Use NT AUTHORITY\LOCAL SYSTEM as account name for both. If you need to backup you DB to a network location, for the database service, use a user that as the proper rights to access the network (or NT AUTHORITY\NETWORK SERVICE but not recommended by Microsoft). LOCAL SYSTEM account has all rights on the local system but can t access the network.

Database maintenance This advises are for I-Motion database but most of them are also true for most databases. Backup plan Backups do not impact DB performances. But still are very important. You can backup two database distinct items: The data file The transactional log file The data file backup It only contains the state of the database. So if you backup everyday your data file, you can lose up to one day of data. The transaction log file backup It contains the description of what happened since the last backup. This permit two things: You can revert your database to a previous state by Rollbacking transactions from the log file. You can rebuild data by replaying transactions that happened between the last data backup and the current log backup. Backuping your log file every 15mn or 30mn is advised because: When you backup the log, the database log file inner space is freed up. This prevent the database log file to grow indefinitely Frequent backup are fast and small because it only backup what happened since last log backup Most of all, if you do a daily data backup and a log backup every 15mn, you can lose only up to 15mn of data instead of one day. System database backup Optionnaly you may want to bakup this two system databases: master database: records all the system-level information for a SQL Server system. This includes instance-wide metadata such as logon accounts, endpoints, linked servers, and system configuration settings. Also, master is the database that records the existence of all other databases and the location of those database files and records the initialization information for SQL Server. (MSDN) msdb database: is used by SQL Server Agent for scheduling alerts and jobs and by other features such as Service Broker and Database Mail (MSDN)

Backuping to a network share To be able to store your backup on a network share you need to: Use the UNC notation ( \\server\share\directory\filename.bak ) for the file location. Network drives are not accessible by the service which runs under another account. You may need to change the SQL agent backup job owner to NT AUTHORITY\SYTEM. Files size and shrinking SQL server data files are similar to hard drives. When you delete a data, it just marks the data space as available. This doesn t reduce the file size. But this space can be reused later. Shrinking the file remove all these free spaces, reducing the file size. The issue is that when a new data need to be added, the file needs to grow again. And this has for consequences performance lost and possible file fragmentation impacting a bit more performances. So typically the database data file has to be created with its estimated final size (let s say initial size 1GB for I-Motion data). The Autogrowth parameter need to be at least half of this value to reduce number of time SQL Server will have to grow the file if required. The size of the log file really depends on the backup frequency and the activity of your DB. But Hard drive space is cheap, so start with at least 1GB. Most of all, avoid using the Auto Shrink option or the Shrink tasks in your maintenance plan. A big file doesn t degrade DB performances and hard drives are big enough to handle files of few GB. So why would you want to shrink files anyway? If you don t do any transactional log backup, your log file will quickly become huge. Either setup a proper backup plan (see previous section) or (if you re on a test system for example) change the database recovery model to Simple in the database properties: You can still use the shrink tasks manually if: your DB grown abnormally large, that you fixed the abnormal grow causes you do not shrink to the minimal size of the file but to the normal expected size of the DB Indexes maintenance Current version of I-Motion identifies data with GUID. One of the disadvantages of this is that indexes get fragmented quickly, decreasing the performances. So a good thing is add a maintenance plan that rebuilds indexes during the night.

I- Motion DB basics Major tables In order to debug some issue on site, you may need to check data directly into the database. Table DBInfo Store the current version number of the database and the installation date. Table Engines Table It stores the unique identifier of each engine. It is used to associate a configuration stored in the database and an engine running somewhere in the network. The engine stores this ID in its local INI file. This means that if you delete the engine INI file, the engine will ask you to pick one of the existing configurations in the DB. If you create a new ID, you start with an empty configuration for this engine. But the old configuration is still stored in the database. DEVICE_STORAGE_INFORMATIONS It stores the list of instanced plug-ins for each MediaSpy engines. Important columns: uuidstorage is the plug-in unique identifier DeviceAlias is the name of the instance MediaSpyOwner refers to the engine unique identifier Table MEDIA_INFORMATIONS It store the list of files found on each storage. Important columns: uuidstorage refers to the storage unique identifier ID is the file name (with or without extension depending on the plug-in and your parameters) Table chprocesses It stores the list of all configured workflows (published or not) for each ContentHandler engines. Important columns: uuidprocess is the unique identifier of the workflow

processname is the name of the workflow ispublished tells you if the workflow has been published Table chenginesprocesses It associates one workflow to theoretically one or several engines. In practice today one process can only be associated to only one engine. Table chrequests It stores all the requests submitted to the system. Important columns: uuidrequest is the unique identifier of the request uuidprocess and uuidengine tells to which process and engine the request is targeted Table chrequestvariables It stores all variables used by a request, including the input parameters. Important columns: uuidrequest refers to the unique identifier of the request isoriginal tells you that this variable value has been passed as an input parameter variablename and variablevalue are self-explanatory For instance if you want to find all request for MediaID starting with AD that failed you can use the following request that joins three tables: SELECT P.processName, R.startDate, R.endDate, R.status FROM chrequests AS R INNER JOIN chprocesses AS P on R.uuidProcess = P.uuidProcess INNER JOIN chrequestvariables AS V on V.uuidRequest = R.uuidRequest WHERE V.isOriginal = 'TRUE' AND V.variableName = 'MediaID' AND V.variableValue LIKE 'AD%' AND (R.status = 'FAILED' OR R.status = 'ABORTED')

Appendix A Maintenance Plan tutorial In SQL Server 2008, maintenance plans are really easy to setup. Start the Maintenance Plan Wizard and skip the introduction dialog if any:

Name your plan and select Separate schedules for each task : I - M o t i o n S Q L S e r v e r a d m i n c o n c e r n s

Pickup the task you want to include in your plan: s Check Database Integrity (Optional): check your database but can be very time consuming. Rebuild Index (Recommended): defragment and reorganize your indexes for better performance Clean Up History (Optional): Remove old historic entries Back Up Database Full (Mandatory): This one is obvious. Back Up Database Differential (Optional): Back up only changes since the last backup so it reduce backup size. You can make a full back up every Sunday for example and a differential one every other day. Back Up Database Transaction Log (Mandatory): Prevent your log file to grow too much and allows to roll back your database to any previous state. Maintenance Cleanup Task (Optional): To delete old backup files

Now you have to setup each individual task. Again it s quite straight forward: Check Database Integrity: Select the DB (IMotion, master and msdb for example), and choose a schedule (every Sunday for example).

Rebuild Index: Seelct the DB (IMotion) and choose a schedule (every day for example). The Free space options should be set to 25% at least. This allocates free space in the indexes pages preventing fragmentation when new values have to be inserted. The small drawback is that your data file will be bigger.

Cleanup History: select a schedule (every Sunday for example)

Backup Database (Full & Differential): Pickup DB to backup, backup directory and schedule (every Sunday for full and other days for differential).

Backup Database (Transaction Log): Pickup DB to backup, backup directory and schedule (every 15 minutes for example).

Maintenance Cleanup: Select a folder to cleanup and an extension. Then pickup a schedule.

I - M o t i o n S Q L S e r v e r a d m i n c o n c e r n s Report options: choose how you want to get the Maintenance Plan logs. Click finish after reviewing the summary:

I - M o t i o n S Q L S e r v e r a d m i n c o n c e r n s The wizard will then create the Maintenance plan and the SQL Server Agent job associated to each task: You can select, review and edit each plan s task later on.