As we start this session, please keep handy your 3 ring binder which should have the checklists at the front of it and the presentations after.

Transcription

1 1

2 As we start this session, please keep handy your 3 ring binder which should have the checklists at the front of it and the presentations after. Please keep good notes in it. 2

3 My name is Jeremiah Curtis and I ve been with Infor for about 13 years. I work in the Engineering Services group and we install and upgrade SXE and other associated products. Engineering Services is part of the Infor Consulting Service team. 3

4 As we go through the system admin class, I encourage you to ask questions. If you want to hold them until the end, that is fine, too. We ll first go over some concepts regarding how the database works. After we get through some of that theoretical discussion, we ll get into the specific details of how to work in and manage your environment. We ll review your servers and database. We ll also try to crash it! And do some crash recovery. Of course, if you are live with SXE, we ll skip the crashing part of it.. 4

5 5

6 It is important to understand what Infor expects from the Systems Administrator. Infor offers classes like this to teach Sys Admins how to administer the SX.enterprise database and associated environment The following is to help you understand what types of things Infor expects the Sys Admin to know how to do. Obviously Infor is here to help, and if you are having trouble with any of the following procedures, simply log a call with support for assistance. 6

7 Disconnecting users - This should be done by way of the shutuser script. This is a perfect example of something that Infor expects the Sys Admin to know how to do, and to be able to perform this function without having to call support every time a user needs to be disconnected from the database. Verify Backup Logs - You should check the backup logs every morning. Monitor/maintain extent structure dbstats.log We will look more at database extents later in this session. Purging Database log files - If you have more than 200 users on your system, you should look at purging your database log files once in a while. This will also be done when you do the dump and load on your database, which is done once a year. Promon - The Progress Monitor or promon is a very useful tool, and you should know how to at least get into and move around in promon. Maintaining Scripts - If you edit the standard Infor scripts, then you need to support and maintain those scripts. Infor only supports our standard scripts. Maintain.pf files - Sometimes you will be asked to modify Progress parameters, this is something you need to know how to do. Maintain library file - You may be called upon to compile and library a piece of code, this is well documented. The document is available from our web site 7

8 Brokers start & stop - Starting and stopping the database broker in the event of a crash is something a Sys Admin should know how to do. Removing.lk files - As a part of crash recovery, you may be required to remove the database lock file. It is important to know when it is safe to do so, and how to do this if necessary. Clearing shared memory - As a part of crash recovery, you may be required to clear the shared memory. It is important to know when it is safe to do so, and how to do this if necessary. Disconnecting users - If you have a power outage, and your database server is still running on battery power, it is important to know how to clean up all of the user processes that are still on the system even though the users are no longer connected, due to the power failure. Truncating BI files - It is important to know how to truncate your before-image file after a crash. Error messages in log files - It is important to know where and how to look at the errors in the various different log files associated with the SX.enterprise application. 8

9 Some work, like upgrades, applying patches, and dump/loads are considered schedulable and billable work. Infor recommends you to call ahead when doing these types of procedures and scheduling a tech to be available to assist you should anything go wrong with the procedure. Infor does offer 7x24 support, but the 7x24 or after hours support is not required to assist with these types of procedures, and will simply help you restore back to where you started. Scheduling a tech for a prearranged type of procedure can save your company a lot of time and money should something go wrong during that procedure. 9

10 Infor has a partnership relationship with Progress. Infor is one of Progress largest customers. You ll often hear Progress and OpenEdge used interchangeably. To be clear, OpenEdge is the name of the product and Progress is the name of the company. Infor recommends that you call Infor Support for assistance with your Progress database, instead of calling Progress Support. 10

11 It is important to note that OpenEdge is not an operating system. OpenEdge is an application software that runs on top of various different operating system platforms. OpenEdge is both a database engine and a programming language. SX.enterprise is written in Progress 4gl language. 4GLs are designed to reduce programming effort, the time it takes to develop software, and the cost of software development As we use both the Progress OpenEdge database engine and the OpenEdge code engine, this makes compatibility issues between the code and the database nonexistent. You ll often hear Progress and OpenEdge used interchangeably. To be clear, OpenEdge is the name of the product and Progress is the name of the company. 11

12 The database consists of 4 primary physical file types It is important to understand that when we say physical elements of the database, that we mean the files located on disk. This is because the Progress database engine utilizes a lot of processes and memory, and in the event of a crash, the files on disk will be the only elements that are left. (more on this in the crash recovery section) The four elements include:.db - the database file that contains the data, this includes all data files for the database that might not have a.db extension.bi - the before-image file, this file is used for transaction logging and is a critical piece for the crash recovery mechanism..lg - the database log file, this file is very important for troubleshooting..lk - the database lock file, this the only physical piece of the database broker, and should only exist when the database broker is up and running. 12

13 This is our database diagram, and we will be adding to this screen as we go along. The.db,.bi, and.lg are represented by a cylindrical icon. This icon will be utilized to represent physical elements of the database. Representing these elements as separate objects does not imply that they are to be located on separate disks/spindles. In fact the.db and.lg will always be located in the same directory. 13

14 The primary goal of a database broker is to allow multi-user access to a database. The database broker consists of 3 primary elements: The database broker process - _mprosrv The Shared memory segment The database lock file -.lk (note the database lock file should only exist when the database broker is running) 14

15 15

16 When a database broker is started against the Progress database, the first thing that happens is a database broker process gets spawned. That database broker process is _mprosrv Note: The process may be seen by running the following command: Unix: # ps -ef grep mpro Windows: By looking in the task manager. This process attempts to allocate a pool of shared memory. 16

17 Once the _mprosrv process creates the shared memory area, it then allocates portions of this shared memory into various structures or memory tables. One of the shared memory structures that we will be looking at is the User Table, which contains the User ID, TTY, and Process ID of the users that are logged into the database. One of the other shared memory structures that we will be discussing is the Lock Table. 17

18 After the _mprosrv process has allocated the shared memory and built within it the necessary shared memory structures, it creates a.lk or database lock file. Note: This is the only physical part of the database broker, and should only exist if the database broker is up and running. The three parts of the database broker are: 1.) Database Broker process 2.) shared memory 3.) database lock file. 18

19 Now that the database broker is running, users may start to log into the database. For this discussion we will be looking at a Direct Connect Client. This discussion does however fully apply to both a Direct Connect Client and a Remote Client. The only difference between a Direct Connect Client and a Remote Client is the actual connection to the shared memory. After this discussion we will take a look at how the Remote Clients connect to the database. A direct connect client is a CHUI user, a report manager or report scheduler, or any process that connects from the DB server directly. 19

20 When the client User 1 attempts to connect to the database, the process that gets created is the _progres process. This can be seen on your system by running the following command: Unix: # ps -ef grep _progres Windows: By using the task manager. That _progres process talks to the database broker process (_mprosrv). 20

21 The database broker acknowledges the user and logs them into the user table in shared memory. 21

22 Here we can see that the database broker has logged User 1 into the user table in memory. You can also see that the user table is storing the User ID, TTY, and PID for each user that logs into the database. 22

23 The last thing that happens during the user login process is, an entry for the user gets logged into the database log file. Example: 07:19:23 Usr 10: Login by User1 on /dev/pts/16. (452) The database log file is a very useful file for troubleshooting. It keeps track of when users log into and out of the database, and any errors that the database broker receives. Note: If your company has more than 200 users you may have to keep an eye on the growth of your database log file. This file will write an entry into the log file each time a user logs into or out of the database, so it may grow quite large. Lets review where this file is and put it in our checklist. 23

24 The following data flow discussion will help you understand what is going on behind the scenes. With this understanding, you will be able to better know what, if any, potential data loss may result, in the event of a crash. What you are about to see is quite a complex picture. There are a lot of things happening behind the scenes. All of these pieces are in place for 2 reasons: 1.) speed 2.) data integrity The bulk of this process happens in memory, so it is very fast. This process is also designed so that should anything go wrong, your data in the database is not compromised. Note: If nothing else is learned from the following data flow discussion, it is important to at least know the following: 1.) Users never get to see, change, delete, or work with the data in the database in any way. 2.) Users only ever get to work with data in memory. 24

25 As we can see here the record ABC is located in the database. 25

26 When a user process wants to work with a particular record, it first scans memory to find out if the record is already in memory. If that record is not found in memory then it sends a request to the database broker to pull the record from the database on disk to the buffer pool in shared memory. 26

27 Once the record is in memory all of the users have access to that data and can begin reading and possibly changing this data or record. 27

28 Now that we have our User1 logged in to the database, and the record has been pulled into memory, lets see how this record gets updated. During this discussion we will also be talking about the lock table in memory. 28

29 The first thing that happens when a user wants to update a record is an entry is logged into the lock table in memory. This is done so that no two users can be updating the same record at the exact same time. This is just one of the ways that Progress provides data integrity for the data in the database. 29

30 The changes made to the record occur in memory, which again is very fast. Then once the change is complete the lock goes away, and other users are now able to change/update the records once again. 30

31 Everything to this point has occurred in memory, so it is very fast. However, since memory is volatile, we need to get this information to disk. This information however does not get updated to the database. This is where the before-image file comes into play. When we talk about the before-image file, there are 2 parts to consider. 1.) the before-image or.bi file itself 2.) the before-image writer or BIW 31

32 Here we can see the.bi file expanded out. The.bi file is made up of clusters which are made up of blocks. 32

33 The process that is responsible for writing the information from memory to the.bi file is the BIW or before-image writer. Since there was a change in memory, we need that information written to the.bi file. So the biw wakes up and writes out the changed memory blocks to the.bi file. 33

34 However, the biw does not simply write the new value. The biw first writes a begin transaction note or bt note. 34

35 Then the biw writes a note describing a single change. (shown here as c -> d). 35

36 Then finally the biw writes an end transaction note or et note. This gives us the full scope of the transaction. The scope of the transaction being a begin note, all of the notes describing the changes to the transaction, and then an end note. This is so that Progress knows all of the changes necessary to reproduce a transaction. This is important when we look at crash recovery a bit later. Note: The.bi file is a sequentially written file. All of the changes that occur to the database are written out sequentially to the.bi file or transaction log. 36

37 Let s Review what we have so far... 37

38 The data started in the database. The user scanned memory for the record and did not find it. The user then sent a request to the database broker or _mprosrv process. The database broker wrote the record from the database on disk into memory. The user could then see the record and begin working with that record. The user changed that record and those changes occurred in memory. The BIW then woke up and wrote the changes to the before-image file. The changes were recorded in the.bi file by the biw in the form of begin-transaction, a note describing the change, and an end transaction. This is to maintain the full scope of the transaction. 38

39 This is where the after-image file comes into play. The after-image file is an optional piece to the Progress database. You do not necessarily have after-imaging running on your system. 39

40 Here we have the after-image or.ai file. It is shown here as a cylinder because it too is a physical part of the database. I have also shown it over on the right-hand side, because the.ai file must be located on a separate spindle (disk) than the database or before-image files. 40

41 The AIW process is responsible for writing the information from memory to the.ai file on disk. The information that gets written to the.ai file is identical to the information that was written to the.bi file. The difference is that once the information has been eventually written to the database from memory, the space in the.bi file is marked for reuse. The.ai file is not marked for reuse and is a consistent log of all of the transactions that have occurred to the database since the last backup. So, if we lose the database or.bi file, we simply restore the database and, using Progress utilities, roll-forward all of the changes that have occurred to the database since that last backup. Therefore getting the database back to the point that it was in just before the crash. 41

42 The Asynchronous Page Writer or APW is the process that eventually writes the information from memory back to the database. 42

43 So as we see here, it is the APW process that gets the job of writing the information from memory back to the database on disk. We can also see that the database finally has the new value for the record that was changed by User1. Up to this point I kept saying that the information eventually gets written to the database. This is absolutely correct. This is because, once the information has been written out to the before-image file, it is preserved on disk. We do not care when the information gets written out of memory back to the database, because if anything happens to memory, the information is already on disk. Thus it is not important to write the information from memory back to the database right away. So when is eventually? We usually tune the database so that it synchronizes memory to disk approximately every 5 minutes. We ll take a look at this in more detail when we visit the performance tuning section of the class. 43

44 So, let s take a look at the whole picture in a final review... 44

45 The data started in the database. The user scanned memory for the record and did not find it. The user then sent a request to the database broker or _mprosrv process. The database broker wrote the record from the database on disk into memory. The user could then see the record and begin working with that record. The user changed that record and those changes occurred in memory. The BIW then woke up and wrote the changes to the before-image file. The changes were recorded in the.bi file by the biw in the form of begin-transaction, a note describing the change, and an end transaction. This is to maintain the full scope of the transaction. Then the AIW process writes the same information that was written to the.bi file to the after-image file. And finally the changes that occurred in memory get written out to the database by the APW process. 45

46 46

47 Everything we have looked at to this point has dealt with a direct connect client. As I noted earlier, the only difference between a Direct Connect Client and a Remote Client is how they connect to the database. Now we will look at how a Remote Client connects to the database. We will also look at the Progress appserver. 47

48 The client in this slide is a direct connect client. The prowin32 process is the PC equivalent to the _progres client process. When the User1 double-clicks on the SX.enterprise icon on their desktop, it launches the prowin32 process. This process then connects through the network to the database broker. The database broker then logs the user into the virtual system tables in memory. 48

49 Once the broker has logged the user into the virtual tables in memory it then spawns an auto-server for that user. The database broker then tells the user to disconnect from the database broker and reconnect, through the network, to the auto-server that was spawned for them. The auto-server layer is important to the graphical client, because since the prowin32 is a process on a PC, it cannot talk to Unix memory. Therefore the prowin32 process sends its requests to the auto-server. The auto-server then talks to the Unix shared memory and relays the information back, across the network, to the connected user. 49

50 The users are also connected to shared memory through a Progress appserver. This gives each graphical client 2 connections to the database. Note: The auto-servers are a one to many clients relationship and the appserver has a one to one client relationship. 50

51 This usually brings up the question of what is the Progress appserver and why do we have them? Let s take a look at how the Progress appserver works... 51

52 Typically the appserver is running on the system where the database is located. For the purpose of this discussion we have moved the appserver to its own box separate from the database server. The client in this example is an SX.enterprise client connected to the appserver and connected to an auto-server on the database system. 52

53 53

54 Example 1: User1 requests an address for a specific customer. Since this is a small and static amount of data, this request goes to the auto-server on the Database Server. This is a small amount of data going across your network. 54

55 55

56 Example 2: User1 requests a specific price, for a specific product, for a specific customer. Since this request requires business logic to determine what the price for this product will be, this type of request goes to the appserver. 56

57 The appserver then takes over the request and pulls across all of the necessary data needed to determine the price for the appropriate product. Then once the appserver has determined the price, 57

58 it sends just the price back to User1 across the network. This again is a small amount of data going across your network. As you can see, if the appserver is running on a system other than the database system, you need to have a large network pipe between these two systems, and this should be on a separate, dedicated segment of the network than the rest of your network. 58

59 As we saw, each graphical user has 2 connections to the database. One through the auto-server, and one through an appserver. Smaller requests get routed to the auto-server. Larger requests that require business-logic go to the appserver. Infor has written the client code to determine which requests use the appserver and which requests do not. Infor requires all users be running on appserver. The appserver system must be located on the same Local Area Network or LAN as the database server. You will see a significant performance degradation if users are running SX.enterprise graphical across a Wide Area Network or WAN and not using the Progress appserver. If you choose to move the appserver to a system other than the database server, you must have the largest network connection possible between the two systems, and they should be on a dedicated segment of the network. We typically do this on customers with 1500 users or more. 59

60 The Architecture at first glance can be a bit complex, let s try to simplify it a bit Before users can start connecting to the database, a lot of things have to happen. There are several pieces to the architecture and this section will explain how they all work together. 60

61 We start off with the database. 61

62 Admin Server: Before any Progress brokers or servers may be started we have to start the Admin Server. The admin server is the key to the OpenEdge architecture. It is similar to the services on an Windows NT system. It is a service that you start up and it handles the starting and stopping of the other Progress Brokers and Servers. 62

63 Name Server: The Progress nameserver is similar to a network name server or DNS that you may be familiar with. It is responsible for directing Progress clients to the available application brokers. The nameserver is started automatically by the Admin Server. 63

64 Database Broker: The database broker then can be started. A request gets sent to the Admin Server to start the database broker against the specified database. Once that is done, we can see the process running, the lock file gets created, and the shared memory gets allocated 64

65 Application Broker: The application broker then can be started. A request gets sent to the Admin Server to start the specified application broker. 65

66 Once the appbroker is running it registers itself with the nameserver that the admin server started. 66

67 Now that the admin server, name server, database broker and application broker are all running, we can begin connecting users to the database. 67

68 Here User1 double clicks the SX.enterprise icon on the desktop. A prowin32 session begins and communicates through the network to the database broker and requests to log in. 68

69 Auto Server: The database broker acknowledges the connection for User1 and spawns a child process called an Auto Server. The auto server makes a connection to the database. Note that this connection will remain throughout the entire life of the database broker. 69

70 The database broker then sends a message back to User1 instructing the prowin32 process to disconnect from the database broker and 70

71 reconnect to the auto server. Note that this connection will remain active throughout the entire life of the prowin32 client session. On the client, the SX.enterprise login box is now displayed. 71

72 User1 logs in and hits enter the prowin32 process then sends a message across the network to the Progress nameserver requesting the location of the registered application broker. The nameserver then responds with the host name and port number for the application broker. 72

73 The prowin32 process then disconnects from the nameserver and 73

74 connects to the application broker at the host/port provided by the nameserver. 74

75 The application broker acknowledges the login of User1 and spawns an application server. The application server connects to the database and this connection is maintained throughout the life of the application server. 75

76 The application broker then tells the prowin32 process to disconnect and 76

77 reconnect to the application server designated for User1. This connection will be maintained throughout the entire client connection. 77

78 User1 is now has two connections to the database, once via an auto server and once via the application server. This architecture model allows us to scale your environment horizontally. That is, by adding an additional server or 2, we can increase the performance of the appbroker/appservers. The Name servers allows load balancing between the appbrokers. 78

79 The admin server is the key to everything. Nothing else may be started without the admin server running. The nameserver directs clients to the application brokers. Application brokers spawn application servers. Database Brokers spawn auto servers. Summary: Brokers handle logins. Servers serve data. The Auto Server connections are a one to many relationship, so one auto server will typically be servicing multiple clients. The Application Server connections are a one to one relationship, so one application server is only ever servicing one client at a time. 79

80 80

81 Next we discuss how to properly disconnect users from the Progress database. It is important to use the Infor shutuser script and never use the Unix kill command or the Windows end task to disconnect a user. There are also risks involved in shutting out a user from the database. 81

82 When attempting to disconnect a Direct Connect Client connection to the database, the process that we want to cleanup is the _progres process. Example: A user has run a huge report without any options, therefore the report is going to be huge, and probably not what the user wanted. If we simply used the kill command or end task to kill the _progres process, then we definitely clean up the client process. However, since the database broker process is responsible for transferring data from the database into memory, we would not stop the reporting process. The database broker would continue to transfer all of the data necessary to create the report and then attempt to deliver that report to the user, and then realize the user is no longer connected. Meanwhile using lots of disk space, memory, and system resources while building the report. If we use the shutuser script to disconnect a user from the database, this uses Progress commands to disconnect the user. The shutuser script will send a message to the database broker, telling the process that this user needs to be disconnected. The database broker will log the user out of the user table in memory and stop processing anything for that user. Also, if a user has a lock in memory at the time you execute a Unix kill or Windows end task on the _progres process, Progress will panic and bring the database broker down. This is to provide data integrity to the database. 82

83 For a Remote Client connection, the process we want to cleanup is the prowin32 process. This connection can be jeopardized in many ways. Through network connectivity, loss of power, the PC becoming locked up, etc Since this is the case, the auto-server acts as a buffer to shared memory. If the user gets disconnected the auto-server will recognize this and send a signal to the database broker to log this user out of the system tables in memory and stop processing anything for that user. There will still be times that you will need to use the shutuser script on graphical processes. The following is the correct syntax to use with the shutuser script: # shutuser 0001usr Where 0001 is the company number that the user is logged into for SX.enterprise. The company number must be padded to 4 characters with zeros. And, usr is the SX.enterprise login for the specific user. So user jab logged into company 500 would be as follows: # shutuser 0500jab 83

84 84

85 Lets try to crash our database, as long as users aren t in. There are several ways to purposefully crash an Opendge database. We could remove the lock file, which is the only physical part of the database. We could change the hostname of the server, or we could delete a data file. Before we crash it, lets back it up first. That way, if we mess it up too badly, we can just restore from backup. This backup command is one I like to run when I m feeling extra paranoid. We ll talk about backups later on in more detail, but for now, lets run backup.online /db/nxt and see that it creates backup files in the backup directory 85

86 Next we will take a look at Crash Recovery. Recovering from a crash is a 5 step process. The last 3 steps are easy and straightforward. When we talk about crash recovery, it is nice to use a power-outage as an example. This is because it is a very clean way of crashing the database, and we know that we will lose all processes and anything in memory. This does not mean that the following examples will only apply to a power outage situation. In any crash, you will lose anything that was in memory at the time, and any associated processes that were connected to the database. So, keep in mind that the following applies to all crashes. 86

87 Figuring out why we crashed is essential. If we crashed due to a power outage, nothing will be written to the log file. For some reason, hard drives need electricity in order to write data. In most other scenarios, something should be written to the database log file. If we don t figure out why the database is down, then when we start it back up, it could come down again. 87

88 Now that we ve figured out why it crashed, we need to fix it. Did our disks fill up? Then add more disk, or delete some files. Did the power go out? Write a check to the power company and get them to turn it back on. These first 2 steps Figure out why and Fix it could take a while to finish. Happily, the next 3 steps are simple and straightforward. 88

89 We ll discuss 3 scenarios to discover why the last 3 steps are necessary 89

90 In our first power outage scenario we see that the information started in the database, then the user could access and change that data, but the biw did not get a chance to wake up and write that information from memory to the.bi file. 90

91 91

92 When we lose power everything in memory was lost, along with any processes that were running. 92

93 93

94 In this scenario, when we restored power, the only place that the change was ever recorded was in memory, so we lost the change. This is why we tell our users to check their last 3 minutes of work after a crash. Now, 3 minutes is a really long time in computer terms, but it is a nice safety net to give the users so that they can comfortably know what data they may lose. 94

95 We ll discuss 3 scenarios to discover why the last 3 steps are necessary 95

96 In our second power outage scenario we see that the information started in the database, then the user could access and change that data, the biw was awakened and started writing the changes from memory to the.bi file, but it did not get a chance to write the end-transaction note. 96

97 97

98 When we lose power everything in memory was lost, along with any processes that were running. 98

99 99

100 In this scenario, when we restored power, the change was recorded in the beforeimage file by the BIW, and the only thing that did not get written was the endtransaction note. When we start the database broker, it reads through the.bi file and processes all of the transactions logged there. WE DO NOT WANT THIS TO HAPPEN AUTOMATICALLY! Since Progress does not have the full scope of the transaction, signified by the begin and end transaction notes, it must eliminate the transaction. Infor recommends the use of the truncate.bi script to process the transactions in the.bi file before bringing the database broker up. This is so that you know everything is ready to go before you start the database, and you don t have to find out that the database will not start for some reason. 100

101 We ll discuss 3 scenarios to discover why the last 3 steps are necessary 101

102 In our last power outage scenario we see that the information started in the database, then the user could access and change that data, the biw was awakened and wrote the changes from memory to the.bi file, but the APW did not get a chance to write the change back to the database. 102

103 103

104 When we lose power everything in memory was lost, along with any processes that were running. 104

105 105

106 In this scenario, when we restored power, the change was recorded in the beforeimage file by the BIW, and the only thing that did not get done was the APW did not get a chance to write the change back to the database. When we truncate the bi file, it rolls all of the completed transactions into the database, and discards the incomplete transactions. Then, we can start the database without additional delay and the users can begin checking their last 3 minutes worth of work. 106

107 Infor recommends the following five things after a crash: 1.) check the database log file. Remember that anytime the database broker comes down, it is for a good reason, and Progress will record why it came down in the database log file. 2.) fix it Fixing it may include restoring from backup, building a new server, or simply getting the power back on. 3.) use the truncate.bi script to truncate the.bi file. This will ensure that the database is in a known state before you bring the database broker back up. When you bring the database broker back up after truncating the.bi file you know it will be ready to go. 4.) Bring the database back up. 5.) have your users check their last 3 minutes of work. 107

108 108

109 When we talk about parameters, there are actually several different types of parameters. We will now take a look at three different types of parameters. 1.) database parameters - The parameters that the database broker uses to start up. 2.) client parameters - The parameters a client process uses at start up to configure the client environment. 3.).bi file parameters - The parameters we use when truncating the before-image file. We ll just discuss the ones that can affect database performance and access. 109

110 For the database broker, we should use the OpenEdge Explorer tool to modify these files. We could alternately, if we wanted to be bold and daring, modify the conmgr.properties or ubroker.properties file with a text editor. Lets get into your OpenEdge Explorer and take a look at these parameters. We ll also take advantage of this time to make some changes that might speed things up a bit. 110

111 BlocksInDatabaseBuffers or -B - This is the Database Buffer parameter. It tells the database how many shared memory blocks to allocate to the database buffer pool when we start the database broker. The size of each block is dependant on the database blocksize. Infor s installation value for -B is The recommended size is 10% of database size (3750 blocks x 8k database blocksize) = 30 MB memory 111

112 112

113 LockTableEntries or -L - This is the Lock Table parameter. It tells the database how many entries to allocate, when it creates the lock table in memory. The size of each entry is 18 bytes. Infor s standard value for -L is (50000 x 18 bytes) = 900 KB 113

114 114

115 MaxUsers or -n - This is the User Table parameter. It tells the database how many entries to allocate, when it creates the user table in memory. This parameter does not use a lot of memory, however all of the other memory calculations are driven off this parameter, so do not over allocate it. The correct values are as follows: Character only - SASA number of licensed users + 20% = -n value Graphical only - (SASA number of licensed users x 2) + 20% = -n value 115

116 116

117 BeforeImageBuffers or -bibufs - This is the BI Buffer parameter. It tells the database how many memory blocks to allocate to the before-image buffer when we start the database broker. The size of each block is dependant on the before-image blocksize. Infor s standard value for -bibufs is

118 118

119 Directio or -directio - This is the Direct I/O parameter. This tells the database broker, to bypass the OS buffers and write the Progress buffers directly to disk. Instead of writing Progress buffers to the OS buffers and then to disk. Note: there is no value for -directio it is either on or off. Infor recommends running with direct i/o turned on. 119

120 120

121 121

122 SpinLockReTries or -spin - This is the Spin Lock parameter. This tells the client process to continue checking to see if a lock is free x number of times before napping. This is the only current parameter that is tunable with the database running, but it is only tunable if it is turned on. Infor s recommended value for the -spin parameter is 1. The -spin parameter should be increased by for each additional processor. 122

123 123

124 124

125 125

126 DatabaseName or -db - This is the Database parameter. This tells the database broker to start against which database. Host or -H - This is the Hostname parameter. This should be set to the hostname of the system you are starting the database broker on. This can be found by typing hostname at the Unix prompt. Port or -S - This is the Service Port parameter. This is the service port that the database broker will be listening on. MinDynamicPort & MaxDynamicPort or -minport & -maxport - These are the Firewall parameters. These tell the database broker to spawn the auto-servers on the specified range of ports. Example: -minport maxport 7200 would cause the auto-servers to only spawn on ports 7150 through

127 -l - This is the Local Buffer Size parameter. This is a soft limit and tells the client how much memory to allocate for workspace. This value will be increased as necessary. -q - is the Quick Load parameter. This tells the client to reread the code from memory, if that code is already in memory. -T - This is the Work Directory parameter. This tells the client where to create the workspace files. If this is not specified, the default is current working directory, which for the users that would be /home. Infor recommends this parameter to be set to /db/sort -t - Is the Save Temp Files parameter. This parameter keeps Progress from unlinking the filename of the temp files after they are created. In other words if you do not use this parameter the workspace files are hidden. -db - This is the Database parameter. This tells the client which database to connect to. -U & -P - These are the username and password parameters for database security. If you disable the blank user id then these parameters will need to be configured. -yy This is the century marker parameter, needed for Y2K. For the character client, the parameters are kept in /rd/opsys/client.pf The only difference between parameter file for the SX.enterprise client and the SX.enterprise editor client is the editor.pf does not use the -q parameter. 127

128 For the graphical client the above parameters are kept in several different parameter files: The Command line - This contains the initial call to the.pf files and sets the location for the Progress client workspace files. The following parameters are found here: -pf login.pf -T "%TEMP%" C:\nxt\code\login.pf - This contains the calls to the other parameter files in the appropriate order and then the call to the li.p program for logging into SX.enterprise. The following parameters are found here: -pf..\local\netpath.pf -pf startup.pf -p li.p C:\nxt\local\netpath.pf - This contains the necessary information to connect the SX.enterprise client to the Progress Appserver. The following parameters are found here: -param "START, PROPATH=server-propath.ini ConnectParms='-H <hostname> - AppService appsrv -S 7182'" C:\nxt\code\startup.pf - This contains global parameters, and this file gets overwritten during the push process. The following parameters are found 128

129 here: -pf..\local\local.pf -rereadnolock -D 500 -devevent -filterocxevents -mmax nochkttnames C:\nxt\local\local.pf - This contains parameters that are local to this client only. This file does not get overwritten during a push process. The following parameters are found here: -TB 8 -pls C:\nxt\code\connect.pf - This contains database connection parameters. This parameter file must be called connect.pf, and is called from the code. The client uses this parameter file to connect to the database across the network. The following parameters are found here: -db nxt -N tcp -H dixie -S nxt -cache nxt.csh 128

130 -bi - This is the Cluster Size parameter. This parameter tells the truncate.bi script what size to set the before-image cluster size to. Infor s recommended value for this parameter is -bi This sets the cluster size to 1MB. -biblocksize - This is the Blocksize parameter. This parameter tells the truncate.bi script what size to set the before-image block size to. Infor s recommended value for this parameter is to have it the same as your database blocksize By changing the bi number, we can tune the database to be a little faster. Larger cluster sizes generally increase performance. However, they also have significant drawbacks which includes Increased disk space usage for the BI file and longer crash recovery periods. Cluster sizes from 512 to 4096 are the most beneficial and even as high as 16 MB may speed things up even more. We can use promon to see if we need to increase this. We ll examine that later when we see promon and the performance tuning section. 129

131 130

132 131

133 When it comes to database extents, there are 2 different types of database extents. Variable extents - These extents will grow as needed to accommodate more data. Fixed extents - These extents are pre-formatted when the database is created, and never grow. 132

134 If we imagine our database is like a 5 gallon bucket and the data in it are like M&Ms, where the green M&Ms are OE records, the blue ones are Inventory records, etc, As we add data, we are pouring M&Ms into that bucket. We are just adding data to that file. This results in data being written quickly. But it does have a down side. 133

135 134

136 With fixed extents, the database was already formatted with disk space when the database was created. When Progress needs to write the same record to a database using fixed extents, it simply writes that data directly to the disk. This we consider to be a 1-step write process. However, when we get to the end of that fixed extent, we can t add more data to the database, and it crashes to protect the integrity of the database and 135

137 We end up with M&Ms all over the floor when we overflow our bucket 136

138 The OS will format a little more space and then Progress may continue writing data to the database. If there still is not enough room, then Progress will need to get the OS involved again, to format a little more space, and then Progress may continue writing the data to the database. 137

139 138

140 139

141 This must continue to happen until all the data has been written to the database. Therefore, this can be looked at as a 2-step write process. So, it is slower than fixed extents. 140

142 141

143 This is like having our 5 gallon bucket footprint, but no height. Then when we add M&Ms, we need to duct tape a 1/8 high layer to our bucket. 142

144 143

145 Inodes are like doors into Fenway Park but for files on a Unix system. If 100,000 people tried to go see the Red Sox play and they only had one door to go in, we d see a big line outside the door. Likewise, when we have just one large file and 100,000,000 writes trying to happen, then this can result in poor system performance 144

146 When the OS needs to write to a file on disk, the inode for the file being updated must be locked. This is because the OS cannot allow multiple processes to write to the same file at the same time. If we add more entries into our stadium. With a multi-volume database, there are several extents, each with a different inode that may be locked. Therefore, a multi-volume database may have more than one update at a time, while a single-volume database may not. 145

147 By having a combination of fixed and variable extents, it gives us both the speed and flexibility we need. Database extents give us the flexibility to add more space to the database in a variety of ways. 146

148 We ve split up the database into various storage areas. Each storage area can be considered as a kind of mini database. 147

149 These 12 tables are the tables we consider the busiest and/or that hold the most data. By examining their function, we can see why that might be. Hopefully we are adding lots of orders! That s the OEEH and OEEL tables. We recommend that you analyze your database periodically to see if we should isolate any other tables. 148

150 We have 3 standard areas called default, transient and custom these areas contain all the rest of the tables and the fields that weren t part of our previous slide. 149

151 Each area has their own set of fixed and variable extents. Here we can see aret. The last extent, extent 9, is variable. Lets take a look at your system at the various files of the database. 150

152 151

153 Now we will take a look at how data gets added to the database at a database block level. This will offer you an understanding of what is going on inside the database when you add or delete records. For this discussion we will look at the following terms: High Water Mark - any blocks above the high water mark, have never had data in them. RM Blocks - RM or Record Management blocks are blocks that are considered to be full. RM Chain Blocks - RM or Record Management chain blocks are blocks that are considered to be partially full or now empty, but had data in them at one point. Empty Blocks - Empty blocks are blocks above the high water mark that have never had data in them before. 152

154 RM Blocks - These are full blocks, they are below the high water mark. 153

155 RM Chain Blocks - These are blocks that are partially empty or empty but have had data in them at one point. These blocks are indexed for speed in a chain. Each block has a pointer in it to the next block on the chain. This is so that no matter where these blocks are in the database, Progress can find the next one quickly. 154

156 Empty Blocks - These are empty blocks, they are above the high water mark and have never had data in them. 155

157 We are going to look at 2 different scenarios of adding a new record to the database. 156

158 157

159 158

160 159

161 160

162 Let s add a record to our database and see what happens. You can see our New Record waiting to be added to our database. The first thing that happens is that Progress tries to check the first 3 blocks to see if it has enough room in any of those blocks to write the new record. If Progress finds enough room for the new record, it writes the new record to the block. In this example, Progress fills the block, so the block gets taken off the RM chain. The record does not move anywhere in the database, it simply gets the flag removed that tells Progress this block is part of the RM/Free chain. 161

163 How full is full? Progress considers any block that is at least or more than 93% full to be a full block. Any blocks that are considered full get removed from the RM Chain and becomes simply an RM Block. Progress uses this almost full concept to allow a record to grow within the same block without having to be fragmented. It is important to note that during the dump and load procedure, Progress will reestablish this 7% buffer on the database blocks. This is why your database may actually be larger after a dump and load than it was before hand. 162

164 In this second scenario the block will not fit on the RM/Free chain 163

165 Let s add another record to our database and see what happens in this scenario. You can see our New Record waiting to be added to our database. 164

166 The first thing that happens is that Progress tries to check the first 3 blocks to see if it has enough room in any of those blocks to write the new record. 165

167 In this case Progress does not find enough room for the to write the new record to any of the first 3 blocks of the RM/Free chain. Progress then makes a decision and pulls the next Empty Block and adds it to the RM/Free Chain. 166

168 Progress then adds the new record to the newly added block. Since in this case, the block does not fill up, Progress leaves the block on the RM/Free chain. 167

169 Progress then moves the first 3 blocks of the RM/Free chain to the end of the chain. This is so that the same 3 blocks are not being checked over and over again. 168

170 This process is built for speed. If there were 10,000 blocks on the RM chain, you would not want to wait for potentially 10,000 blocks to be read every time Progress needed to add a record to the database. 169

171 Any new record that is going to be added to the database will have a maximum of 3 blocks checked to see if there is room for the new record. After the first 3 blocks have been checked, Progress makes a decision and grows the database as necessary, even if the fourth record that it would have checked would have held the record. The first 3 blocks of the RM chain always get moved to the end, this rotation is to keep Progress from always hitting the same 3 blocks over and over. With this new information, we can see why, if you delete or purge data from your database, you cannot guarantee that Progress will reuse that space. The only way to ensure that Progress reuses that space is to do a dump and load of the database. Progress considers blocks to be full when they are approximately 93% full. Progress will reestablish this 7% buffer when we do a dump and load of the database. If a dump and load of the database has not occurred in a while, the database may grow after a dump and load. 170

172 We have been talking all through this session about dump and loads. Let s finish this session by taking a deeper look at what a dump and load is all about. 171

173 A dump and load is a process to defragment the Progress database. When we took a look at Record Management, and we added a new record to the database, the procedure was designed for speed. The record management process was not designed to keep the data streamlined. The dump and load process is the process used to streamline and defragment the records of your database. Infor recommends doing a dump and load of the Progress database once a year. The dump and load is a safe procedure, but it may take a long time. It is basically a weekend project. The dump and load process will dump all of the data out of the database, the database will be deleted and then recreated, and then all of the data is loaded back into the database. The dump and load procedure takes approximately 4 hours per gigabyte of data. This process does go much faster on higher performance systems. 172

174 173

175 As data gets added to the fixed extents in a database, it is like pouring m&ms into buckets. As each fixed extent fills up, the data simply starts spilling into the next extent. It does not matter if the record is an OE record, or a PO record, they are all treated the same. 174

176 When we do a dump and load, we export all of the data from the database, then delete the database and create a new one, then we load all of the data back into the database. The data being loaded back into the new database is loaded table by table each using the primary index. Therefore, all of your AP records get put together in the database, then your AR records, the OE records, and so on This helps the performance of reports significantly. It is recommended that a dump and load be performed on the database at least once a year. Since all of the records of the same type are together in the database after a dump and load, reporting performance can be significantly increased. 175

177 How do I know if we need a dump and load? Using the tabanalys tool, we can determine if a dump and load is required. This will generate a report that tells us the scatter factor for each table. Generally, 1.0 is good and over 4 is high. If we have a lot of tables, or a few bigger tables, whose scatter factor is high, then a dump and load is recommended. Lets review your database now by running the tabanalys command Also, an online tablemove is an option that Progress recommends. Please review the Progress kbase for directions on that. 176

178 The Progress Knowledgebase is a great tool. You can access it from Right now, the URL is knowledgebase.progress.com, but that changes periodically. 177

179 178

180 179

181 180

182 181

183 According to Progress Promon reports APW writes as "buffers flushed" in several places. You want that number to be less than 10 for almost every checkpoint. 182

184 183

185 184

186 185

187 186

188 187

189 188

190 According to Progress: Duration for the last 8 checkpoints are displayed in promon's Checkpoints display. If they are at least a minute long, you are fine. Longer is ok but not needed. If they are shorter than a minute, you should increase the cluster size. 189

191 190

192 191

193 192

194 193

195 194

196 195

197 196

198 197

199 198

200 199

201 200

202 201

203 202

204 203

205 204

206 205

207 206

208 207

209 208

210 209

211 210

212 211

213 212

214 213

215 214

216 215

217 216