Blackboard Learn TM, Release 9 Technology Architecture John Fontaine
Overview Background Blackboard Learn Deployment Model and Architecture Setup and Installation Common Administrative Tasks Tuning Integrating Monitoring
Background
Design Philosophy and Vision Scalable Secure Reliable Agile Usable Blackboard Learn Open 4
Java Server Basics The Language Java is a programming language developed by Sun for cross platform enterprise computing The Virtual Machine Java programs run inside a virtual machine that manages memory and resources independent of the underlying OS The Container The container (tomcat, Weblogic, etc) provides a common set of services used by application components to deliver functionality to the end user Java EE A set of Java APIs that provide additional features for application development. Bb Learn uses a subset of Java EE (servlet) using the Tomcat container; but does not use full heavyweight J2EE (e.g. no JMS, no EJB)
Notes for CE 4 Clients CE 4 Data Storage Flat File Simple File Storage Large File Storage Files stored on file system Application Container Apache / IIS CGI Scripts Programming Language(s) Perl C API Web Services (SIAPI) Load Balancing Required Services Blackboard Learn Data Storage RDBMS (Oracle / SQL Server Some DBA skills needed Large File Storage Xythos File System* Application Container Tomcat Programming Language Java API Web Services (Proxy Tools) Installed Tools Snapshot Load Balancing Supported Configuration Available for all enterprise license holders for unlimited nodes.
Notes for CE/Vista 8 Clients CE 8 Data Storage RDMBS File Storage In Database Application Container Weblogic Programming Langauge Java (J2EE with EJB) API SIAPI (Web Services) Blackboard PowerLinks (Web Services) Load Balancing Weblogic Clustering / Admin Node License restrictions Blackboard Learn Data Storage RDBMS File Storage Xythos File System* Application Container Tomcat 6 Programming Language Java (Servlets) API Proxy Tools (Web Services) Installed Tools Snapshot Load Balancing Unlimited nodes Separate instances
Note for Blackboard Classic Clients Blackboard Academic Suite, Release 8 Data Storage RDBMS File Storage Xythos and Plain Disk mixture Application Container Tomcat 6 Programming Language Java (servlets) Perl API Installed Tools Snapshot Load Balancing Unlimited nodes Separate instances Blackboard Learn Data Storage RDBMS File Storage Xythos File System* Application Container Tomcat 6 Programming Language Java (Servlets) API Proxy Tools (Web Services) Installed Tools Snapshot Load Balancing Unlimited nodes Separate instances
Deployment and Architecture
Architecture Framework Design Deployment Operations Monitoring Systems Technologies Infrastructure Planning Governance Release Lifecycle Application Regulatory Security 10
System Design 11
Deployment Planning Functional Data Integration Performance Deployment Rollout Training Verification Database Maintenance Production Upgrade Staging Server Check Live Integration Configuration / Setup Integration Planning Optimization Tuning Hardware Capacity Plan Customer Development QA Test Execution Auditing Coding Data QA Sample Data Generation Update SQL Unit Test Execution Sample B2 / Integration Data Unit Test Harness Hardware Sizing Guides Performance Verification Test Simulation Development Blackboard Requirements Data Model API Development Usage / Sizing Model 12
Governance of Operations Key Drivers Integrity & Security Best Practice Alignment Leverage Assets and Infrastructure Manage Risks and Change Design Deploy Implement
Blackboard Learn Product Lifecycle Service Pack 1 Service Pack 3 Release 1.0 Development Beta Priority Maintenance Secondary Maintenance Continued Support Continued Support General Availability Service Pack 2 Service Pack 1 Service Pack 3 App Packs Development Beta Priority Maintenance Secondary Maintenance Continued Support Continued Support General Availability Service Pack 2 Service Pack 1 Service Pack 3 Release 1.2 2.0 Development Beta Priority Maintenance Secondary Maintenance Continued Support Continued Support General Availability Service Service Pack Pack 2 2 14
Security Methodology Identify Assets and Threats Support & Maintenance Secure Design Monitoring Develop & Test Deploy Incident Response Planning
Countermeasures Assessment Top Threats + Vulnerability SQL Injection Remote Code Execution URL Access Restrictions Cross Site Scripting Cross Site Request Forgery Direct Countermeasures Additional Customer Responsibilities Session Hijacking Insecure Configuration Management Authentication and Session Management Denial of Service Buffer Overflows 16 + Defined by Open Web Application Security Project (OWASP)
Keeping Blackboard Learn Secure Be on the latest release Communicate your security policy Identify critical areas of concern promptly Security-Alerts@blackboard.com 17
Setup and Installation 18
Setup and Installation: Technical Skills Inventory Project Management Responsibilities: Manage overall project schedule, deliverables, budget, and resources. Skills Required: IT implementation and change management experience Database Administration Responsibilities: Manage the database for Blackboard Learn, Release 9 Skills Required: Oracle or SQL Server database administration System and Network Administration Responsibilities: Administer Blackboard Learn, Release 9 and the supporting system infrastructure Skills Required: Intermediate-level system administrator with expertise in TCP/IP networking, java and experience with OS platform Application Administration Responsibilities: Manage the Blackboard Learn, Release 9 application Skills Required: Blackboard Learn, Release 9 features and functions Programming (Optional) Responsibilities: Execute integration between Blackboard Learn, Release 9 and your student information system, portal, authentication system, etc. Skills Required: Programming expertise in Java, Web Services and XML, as well as familiarity with involved products and APIs
Setup and Installation Types of Environments Development used for creating building blocks and testing customizations Test/Staging model s production used for upgrades, training, bug replication and restoring backups Production the live system 20
Setup and Installation: Documentation Blackboard Learn, Release 9 Documentation https://behind.blackboard.com/shared/r9/bb9.aspx Upgrade Process Overview Specific manuals for CE4 / CE8 Setup Guide Choose your configuration (load balanced, single server, etc) Review the Sizing Guides Use the Pre-installation worksheets Sizing Guides Determine hardware requirements based on capacity Server Administration Guide Performance Tuning SSL Authentication Schemes Tomcat Clustering Advanced Integration and Data Management Note: co-production allows you to keep existing integration intact 21
Setup and Installation Single Server Typical of Development Environments Also suited to small scale deployments (under 500 concurrent users)
Setup and Installation Multi-Server Separate all elements Well suited for staging environments Used by many mid- sized institutions
Setup and Installation Load Balanced Typical high performance / high availability configuration Scale out with multiple tomcat instances No Admin Node Single Collab Server
Advanced Setups Database Clustering Scalability and Redundancy Oracle RAC Supported via Services engagement only Supports failover and clustering SQL Server Clustering Available out of the box Used for failover only Tomcat Clustering Support multiple Tomcat instances on a single server Maximize use of memory and CPU See Server Administrator Manual Page 76
Common Administrative Tasks
Common Administrative Tasks Configuration Settings NOTE: Default configuration settings are not tuned for production config/bb-config.properties Master Configuration file Updates other components via PushConfigUpdates tool Configured / application server config/authententication.properties Controls Authentication Integration (LDAP, ActiveDirectory, etc) Other Configuration Files Note Changes to non.bb files will be over written by PushConfigUpdates Startup and Shutdown Always use ServiceController tool (tools/admin) services.restart services.stop services.start
Common Administrative Tasks Managing Background Tasks config/bb-tasks.xml.bb Session Expiration Expiration in Milliseconds via invalid Cartridge Download View from Admin Panel Email Email is logged in logs/bb-email-log.txt System Task Status Page http://<url>/webapps/blackboard/admin/view_system_ta sks.jsp
Tuning
Tuning Overview Basic Application Tuning Items Java JVM Options Tomcat Thread Settings Database Connection Settings Clustering
Tuning Java JVM Settings Manage in config/bb-config.properties Control the settings for the JVM including heapsize, permsize, stacksize and any additional options Windows Recommendations bbconfig.min.heapsize.tomcat=1378m bbconfig.max.heapsize.tomcat=1378m bbconfig.max.permsize.tomcat=256m bbconfig.max.stacksize.tomcat=128k bbconfig.jvm.options.extra.tomcat=-xloggc:c:/bb/blackboard/logs/tomcat/gc.log - XX:NewRatio=4 -XX:+UseParallelGC -XX:ParallelGCThreads=4 -XX:+PrintGCDetails - XX:PermSize=256m Linux Recommendations bbconfig.min.heapsize.tomcat=3328m bbconfig.max.heapsize.tomcat=3328m bbconfig.max.permsize.tomcat=256m bbconfig.max.stacksize.tomcat=128k bbconfig.jvm.options.extra.tomcat= -Xloggc:/foo/bar/logs/tomcat/gc.log -XX:NewRatio=2 - XX:+UseParallelGC -XX:ParallelGCThreads=4 -XX:+PrintGCDetails -XX:PermSize=256m
Tuning Tomcat Thread Settings config/bb-config.properties bbconfig.jk_connector.pool_size Ideally equals bbconfig.appserver.maxthreads and bbconfig.database.bb_60.db.maxpoolsize bbconfig.appserver.minthreads Minimum number of threads allocated to tomcat. bbconfig.appserver.maxthreads Maxmimum number of requests tomcat can simultaneously process. Should equal bbconfig.jk_connector.pool_size and bbconfig.database.bb_60.db.maxpoolsize Typically set to 500
Tuning Data Stores config/bb-config.properties Configure how Blackboard software talks to the database Pool Size Oracle will need to be configured to accept total number of connections Max Size should equal max threads and jk_connections bbconfig.database.bb_60.db.minpoolsize recommended value 1 bbconfig.database.bb_60.db.maxpoolsize recommended value 500 bbconfig.database.bb_60.db.timeout Sets the length of time in seconds a query can execute before the connection is timed out. Typical value is 300
Tuning Vertical Clustering Run Multiple JVM s on a single server Allow for efficient memory management avoid GC wait times config/bb-config.properties bbconfig.tomcat.enable_cluster =true tools/admin/installcluster.(sh bat) <cluster_id> <shutdown port> <jk connector port> <listen port> See Server Administration Manual for more details (p76)
Integration
Integration Authentication Settings See: Server Administration Manual for details config/authentication.properties Built in Authentication Schemes RDBMS (Blackboard Learn Native) LDAP Webserver Delegation ActiveDirectory Shibboleth Custom Authentication Extensions Extend Existing Authentication Schemes Create Your Own
Integration Snapshot Command Line Utility for Batch Data Integration See: Blackboard Learn Advanced Data and Integration Management Manual Support for XML (IMS Enterprise 1.0 extracts) Flat File (defined by user) Connection Modes JDBC Direct to database writes requires firewall rules SOAP XML-RPC based interface for remote connections no firewall changes Snapshot Controller Blackboard Consulting provided manager for Snapshot
Integration Blackboard Building Blocks Documentation Getting Started Guide API Specifications Introduction to the Runtime API Module Developers Guide http://www.edugarage.com Installed Tools Run Inside Blackboard Application Container Leverage Authentication and other Blackboard Frameworks Event API Allows for real time integration events Proxy Tools Connect to Blackboard Learn via Web Services Designed for existing applications with their own container
Application Monitoring Overview Usage and Activity System Reports Detailed Log Files Reporting Database Service Quality Quest Foglight Coradient TrueSight Up.Time 39
System Reports Built in reports view data in real time ASR Database Snapshot of production Crystal Report Templates provide dataviews Can be integrated into data warehousing solutions 40
Log Files Location logs/* Rotation Log files are auto rotated by background task and archived Process should be established to cleanout logs Specific Log files bb-email-log.txt tracks emails sent by the system bb-services-log.txt tracks background services logs/tomcat contain regular reports Note error messages on the screen will often have a GUID which corresponds to an entry in the tomcat log. 41
Monitoring JMX The JMX API provides insight into the internals of Tomcat How to Enable Edit bb-config.properties How to use See Edugarage Presentation http://www.edugarage.com/download/attachments/ 13271042/DevCon_BbJMXJConsoleGlassbox_Gr ay_final_bbbc_7.13_2.30.pdf
Monitoring Enable Verbose Java Garbage Collection bbconfig.jvm.options.extra.tomcat= -Xloggc:C:/Bb/blackboard/logs/tomcat/gc.log -XX:+PrintGCTimeStamps -verbose:gc How to Use GC Portal -XX:+PrintGCDetails http://java.sun.com/developer/technicalarticles/pro gramming/gcportal/readme.html Model Object lifetime and GC Pausing Adjust generation settings and heap size.
Monitoring Advanced Monitoring Tools Coradiant TrueSight Rack mounted hardware providing advanced monitoring capabilities http://www.coradiant.com/assets/pdfs/data_sheets /Blackboard_COR_DS_TSBB.pdf Quest Spotlight Monitors database operations http://www.quest.com/public-sector/docs/bbperformance-engineering-spotlight.pdf
Q&A Questions