What is a database system? Course introduction. In the beginning. Early computing challenges

Transcription

1 What is a database system? 2 Course introduction Introduction to databases CSCC43 Winter 2013 Ryan Johnson Database: a large, integrated collection of data. Models a real world enterprise Entities (teams, games) Relationships (Orphan Pamuk received the Nobel Prize) Constraints (at least one doctor on duty during off hours) More recently, active components ( business logic ) Database Management System (DBMS): a software system designed to store, manage, and facilitate access to databases. Thanks to Arnold Rosenbloom and Renee Miller for material in these slides In the beginning 3 Early computing challenges 4 There was The Mainframe Cost: millions Watts: millions Size: acres Speed: 40kHz Memory: 2kB Storage: 3.5MB (tape) Time sharing ~100 terminals per mainframe Users share hardware Want to share data, too Bare hardware No OS No device drivers No file system SAGE (1954) SABRE (1960) UNIVAC (1951) Few organizations could afford two! => The Database => File Management System 1

2 The Database 5 File management systems (FMS) 6 Abstract concept dating back to the 1950 s File management ca Centralized repository for all the enterprise s data Realtime updates from many sources Concurrent access by many users Interactive (ad hoc) exploration and reporting Semi Automatic Ground Environment (SAGE) File: box of punchcards Metadata: label on the box Ad hoc report: no big deal Hardware change: no big deal File management ca Computer aided tracking and interception of aircraft Dozens of SAGE installations (big one in North Bay) Hundreds of radar stations throughout North America Thousands of operators File: several km of magnetic tape Metadata: embedded in application logic Ad hoc report: hire a couple programmers Hardware change: hire a dozen programmers Goal: all relevant information at your fingertips Huge need for portability, abstraction Database Management System 7 Why study databases?? File management systems meet The Database Shift from computation to information Protect users from each other (isolation, consistency) Protect application from data changes (at logical level) Protect data from hardware changes (at physical level) Split personality remains to this day Theory/applications (declarative access to changing data) Systems (make it run fast on ever changing hardware) Why so important? Rate of change of DB applications is incredibly slow dapp/dt << dplatform/dt always true for corporate computing Web made this point for personal computing more and more true for scientific computing Need for DBMS has exploded Corporate: retail swipe/clickstreams, customer relationship mgmt, supply chain mgmt, data warehouses, etc. Scientific: digital libraries, Human Genome project, Sloan Digital Sky Survey, physical sensors, grid physics network A practical discipline spanning much of CS OS, languages, theory, AI, multimedia, logic Yet with a focus on real world apps This semester: the theory/application side 2

3 What s the intellectual content? 9 Is the WWW a DBMS? 10 Representing information data modeling Languages and systems for querying data complex queries with real semantics* over massive data sets Concurrency control for data manipulation controlling concurrent access ensuring transactional semantics Reliable data storage maintain data semantics even if the lights go out Fairly sophisticated search available Crawler indexes pages on the web Keyword based search for pages But Data is mostly unstructured and untyped Search only (can t modify, summarize, analyze, correlate, ) Few (zero) guarantees of freshness, accuracy, durability, consistency DBMS lurking behind most Web sites provides these functions The picture is changing New standards like XML can help data modeling The WWW/DB boundary is blurry! * semantics: the meaning or relationship of meanings of a sign or set of signs Search vs. Query 11 Is my file system a DBMS? 12 What if you wanted to find out which actors donated to Steven Harper s campaign? Try actors donate to harper campaign in your favorite search engine. Stephen Harper (politician) or Hill Harper (actor)? Did Harper give or receive the donation? Year? Comparison with other donations? Strong shared heritage Direct descendant of file management system Excellent insulator against hardware changes But Data is mostly unstructured and untyped No concept of constraints, relationships Minimal support for atomicity, isolation, consistency The picture is changing File systems adopting database concepts (logging, transactions) Object oriented file systems provide finer grain data model The FS/DBMS boundary is blurry! 3

4 Database vs. file system 13 OS support for data management 14 Thought experiment #1 You and your project partner are editing the same file. You both save it at the same time. Whose changes survive? A) Yours B) Partner s C) Both D) Neither E) Who knows Thought experiment #2 You re updating a file when the lights go out Which of your changes survive? A) All B) None C) All since last save D) Who knows How to code against who knows??? Very, very carefully Again, strong shared heritage Another direct descendant of file management system Powerful API abstractions Bring your favorite programming language Enforces protections on files, objects But Scheduling, resource management inadequate for big data Error handling: program terminated with SIGSEGV Ad hoc query? Hire a programmer Concurrency? Write code very, very carefully DBMS vs. {OS, FS, WWW} Key services missing from some or all Recovery, isolation, consistency Support for ad hoc queries Effective concurrency control Preserve semantics across crashes, outages SMOP? Simple matter of programming? Not really (we ll see this semester) In fact, OS/FS often get in the way (next semester) Analogy: Memory management in C++ vs. Java Misquoting Greenspun s tenth rule: Any sufficiently complex data processing system resembles a buggy, half implemented, and poorly performing DBMS 15 Concept: transaction Business transaction Old idea: withdraw money, reserve seats, escrow, etc. Atomic: I deliver and you pay, or neither Consistent: Sell each seat to only one person Isolated: Doctor doesn t talk about the patient next door Durable: Sales receipt, confirmation number, etc. Database transaction Sequence of reads and writes to underlying data Writes [appear to] take effect atomically Each transaction moves the system between consistent states** Transactions can t see (or interfere with) each other Once the system returns success it will not lose the data ** user responsible to write sane transactions Formalized into an entire programming model 16 4

5 Concept: concurrency control Concurrent execution: key to high performance. Disk accesses frequent, pretty slow Keep the CPU working on several programs concurrently Interleaving two programs actions: trouble! Print statements during active account transfer He and She both withdraw the last $100 from the ATM DBMS ensures anomalies don t arise Give users/programmers illusion of a single user system Thank goodness! Don t have to program very, very carefully. Concept: data models Data model: a collection of concepts for describing data. Schema: a description of a particular collection of data, using a given data model. Many possible data models Network, hierarchical, relational, object oriented, The relational model is the most widely used today A good data model is key to data independence Concept: data independence 19 Advantages of a DBMS 20 FMS (1950 s) File, metadata management Hardware abstraction layer CODASYL/DBTG (1965) Decouple application from schema Decouple schema from physical data layout Edgar Codd (1970) Relational algebra Move from procedural to declarative Charles Bachman (1973) Programmer navigates data instead of (merely) writing code Move from machine centric to data centric programming Fast forward to today SQL, ODBC/JDBC, federation, web services, Data integration, cleaning, performance tuning, Data independence Efficient data access Data integrity & security Data administration Concurrent access, crash recovery Reduced application development time So why not use them always? Expensive/complicated to set up & maintain Cost & complexity must be offset by need General purpose, not suited for special purpose tasks (e.g. text search!) Big Deal but still a work in progress 5

6 Databases make these folks happy... DBMS vendors, programmers Oracle, IBM, MS, Sybase, NCR, End users in many fields Business, education, science, DB application programmers Build enterprise applications on top of DBMSs Build web services that run off DBMSs Database administrators (DBAs) Design logical/physical schemas Handle security and authorization Data availability, crash recovery Database tuning as needs evolve Summary (part 1) DBMS marries two very old concepts The Database (idealistic vision) File management system (imminently practical) Benefits Maintain, query large datasets Manipulate data and exploit semantics Recover from system crashes, Juggle concurrent access, automatic parallelization Quick application development Preserve data integrity and security Powerful abstractions provide data independence Application safe from changes to data organization, hardware Key when dapp/dt << dplatform/dt Summary (cont.) Course administrivia 24 DB administrators, developers are the bedrock of the information economy Data management R&D spans a broad, fundamental branch of the science of computation This semester: become an effective DBMS user Professor Ryan Johnson (IC481) Office hours: by appointment TA Cicely Zhang cscc43 instructor@cs.utoronto.ca No promises for class related sent elsewhere My main gets hundreds of messages, things get lost Course web site: s13/ 6

7 What is an inverted classroom? Students read lectures outside class Knowledge transfer is the first part of learning Avoids long in class lectures about dry material Instructor as tutor Focus: Q&A, review of tricky material, hands on practice Instructor gives you what Google can t Formative assessment Many assignments marked for effort, not correctness Opportunity to learn from mistakes Some peer marking (see syllabus for privacy issues) Course marks Formative/Participation (15%) Gain marks: in class assignments and activities Lose marks: distract or disturb others, come unprepared => Reflects fact that lectures are hands on and participatory In class homework and quizzes (15%) Lots of small ones (< 1% each) Encouragement to read before class Midterm exam (28%) 2h, week of 4 or 11 Feb (still waiting for room assignment) Final exam (42%) 3h (time/place TBA) Comprehensive, 2h for new material A few do s and don ts 27 Do read materials before class! take assignments seriously ask questions if you don t understand something bring your laptop/tablet to class (we ll use it!) Don t expect a high mark if you ignore reading/class/assignments hand in other peoples work (it s cheating) harass others (it s University policy) distract or disrupt the class (it s immature) 7