Relational Foundations for Functorial Data Migration

Transcription

1 Relational Foundations for Functorial Data Migration David Spivak, Ryan Wisnesky Department of Mathematics Massachusetts Institute of Technology {dspivak, DBPL October 27, 2015

2 Introduction In this talk I will describe an equivalence between a fragment of the relational data model (SPCU queries) and a fragment of the (extended) functorial data model (FQL queries): SP CU F QL The functorial data model (my name) originated with Rosebrugh et al. in the late 1990s. Schemas are categories, instances are set-valued functors. Spivak extends it to solve information integration problems. Sponsored by: OR grant AFOSR grant FA / 28

3 Category Theory A presentation of a category is a reflexive, directed, labelled, multi-graph and a set of path equations: f g M.f.f.g =.f.h h A set-valued functor assigns a set to each node and a function to each edge, such that the equations holds. M tbillu fpxq x ` 1 gpxq hpxq P Category theory was instrumental in the development of two extensions to the relational model, both of which inform work on language-integrated query (LIQ): The nested relational model generalizes sets to nested collections and is inspired by monads. Algebraic datatypes implement nested collections using recursion and are inspired by algebras. 3 / 28

4 The Functorial Data Model manager Emp works secretary Dept last first Dom name Emp ID mgr works first last q10 Al Akin x02 Bob Bo q10 Carl Cork Emp.manager.works Emp.works Dept.secretary.works Dept Dept ID sec name q CS x Math Dom ID Al Akin Bob Bo Carl Cork CS Math 4 / 28

5 Convention Omit Dom table, and draw edges Ñ f Dom as f : manager Emp works secretary Dept last first Dom name manager Emp works secretary Dept first last name 5 / 28

6 The Functorial Data Model (abbreviated) manager Emp works secretary Dept first last name Emp.manager.works Emp.works Dept.secretary.works Dept Emp ID mgr works first last q10 Al Akin x02 Bob Bo q10 Carl Cork Dept ID sec name q CS x Math 6 / 28

7 Functorial Data Migration A functor F : S Ñ T is a constraint-respecting mapping: nodespsq Ñ nodespt q edgespsq Ñ pathspt q and it induces three adjoint data migration functors: F : T -inst Ñ S-inst (like project) S F T I F piq : IF Set ΠF : S-inst Ñ T -inst (like join) F % Π F Σ F : S-inst Ñ T -inst (like outer disjoint union then quotient) Σ F % F 7 / 28

8 (Project) ame ame Salary 1 2 F ÝÝÝÑ Salary Age Age 1 ID ame Salary 1 Alice $100 2 Bob $250 3 Sue $300 2 ID Age F ÐÝÝ ID ame Salary Age a Alice $ b Bob $ c Sue $ / 28

9 Π (Join) ame ame Salary 1 2 F ÝÝÝÑ Salary Age Age 1 ID ame Salary 1 Alice $100 2 Bob $250 3 Sue $300 2 ID Age Π F ÝÝÑ ID ame Salary Age a Alice $ b Alice $ c Alice $ d Bob $ e Bob $ f Bob $ g Sue $ h Sue $ i Sue $ / 28

10 Σ (Union) ame ame Salary 1 2 F ÝÝÝÑ Salary Age Age 1 ID ame Salary 1 Alice $100 2 Bob $250 3 Sue $300 2 ID Age Σ F ÝÝÑ ID ame Salary Age a Alice $100 null 1 b Bob $250 null 2 c Sue $300 null 3 d null 4 null 5 20 e null 6 null 7 20 f null 8 null / 28

11 Foreign keys ame ame Salary 1 f 2 F ÝÝÝÑ Salary Age Age 1 ID ame Salary f 1 Alice $ Bob $ Sue $ ID Age F ÐÝÝ Π F,Σ F ÝÝÝÝÑ ID ame Salary Age a Alice $ b Bob $ c Sue $ / 28

12 Evaluation of the functorial data model Positives: The category of categories is bi-cartesian closed (model of the STLC). For each category C, the category C-inst is a topos (model of HOL). Data integrity constraints (path equations) are built-in to schemas. Data migration functors transform entire instances. The FDM is expressive enough for many information integration tasks. Easy to pivot. egatives: Data integrity constraints (in schemas) are limited to path equalities. Data migrations lack analog of set-difference. o aggregation. Data migration functors are hard to program directly. Instance isomorphism is too coarse for many integration tasks. Many problems about finitely-presented categories are semi-computable: Path equivalence Generating a category from a presentation 12 / 28

13 The Attribute Problem ID ame Age Salary 1 Alice 20 $100 2 Bob 20 $250 3 Sue 30 $300 (good) ID ame Age Salary 4 Alice 20 $100 5 Bob 20 $250 6 Sue 30 $300 (bad) ID ame Age Salary 1 Amy 20 $100 2 Bill 20 $250 3 Susan 30 $ / 28

14 Solving the Attribute Problem Mark certain edges to leaf nodes as attributes. In this extension, a schema is a category C, a discrete category C0, and a functor C 0 Ñ C. Instances and migrations also generalize. Schemas become special ER (entity-relationship) diagrams. The FDM takes C 0 to be empty. The example schema below, which was an abbreviation in the FDM, is a bona-fide schema in this extension: attributes are first, last, and name. manager Emp works secretary Dept first last name 14 / 28

15 Solved Attribute Problem ID ame Age Salary 1 Alice 20 $100 2 Bob 20 $250 3 Sue 30 $300 (good) ID ame Age Salary 4 Alice 20 $100 5 Bob 20 $250 6 Sue 30 $300 fl (good) ID ame Age Salary 1 Amy 20 $100 2 Bill 20 $250 3 Susan 30 $ / 28

16 Functorial Data Migration as SPCU Theorem: migrations of the form Σ F Π G H F is a discrete op-fibration (ensures union compatibility). G is a surjection on attributes (ensures domain independence). all categories are finite (ensures computability). can be implemented using SPCU (select, project, cartesian product, union) and keygen, under set semantics. are closed under composition. 16 / 28

17 using SPCU Given F : S Ñ T and I P T Inst, define F piq P S Inst as: for each node in S, the table F pq is IpF pqq. for each attribute A in S, the table F paq is IpF paqq. for each edge e : X Ñ Y in S mapping to a path F peq : F pxq Ñ F py q in T, compose IpF peqq to obtain F peq. S F T I F piq : IF Set 17 / 28

18 Σ using SPCU Gven F : S Ñ T a discrete op-fibration, a S-instance I, we define Σ F piq P T Inst as for each node in T, the table Σ F pq is the union of the node tables in I that F maps to. for each attribute A in T, the table Σ F paq is the union of the attribute tables in I that F maps to A. Let e : X Ñ Y be an edge in T. We know that for each c P F 1 pxq there is at least one path p c in S such that F pp c q e. Compose p c to a single binary table, and define Σ F peq to be the union over all such c. The choice of p c will not matter. 18 / 28

19 Discrete Op-Fibrations / Union Compatibility b 1 g a 1 h 1 1 c 1 b 2 g 2 a 2 h 2 c 2 g 3 a 3 h 3 c 3 c 4 Ó F B G A H C 19 / 28

20 Π using SPCU Given F : S Ñ T with S finite and a S-instance I, we define Π F piq P T Inst as: too difficult to describe in a presentation. Intuitively, Π is a join all 20 / 28

21 SPCU as Functorial Data Migration Theorem : SPCU (bags) can be implemented using, Σ, Π. Theorem : SPCU (sets) can be implemented using, Σ, Π, dedup, where dedup T : T Inst Ñ T Inst equates IDs which cannot be distinguished along any attribute path. We must encode relational schemas, for example, Rpc 1,..., c n q and R 1 pc 1 1,..., c1 n 1 q becomes: R c n c 1 n 1 R 1 c 1 D c 1 1 A 21 / 28

22 Project using We express π i1,...,i k R as F : πr R i 1 i k F ÝÝÝÑ c 1 c n D D 22 / 28

23 Select using, Π We express σ a b R as F Π F : R σr a c 1 c n b F ÝÝÝÑ x c 1 c n D D Here F paq F pbq x and F pc i q c i for 1 ď i ď n. 23 / 28

24 Product using Π We express R ˆ R 1 as Π F : R c 1 c n D c 1 n 1 R 1 c 1 1 F ÝÝÝÑ c 1 c n RˆR 1 c 1 1 D c 1 n 1 24 / 28

25 Union using Σ We express R ` R 1 as Σ F : R c 1 c n D c 1 n R 1 c 1 1 F ÝÝÝÑ c 1 R`R 1 D c n 25 / 28

26 FQL - A Functorial Query Language The open-source, graphical FQL IDE available at categoricaldata.net/fql.html implements functorial data migration (with attributes) in software. FQL translates migrations of the form into SQL and vice versa. Σ F Π G H Demo 26 / 28

27 FQL evaluation Positives: Attributes. Running on SQL enables interoperability and execution speed. Better Σ semantics than TGD-only systems (e.g., Clio). egatives: o selection by constants. Relies on fresh ID generation. Cannot change type of data during migration. Attributes not nullable. See our follow-up work for solutions to these problems. 27 / 28

28 Conclusion I described the functorial data model and data migration functors, how to extend the functorial data model to have attributes, an equivalence SP CU F QL where F QL is a fragment of the data migration functors a tool, FQL (categoricaldata.net/fql.html) based on this equivalence. 28 / 28