SQL Developer Data Modeler 3.0 Under The Covers: The Reporting Schema Marc de Oliveira, Simplify Systems Introduction This article will show you how I have used the SQL Developer Data Modeler (SDDM) Reporting Schema for much more than simple reporting. By applying a little bit of intelligent pl/sql I turned my data models into natural language descriptions of my business architecture, business terminology definitions etc. I have even integrated my model descriptions and definitions into the on-line help and other functionality of APEX 4.0 applications. Only your imagination limits the possibilities for reusing your model information. In the following I will convince you that data modeling and Architectural Software Development using SDDM 3.0 is certainly worth every minute you invest in it! SQL Developer Data Modeler Overview The SDDM is primarily for Data Modeling but it also supports diagramming of Process Models and simple documentation of some business information, such as documents, persons and locations. The processes and business information are only supported on the conceptual level, while data is supported on conceptual (in SDDM called logical), design (in SDDM called relational), and physical levels. So, Data Modeling is clearly the main focus of the tool but I will cover some of the other areas, too, as they can be very helpful when developing and maintaining complex application systems. The Reporting Schema Unfortunately, SDDM does not use a database repository, so you cannot access all your valuable models and documentation from within your database applications. This is quite disappointing, when the tool comes from the world's largest database vendor. Instead SDDM uses an XML repository that allows for very primitive versioning support using Subversion. Still, SDDM offers the next best thing: It integrates with a so-called Reporting Schema, which is a database schema of 77 tables that mirrors most of the content of the SDDM models, allowing users to build their own database reports based on their SDDM models. The Reporting Schema is maintained by the Export function, like this: File Export To Reporting Schema (then select a database connection and click OK) There are a number of down sides to having a Reporting Schema instead of the actual repository being in a database: The Reporting Schema is not complete, so some of the elements of your model are not included in the Reporting Schema (such as the business information like documents and locations), and in a few cases parts of the model are not transferred correctly. The Oracle team is working on this, but I expect that as new features come to the SDDM, the Reporting Schema will probably always lag behind to some extent. It takes a lot of time to export your model, when you start to get a lot of content (which is when reusing your model has the most value). With 200 entities and 400 processes I am 1 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
waiting more than a minute for each export. When supporting a group of developers that are depending on your updates of the model, it does become cumbersome that others cannot access the updates to the model as they happen. Unfortunately, it is a one way transaction. You cannot update the SDDM model by updating the Reporting Schema. It would have been of tremendous value to be able to update the model from one's PL/SQL applications. This would have been possible had SDDM used a database repository. Besides these drawbacks, the Reporting Schema is a very strong feature of the SDDM that can be used to integrate and communicate your business knowledge in many ways. Understanding the Reporting Schema The first time you look at the 77 tables of the Reporting Schema it may seem a little overwhelming, and it does not help that the model behind the tables is not very consistent. Without going into too much detail about each table, let me try to explain how you should understand the Reporting Schema. Design, Model, and Model Subview. Basically, every time you select File Save in SDDM, what you are saving is called a Design (see Illustration 1: Design). Illustration 1: Design The next level of detail are the different models that you may define. Models can be Logical Models, Multidimensional Models, Relational Models, or Process Models. A Design always contains exactly one Logical Model and one Process Model but it can have any number of Multidimensional Models and Relational Models (see Illustration 2: Model). 2 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
Illustration 2: Model The main diagram element of the SDDM is the Model Subview. Each model may be displayed through one or more Model Subviews (see Illustration 3: Model Subview). Illustration 3: Model Subview Exporting a Design When you are saving the Design with File Save, you are just saving a single copy of the Design. When you export the Reporting Schema (with File Export To Reporting Schema) you are exporting a version of the Design. With Save you disregard how the Design looked previously. With Export you store a new copy of the Design every time. This may seem like a small difference, but it does create some important implications to the required database structure of the Reporting Schema. 3 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
Firstly, a Design Export does not correspond to a Design as each Design can be exported many times (see Illustration 4: Design Export). Illustration 4: Design Export The primary key of a Design is DESIGN_ID, while the primary key of a Design Export is DESIGN_OVID. Even though the tables of the Reporting Schema are named much like Illustration 3: Model Subview the content of the tables is more structured as Illustration 5: Model Subview Export. 4 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
Notice how the entities Design, Model and Model Subview are mirrored in Design Export, Model Export and Model Subview Export, with ID columns mirrored as OVID columns. For example. the diagram shows how a Model Export is always both part of a specific Design Export and the export of a specific Model. Illustration 5: Model Subview Export The Reporting Schema table called DMRS_DESIGNS corresponds to Design Export, DMRS_MODELS corresponds to Model Exports, and DMRS_MODEL_SUBVIEWS corresponds to Model Subview Export. The tables of the Reporting Schema do not include the right side of Illustration 5 (Design, Model, and Model Subview) but they do include the foreign keys to them (DESIGN_ID, MODEL_ID, and SUBVIEW_ID) as if twice as many tables existed. Another thing that can be seen on Illustration 5 is that some columns (such as DESIGN_NAME) are duplicated in multiple tables. This redundancy is basically implemented to simplify queries against the Reporting Schema, ie looking up the DESIGN_NAME from the Design Export linked to a Model Export will always return the same name as the DESIGN_NAME stored directly on the Model Export. Another inconsistency of the Reporting Schema is that primary key columns are some times named after the table they belong to (such as DESIGN_OVID, MODEL_OVID etc) but other times they are just called OVID (examples of this are the tables DMRS_ENTITIES, DMRS_ATTRIBUTES, and DMRS_DOMAINS). Also the names for foreign key columns to the (non existing) right side tables are some times named from the table they reference (such as DOMAIN_ID in DMRS_DOMAINS), and some times they are just called OBJECT_ID (such as OBJECT_ID in DMRS_ENTITIES). If you keep these things in mind, using the Reporting Schema should be pretty straight forward. A first step could be to create a set of views that help isolate a single SDDM design export. 5 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
Isolating a Design Export When building a report that shows your current SDDM Design, you are not really interested in all the exports stored in the Reporting Schema tables, so you will have to isolate the most resent record in Design Export that matches the Design you are interested in. To help you with this, you could create a set of views that represent the missing right side tables. First you need to isolate the current designs, like this: create or replace view CURRENT_DESIGNS as select * from DMRS_DESIGNS DATE_PUBLISHED in (select max(date_published) from dmrs_designs group by design_id); Now, the rest of the views can be derived from the CURRENT_DESIGNS view, like this: create or replace view CURRENT_MODELS as select DMRS_MODELS.* from DMRS_MODELS, CURRENT_DESIGNS DMRS_MODELS.DESIGN_OVID = CURRENT_DESIGNS.DESIGN_OVID; create or replace view CURRENT_MODEL_SUBVIEWS as select DMRS_MODEL_SUBVIEWS.* from DMRS_MODEL_SUBVIEWS, CURRENT_DESIGNS DMRS_MODEL_SUBVIEWS.DESIGN_OVID = CURRENT_DESIGNS.DESIGN_OVID; Etc. With these views it becomes quite simple to extract Design information for use outside SDDM. Let us look at some examples. Using SDDM Models through PL/SQL An obvious usage of the SDDM Reporting Schema is to generate easy-to-read documents that explains your models to non-it people. In the following I will describe a few usages of SDDM models extracted from the Reporting Schema in PL/SQL to be used as simple reports or HTML output on your company's intranet. Creating a Terminology List One of the most basic artifacts, with huge value to a business, is the Terminology List. A list of terms used in the business with descriptions of what each term means. Such a list helps new employees understand the terminology used by other employees. All employees, and the business as a whole, benefit from it when everybody use the same terminology, and assign the same meaning to the terms they use. Obviously, communication becomes much easier, as employees on all levels can avoid the complications that would result from every area in the business defining their own terminology, or even worse if every employee had a different interpretation of some of the terms used by the business. This is exactly the purpose of the information analyst/data modeler. She has to understand the 6 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
meaning of all things of significance to the business, identify when different persons use different terms about the same thing, and when different persons use the same terms about different things. So, now that the information analyst/data modeler has collected all this important business knowledge in a consistent information/data model, it would be of tremendous value to make this knowledge available to the entire business in an easy way every term can easily be looked up to get an explanation of what it means. Such a dictionary, thesaurus, or terminology list can easily be created from the SDDM Reporting Schema. The Elements of a Terminology List Obviously, every entity should represent a term of significance to the business. If they were not significant to the business, we would not model them in the first place. So terms like PRODUCT, PRODUCT PRICE, SERVICE CALL, ENROLLMENT etc all need to be explained and included in the business's Terminology List. The same thing can also be said about every attribute of each entity. Why are we specifying an attribute in a data model? Because the business needs to keep track of it, of course. By being a little clever about naming the attributes, they, too, can be part of a Terminology List. Terms such as PRODUCT DECLARATION, PRODUCT PRICE CURRENCY, SERVICE CALL TIME, and ENROLLMENT IS CONFIRMED should also be explained and included in the Terminology List. When naming an attribute always combine it with the entity name to make sure that the combined expression is valid. So, including the entity name in the attribute name is wrong (such as PRODUCT CODE in the entity PRODUCT). Instead of using attribute names like VALID, ACTIVE, ACCEPTED etc for boolean attributes, use terms like IS VALID, IS ACTIVE, and IS ACCEPTED, as that will read as a valid term together with the owning entity name, like this: TICKET IS VALID, PROCESS IS ACTIVE, and ENROLLMENT IS ACCEPTED. The Quality of a Terminology List Spending the time to analyze all these terms (entities and attributes) thoroughly and coming up with good names to be able to generate a usable Terminology List, is a win/win situation as the extra cost automatically adds to the quality of both the data model (and, hence, the final system to be developed) and the Terminology List (and, hence, communication within the business). The PL/SQL of a Terminology List To create the terminology list we need the union of CURRENT_ENTITIES and CURRENT_ATTRIBUTES, like this: create or replace view CURRENT_TERMS as select OBJECT_ID, ENTITY_NAME NAME, 'entity' TERM_TYPE from CURRENT_ENTITIES union select OBJECT_ID, ENTITY_NAME ' ' ATTRIBUTE_NAME, 'attribute' TERM_TYPE from CURRENT_ATTRIBUTES FK_FLAG is NULL order by 2; This will return an alphabetized list of all terms defined as entities or attributes in the SDDM. Notice, the clause on the CURRENT_ATTRIBUTES view (FK_FLAG is NULL). This is necessary because SDDM adds (hidden) attributes to entities to represent foreign keys. These are 7 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
copies of the corresponding primary key attributes of the related entities, so including them in the terminology list would duplicate attribute definitions. Attributes FK_FLAG is NULL selects just the real attributes that you defined on the entities. Getting the description of each term requires an extra join as large text is stored in a separate table called DMRS_LARGE_TEXT, like this: select term.name, text.text from CURRENT_TERMS term, CURRENT_LARGE_TEXT text term.object_id = text.object_id order by term.name; Actually, SDDM has different types of large text, such as 'Comments', 'CommentsInRDBMS', 'Notes', 'Footnotes', etc. So, depending on you decide to store entity and attribute definitions your select, of all terms and their definition may look like this: select term.name, text.text from CURRENT_TERMS term, CURRENT_LARGE_TEXT text term.object_id = text.object_id and text.type = 'Comments' order by term.name; This will result in a simple list, like this: ALLERGY ALLERGY RESOLUTION PROCESS ALLERGY TYPE ALLERGY TYPE IS DIETARY ALLERGY TYPE NAME ALLERGY TYPE TITLE A cause of negative medical reactions to some people. Examples: - Tomato Allergy - Grass Allergy - Dog Allergy - Cat Allergy An explanation of how a person should be helped when the specified ALLERGY TYPE breaks out. A kind of ALLERGY. Indicating that the ALLERGY TYPE has to do with food. A unique and understandable identification of an ALLERGY TYPE. A unique short description of the ALLERGY TYPE. With a little HTML manipulation this list can be presented in a friendlier way on the company's intranet, for everybody's reference, as shown on Illustration 6: 8 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
Illustration 6:Terminology List in HTML Now that you understand the basics of the SDDM Reporting Scheme, only your imagination limits the possibilities of using your business model information in a variety of ways, such as on-line item help in applications, module help, advanced module functionality etc. Let me finish off this paper with a more complicated example based on the same principles. Creating an ER-model description A picture is worth a thousand words, it is often said, and that can certainly be true of logical data models if they are done well. Still, some people are not that visual, and they could benefit from the support of a textual description of a data model. Consider the model of Illustration 7 about Incident: 9 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
The model consists of the following types of elements: A diagram with an assigned comment about the overall purpose of the diagram, such as This diagram is concerned with incidents and their related resolution plans. Entities with assigned definitions. Attributes with assigned definitions. Relationships with cardinality, optionality, and descriptive source and target labels. Illustration 7:Data Model about Incident These are stored in the following SDDM reporting schema tables: DMRS_DIAGRAMS DMRS_DIAGRAM_ELEMENTS DMRS_ENTITIES DMRS_ATTRIBUTES DMRS_RELATIONSHIPS As with the terminology list, depending on the quality of your diagram, entity, attribute, and relationship documentation, you will be able to generate a natural language data model diagram description by extracting data from the SDDM reporting schema tables. The PL/SQL of a Data Model Description Basically, you want to show the diagram name and comment and then describe each entity in the order they appear on the diagram, like this: 10 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
create or replace function desc_diagram( p_diagram_id in DMRS_DIAGRAMS.OBJECT_ID%TYPE, p_format in VARCHAR2 default 'TXT') return clob as v_return clob; v_diagram_name DMRS_DIAGRAMS.DIAGRAM_NAME%TYPE; begin -- ----------------------------------------------- -- Get diagram name -- ----------------------------------------------- select DIAGRAM_NAME into v_diagram_name from CURRENT_DIAGRAMS OBJECT_ID = p_diagram_id; v_return:= 'Description of the diagram: ' v_diagram_name format.lf(p_format); -- Get diagram description v_return:= v_return get_large_text(p_diagram_id, 'Comments', p_format); -- Describe each entity on the diagram v_return:= v_return format.lf(p_format) 'The diagram contains the following entities:' format.lf(p_format); for entities in (select object_id from CURRENT_DIAGRAM_ELEMENTS DIAGRAM_ID = p_diagram_id and TYPE = 'Entity' order by LOCATION_Y, LOCATION_X) loop v_return:= v_return desc_entity(entities.object_id, p_format); end loop; return(v_return); end; That's just about it. The query: select desc_diagram(object_id) from CURRENT_DIAGRAMS; will return a textual description of all diagrams in your model, looking something like Illustration 8: The functions used by desc_diagram (format.lf, get_large_text, and desc_entity) are not too complicated either. Let us take a look: 11 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
The function format.lf is just a simple function for formatting the descriptions for either text or html. The format package can be made much more sophisticated to format beautiful HTML documents. I will leave that to your imagination. For now we will just need a simple package like this one: Illustration 8: Diagram Description create or replace package format as function lf(p_format in VARCHAR2) return varchar2; end format; create or replace package body format as function lf(p_format in VARCHAR2) return varchar2 is begin if p_format = 'TXT' then return(chr(10)); elsif p_format = 'HTML' then return('<br>'); else return(''); end if; end; end format; The function get_large_text is also straight forward: create or replace function get_large_text( p_object_id in DMRS_LARGE_TEXT.OBJECT_ID%TYPE, p_text_type in DMRS_LARGE_TEXT.TYPE%TYPE, p_format in VARCHAR2 default 'TXT') return clob as v_return clob; begin select replace(text, chr(10), format.lf(p_format)) 12 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
format.lf(p_format) into v_return from CURRENT_LARGE_TEXT OBJECT_ID = p_object_id and TYPE = p_text_type; return(v_return); exception when no_data_found then return('[text is missing]' format.lf(p_format)); end; As for the function desc_entity, we want to show the entity comment at the top, then describe each non foreign key attribute (as described previously) of the entity, and finally, describe each relationship connected to the entity. Because relationships can be read in both directions we do not need to filter relationships that have already been described as part of another entity. This is how it can be done: create or replace function desc_entity( p_entity_id in DMRS_ENTITIES.OBJECT_ID%TYPE, p_format in VARCHAR2 default 'TXT') return clob as v_return clob; v_entity_name DMRS_ENTITIES.ENTITY_NAME%TYPE; begin -- Get entity name select ENTITY_NAME into v_entity_name from CURRENT_ENTITIES OBJECT_ID = p_entity_id; v_return:= format.lf(p_format) v_entity_name ':' format.lf(p_format); -- Get entity description v_return:= v_return get_large_text(p_entity_id, 'Comments', p_format); -- Describe each attribute on the entity v_return:= v_return format.lf(p_format) v_entity_name ' is described by the following attributes:' format.lf(p_format); for attributes in (select object_id from CURRENT_ATTRIBUTES CONTAINER_ID = p_entity_id and FK_FLAG is NULL order by SEQUENCE) loop v_return:= v_return desc_attribute(attributes.object_id, p_format); end loop; -- Describe each source relationship connected to the entity -- ----------------------------------------------- v_return:= v_return format.lf(p_format) 13 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
v_entity_name ' has these relationships to other entities:' format.lf(p_format); for relations in (select object_id from CURRENT_RELATIONSHIPS SOURCE_ID = p_entity_id) loop v_return:= v_return desc_source_relation(relations.object_id) format.lf(p_format); end loop; return(v_return); end; Notice the columns SOURCE_ID and TARGET_ID of the Reporting Schema table DMRS_RELATIONSHIPS. They indicate that a relationship should have a direction, from source to target, but that is not the case as cardinality is defined in both relationship ends (through the columns SOURCETO_TARGET_CARDINALITY and TARGETTO_SOURCE_CARDINALITY). This means that both SOURCE_ID and TARGET_ID can hold the one or the many end of a relationship. Basically, these columns should have been named ENTITY_1_ID and ENTITY_2_ID or something similar, as there is no direction behind the SOURCE/TARGET names. This means that the desc_entity function should also have a loop for extracting the target relationships. I have omitted it from this example as it corresponds exactly to the loop for extracting the source relationships. The function desc_attribute is very simple: create or replace function desc_attribute( p_attribute_id in DMRS_ATTRIBUTES.OBJECT_ID%TYPE, p_format in VARCHAR2 default 'TXT') return clob as v_return clob; v_attribute_name DMRS_ATTRIBUTES.ATTRIBUTE_NAME%TYPE; begin -- Get attribute name select ATTRIBUTE_NAME into v_attribute_name from CURRENT_ATTRIBUTES OBJECT_ID = p_attribute_id; v_return:= v_attribute_name ':' format.lf(p_format); -- Get attribute description v_return:= v_return get_large_text(p_attribute_id, 'Comments', p_format); return(v_return); end; The function desc_source_relation is a little tricky. Here, we want to combine the two related entity names with the relationship name, the optionality, and the cardinality to construct a sentence, such as Each INCIDENT must be categorized by one and only one INCIDENT CATEGORY or Each PARTY may be the subject of one or more INCIDENTS (see Illustration 7). Unfortunately, SDDM does not support storage of plural names of entities, so we have to cut a corner to make this work. Either, you will have to add an S to the entity names to create the plural version of them (this will work fine most of the time), or you could use another column, like PREFFERED_ABBREVIATION, to store the plural name of each entity. The second choice will also allow you to engineer your logical model with singular entity names into a relational model 14 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
with plural table names (but that problem is for another paper). This is how we juggle SDDM to return the description of a source relationship: create or replace function desc_source_relation( p_relationship_id in DMRS_RELATIONSHIPS.OBJECT_ID %TYPE) return varchar2 as v_return varchar2(1000); v_entity_1 DMRS_RELATIONSHIPS.SOURCE_ENTITY_NAME%TYPE; v_entity_2 DMRS_RELATIONSHIPS.TARGET_ENTITY_NAME%TYPE; v_cardinality DMRS_RELATIONSHIPS.SOURCETO_TARGET_CARDINALITY%TYPE; v_optionality DMRS_RELATIONSHIPS.SOURCE_OPTIONAL%TYPE; v_relationship DMRS_RELATIONSHIPS.SOURCE_LABEL%TYPE; begin -- Get relationship data. select SOURCE_ENTITY_NAME, TARGET_ENTITY_NAME, SOURCETO_TARGET_CARDINALITY, SOURCE_OPTIONAL, SOURCE_LABEL into v_entity_1, v_entity_2, v_cardinality, v_optionality, v_relationship from CURRENT_RELATIONSHIPS OBJECT_ID = p_relationship_id; -- Compose relationship statement. v_return:= 'Each ' v_entity_1 case when v_optionality = 'Y' then ' may be ' when v_optionality = 'N' then ' must be ' end v_relationship case when v_cardinality = '*' then ' one or more ' v_entity_2 'S' when v_cardinality = '1' then ' one and only one ' v_entity_2 end; return(v_return); end; The final result can be seen on the previous Illustration 8. Conclusion I hope that this have given you an impression of how you can significantly enhance the value of your investment in data modeling with SDDM by reusing the modeling information in countless situations throughout your business. Besides the kind of reporting described in this paper, you can also integrate the information in user help for application systems, or even have your application systems' behavior follow the definitions in your data models and process models! This kind of Architectural Software Development, the IT systems that end users interact with are linked directly to a business model, is also covered in others of my papers. Take a look on my 15 Phone: +45 2627 9991 Email: Marc@SimplifySys.com
web site: www.simplifysys.com you can find papers about how to use SDDM content to manage project estimates and progress, or how to build an APEX menu system and breadcrumb based on the SDDM Reporting Schema. 16 Phone: +45 2627 9991 Email: Marc@SimplifySys.com