Fachbereich Informatik Integrierte Schaltungen und Systeme Prof. Dr.-Ing. Sorin Huss. Diplomarbeit. Integration of Domain-Specific Languages

Transcription

1 Fachbereich Informatik Integrierte Schaltungen und Systeme Prof. Dr.-Ing. Sorin Huss Diplomarbeit Integration of Domain-Specific Languages Martin Geisse Matr.-Nr.: Betreuer: Dipl.-Inform. Felix Madlener Abgabe:

2 Zusicherungen zur Diplomarbeit gemäß 19 Abs. 6 der Diplomprüfungsordung Hiermit versichere ich, die vorliegende Diplomarbeit ohne Hilfe Dritter nur mit den angegebenen Quellen und Hilfsmitteln angefertigt zu haben. Alle Stellen, die aus Quellen entnommen wurden, sind als solche kenntlich gemacht worden. Diese Arbeit hat in gleicher oder ähnlicher Form noch keiner Prüfungsbehörde vorgelegen. Gießen, den Martin Geisse 2

3 Abstract Domain-specific programming languages (DSLs) are an increasingly common approach in software engineering. This diploma thesis presents Modelflow, a generic framework for the integration of different DSLs, and illustrates that DSLs can be applied to model digital circuits as well. 3

4 Table of Contents 1. Introduction Problems with Domain-Specific Languages Modelflow Basics Additional Features Compatibility with other IDEs Implementation Issues Model Cache Model Data Copy Policy Implementing Model Types Implementing Model Compilers Implementing Other Tools Support Libraries Example The Problem Creating the Model Type Writing a Compiler Defining a Function Table Another ROM-producing compiler Another ROM-consuming compiler Expanding to Higher Levels Advanced Topics Passing Down Compiler Parameters Abstract Compilers Hardware Simulation Interpreted DSLs Integration of existing DSLs Conclusion Future Work Core System Meta-Models Miscellaneous A. Source Code Organization B. Installation C. User Interface D. Example Source Code Bibliography

5 1. Introduction Complexity is a common problem in engineering. While simple things are simple to design (though they may still be hard to build), complex things soon reach the limits of the abilities of an engineer. This has several negative effects, among them: errors in the final design missing features longer time-to-market higher production cost designs which cannot be adapted to changing requirements One tool to attack complexity is abstraction, which is the key to split a problem into sub-problems and solve one sub-problem at a time. Wikipedia defines abstraction as the process of generalization by reducing the information content of a concept or an observable phenomenon, typically in order to retain only information which is relevant for a particular purpose [1]. In engineering, the details of some sub-problems are reduced to a minimal amount of information to solve the remaining subproblems, ultimately solving all sub-problems one by one. For example, the task to write a program which uses linked lists of data nodes can be split into two sub-problems. One sub-problem is to write an implementation of linked lists, while abstracting from the details of how these lists are used. The second sub-problem is to write the remaining program, while abstracting from the details of how linked lists work. This diploma thesis is concerned with the engineering of both software and digital circuits. It may or may not be possible to transfer the results to other engineering disciplines. The modeling tools used in these two areas are to some extent very similar, and incorporate a great deal of abstraction in themselves. Both use some kind of language to express designs -- a pattern of notation with strictly defined semantics. The abstraction lies in the fact that these languages do not directly resemble the actual building pieces of either software (machine instructions) or digital hardware (logic gates). Instead, they express the same designs in higher-level terms which can be mapped (compiled) to a design expressed in building pieces available in reality. Figure 1, Programming Languages vs. Implementation Results shows the difference between a higher-level language (in this case, VHDL) and the corresponding implementation result. Most software languages (such as C) identify a few usual patterns of control flow and data storage, and let the engineer express these patterns with at most a few keywords. Thus, they hide the details of how these patterns are implemented. Likewise, most digital hardware languages (such as VHDL) identify boolean expressions as a common pattern to specify logical functions. Thus, they hide the details of building actual logic gates and their connections, and let the user think in terms of boolean terms. Figure 1. Programming Languages vs. Implementation Results Both C and VHDL have in common that they are general-purpose languages, meaning that the languages themselves are designed without any specific class of problems in mind (other than being spe- 5

6 cific to software engineering and digital hardware engineering, respectively). There is another class of languages called domain-specific languages (DSLs), which are based on the alternative principle that the languages are built for a specific class of problems and are applied only there. An example for this is the language for makefiles, shown in Figure 2, make: A Domain-Specific Language. This has the obvious consequence that for each class of problems, another domain-specific language is needed. It also means that a DSL is optimized to solve problems that lie within its focus, at the cost of being unable to solve problems outside its focus. Using a DSL or a set of DSLs designed for the problems at hand in some specific task should therefore yield better results than solving the whole task with a general-purpose language. Experience with existing DSLs, such as the parser-generation tool YACC [2], has confirmed this expectation (see e.g. [3]). The advantages of DSLs have led to the suggestion of a new programming paradigm called language oriented programming by Ward [6] and Dmitriev [7]. The same concept is described by Steele in [8]. Figure 2. make: A Domain-Specific Language TOOLDIR =../eco2/eos32-0.9/build/bin SOURCES = start.s $(REGULAR_SOURCES) end.s REGULAR_SOURCES = $(UTIL_SOURCES) $(LOWLEVEL_SOURCES) \ $(GAME_SOURCES) UTIL_SOURCES = util/cpu.s util/kernel.s util/memory.c \ util/random.c LOWLEVEL_SOURCES = lowlevel/timer.c lowlevel/sound.c \ lowlevel/display.c lowlevel/keyboard.c GAME_SOURCES = game/main.c game/draw.c game/engine.c \ game/gamestate.c game/shapes.c game/soundchain.c all: kernel.bin kernel.bin: $(SOURCES) $(TOOLDIR)/lcc -I. -A -Wo-kernel -Wl-m \ -Wlkernel.map -Wl-h -o kernel.bin $(SOURCES) romimage.exo: romimage.bin../bin2exo/bin2exo 0 romimage.bin romimage.exo romimage.bin: $(SOURCES) $(TOOLDIR)/lcc -I. -A -Wo-kernel -Wl-m \ -Wlkernel.map -Wl-h -Wl-rc -Wl0xE \ -Wl-rd -Wl0xC o romimage.bin $(SOURCES) clean: rm -f *~ disk.fsys kernel.bin kernel.map \ romimage.bin romimage.exo A complete discussion of DSLs is beyond the scope of this diploma thesis. A more detailed explanation as well as a bibliography list can be found in [4]. That text also makes a useful distinction between different implementation strategies for DSLs, which yield different levels of freedom in design of the final language. For our purpose, we focus on what they call the classical approach, that is, compilation and/or interpretation. By that we exclude the following domain-specific notations from the term DSL in favor of cleanly designed, uncompromised DSLs: domain-specific libraries, possibly using self-modifying language mechanisms such as the definition of new operator symbols or Lisp-style macro definitions macro preprocessors in general 6

7 extensible compilers for a specific language A description of the negative impact of such DSL implementation strategies can be found in [5]. In a specific scenario, self-extension mechanisms in existing languages can be used in addition to classical DSLs though, and may thus be kept in the engineer's toolbox as yet another possibly helpful tool. An important point that must be regarded throughout the discussion of DSLs is that the term language is used here in the sense of a notation with strictly defined semantics. It does not imply a textual notation, although textual notations are common for programming languages. The sets of valid schematic drawings, state diagrams, or truth tables are also languages in the sense used here. The key point of this definition of the word language is that strictly defined semantics allow us to interpret these languages by algorithms, and thus, computer programs. As a concrete example, a language could be based on the following table structure (which resembles instructions to add HTML markup to a Java source code file): Table 1. A Non-Textual Programming Language (using tables instead of text) Rule Name Pattern Markup Italicalize Identifiers [_a-za-z][_a-za-z0-9]* <i> Boldify class Keyword class <b> Boldify Curly Braces ("{" "}") <b> The remainder is structured as follows: Section 2 discusses the main problems with using DSLs, and draws conclusions how they can be solved. Section 3 then introduces the main subject of this diploma thesis, the Modelflow system, which serves as a foundation for implementation, usage, and integration of different DSLs and aims to implement these solutions. Section 4 describes the programming interface and implementation details of the core system. Section 5 presents an example use case of Modelflow, and is meant as a tutorial for using it. Section 6 gives solutions for some problems that occur in practice when using Modelflow, and shows that the core system is powerful enough to solve those problems. Section 7 summarizes what has been said so far, and section 8 gives an overview of missing features and other future work that is imaginable to improve Modelflow. 7

8 2. Problems with Domain-Specific Languages Most problems with DSLs stem from the fact that for each class of problems, a separate DSL is required. The first, obvious disadvantage is that much work must flow into the creation of the languages themselves as well as into the corresponding tools. An important observation here is that a DSL, like a library in some programming language, constitutes a unit of re-use. Building a DSL to solve a single problem with it may be viable in special cases; in most cases however it is as much a waste of effort as is building a well-encapsulated programming library to solve a single problem. Usually, if a library is well-encapsulated, then it is also built to solve a whole class of problems, whether by intention or by chance, and the same holds for DSLs [4]. It follows immediately that the DSL (or library) can be re-used for any other problem from that class. By doing so, the cost of creating the DSL and its tools is amortized over several problems. Re-use is an important key factor to make DSLs a viable solution. However, the task can be simplified by other means. Like any programming task, the creation of DSL tools can be considered a target for various software engineering means. Among others, this includes the application of DSLs themselves to create other DSLs. For example, YACC is not only an example for a popular DSL, it is also a tool that can be used to implement DSLs (more specifically, the parser of a DSL compiler or interpreter). Other tools include Lex for lexical analysis, make for control flow in the build process, source code generation libraries, binary code generator generators, graph algorithm libraries (e.g. for register allocation), and so on. Some of these tools define and apply their own DSLs, while others are programming libraries. Taken together, they can simplify the creation of a new DSL by a great deal. Figure 3. Traditional Implementation of a DSL The traditional implementation of a DSL (Figure 3, Traditional Implementation of a DSL ) yields stand-alone tool programs such as a compiler or an interpreter for the language. A compiler, for example, would accept command-line arguments that specify its input files as well as additional parameters for code generation. The various tools for different DSLs, general-purpose languages, and other miscellaneous tools are usually hardly integrated: they have different expectations about file names and directories their documentation define commonly used terms differently it is usually impossible to run two interpreters for different DSLs at the same time and have them cooperate, or at least not without immense effort they generate code in different, incompatible output languages 8

9 they use different coding conventions in their output code, which may just look like bad style or actually cause problems some incorporate rules that determine if and when existing output files are re-generated; others always overwrite; still others work in an even more complicated way such as partial overwriting they usually allow to control every little uninteresting detail of code-generation through parameters, because some other tool might require so The remainder of this work will focus on compiled (not interpreted) languages, since multiple interpreted languages would not cooperate well without further effort. Later, in Section 6.4, Interpreted DSLs, the idea of interpretation is picked up again. For now, this leaves code generation as the means to implement DSLs. Even then, one must carefully select those languages and tools that use compatible output languages and coding style. Having done so, one usually needs a lot of glue code that makes the various generated code parts from different DSLs fit together. If this had to be done only once, it might be acceptable. But things are worse: any project, including those that use DSLs, is not written at once. The code must be developed, improved, de-bugged, and extended with new features. Projects also evolve, changing the initial assumptions on which they are based. Therefore, the engineering tools including the DSL compilers must be run every time their input source code has changed. Automation tools exist to detect changes in the input data and re-run the corresponding tools. This includes various integrated development environments (IDEs) as well as the program make. However, while these tools do help, there is still a lot of work left to the user. Tools often do not play well with an IDE or make when they were not written with that in mind. That is, for example, they might employ their own mechanism to determine when recompilation should occur, which then interferes with the one employed by make. It is also often left to the user to decide about file names of intermediate files (files used to pass data from a producing tool program to a consuming one). There is no gain in this freedom of decision -- either the tools agree about file names and things work, or they don't. Moreover, since there is not only freedom but also obligation to choose, beginners have a harder time since they are overloaded with unnecessary decisions. Finally, personal preference of different engineers working on the same project may collide. The icing on the cake is that the details of configuring the tools just to work together vary tremendously from tool to tool. The bottom line is that there are many obstacles in implementing and using DSLs, and their expected advantage is impaired by bad integration. The difficulties in the creation of new DSLs have led to the concept of language workbenches, a term reportedly coined and certainly described well by Fowler [5], and realized by Dmitriev in his Meta Programming System [7]. In a similar vein, this diploma thesis specifically attacks the problem of integration. The ultimate goal is to build a language workbench that solves both problems, and thus opens the road to the envisioned language-oriented approach. The anticipated solution to the DSL integration problems is the definition of strict rules that state how tools are integrated, and demand that the tools follow those rules without exception. For that, the Modelflow system is presented which acts as a foundation for the development and integration of DSL tools. It attacks the integration problems by defining a framework into which tools must be plugged as modules instead of being written as standalone programs by abolishing source code files as the user-visible unit of code management. Data is edited and passed to tools without knowledge about the actual means of data storage by either the user or any tool module by defining a strict naming scheme for source code units which uses a single identifier both to determine its storage place and to refer to the unit from within other code by defining that DSLs are compiled, not interpreted. That is, the result of feeding a DSL code unit to some tool must be comprised of output code units and possibly error messages, and nothing else. Especially, it must not cause any side-effects. 9

10 by abolishing intermediate files. Data is passed between tools by the core system without knowledge of the actual means of data transmission by either the user or any tool module. by defining a strict naming scheme for generated code units which uses a single identifier to determine how the code is generated (i.e. by which tool and from which input units, which can themselves be source code units or generated code units), to determine its storage place, and to refer to the unit from within other code by allowing to compile the same input code unit with different compiler modules to produce different results. For example, some scripting language may be compiled either to produce executable code from it, or to produce documentation from the comments in the source code. by having the core system decide automatically and correctly when some unit of generated code must be re-built because its inputs have been modified by allowing to store code units as structural code, not textual code, to simplify the creation of structural editors and compilers, to avoid even the existence of syntax errors in code, and to prevent disagreement about coding style. The decision whether structural code or textual code is used for some DSL is left to its inventor. Modelflow thus implements the strict rules by which the various editors, compilers, and other tools store and communicate code. It is also a language workbench in the sense that it facilitates the creation of new languages, but the corresponding tools have not yet received the same level of attention as integration. 10

11 3. Modelflow As described in the previous section, Modelflow is a core system for the development and integration of DSLs and DSL tools. The core is written in Java, on top of the Eclipse IDE 1, and must be extended with modules written as Eclipse plug-ins in Java. These modules can extend the core system with new DSLs, DSL compilers, import and export tools, language-specialized editors, and other tools (see Figure 4, Modelflow Architecture ). Although Eclipse already provides a basis for the integration of such tools, Modelflow adds the set of strict rules described in the previous section, as well as code to implement those rules, to alleviate the burden both on the end-user and the tool author. Figure 4. Modelflow Architecture. Extensions build upon both Modelflow and Eclipse Basics The concepts used in the base system are centered around the model, which is the smallest unit of code managed automatically. A model can be defined, meaning that it is entered manually by the user, or it can be generated, meaning that it is produced by a compiler module from other models. The defined models are organized in a hierarchical package structure, like files are organized in folders (see Figure 5, Defined Models, Packages, and Model Types ). Each model is referred to by a unique and unambiguous model identifier. That is, each model has exactly one model identifier, and each model identifier refers to either exactly one model, or to none at all. In the latter case, it can also be said that the identifier refers to an unknown model or non-existing model. Each model carries its main contents, called model data, which is the actual domain-specific information expressed by the model. As is shown in the figure, each model also has a model type describing the domain to which the model data applies and the structure and format in which the data is held. 1 The Eclipse Foundation. 11

12 Figure 5. Defined Models, Packages, and Model Types A model identifier carries in itself the information whether the referred model is defined or generated. The model identifiers for defined models (called defined model identifiers) comprise the name of the model in its sub-package as well as the hierarchical names of the packages in which it is stored, analogous to an absolute file name in a file system. The model identifiers for generated models (called generated model identifiers) comprise the name of the compiler module which is used to generate the model, as well as the identifier of the main input model to that compiler (which can be again either a defined or generated model identifier). Note that the identifiers of secondary models needed for compilation, i.e. models that are referred to by the main input model or other secondary models, are not encoded into the model identifier. This information is stored solely in the model data of the referring model. Figure 6, Defined Models, Compilers, Generated Models, and Model Types shows how generated models are produced by running a compiler on a main input model. Additional models may be consulted to produce the result (called dependencies, since the result depends on them). Either input models may be defined or generated. Figure 6. Defined Models, Compilers, Generated Models, and Model Types The textual syntax of a defined model identifier is the list of package names, from top-level to deepest sub-package and finally the local model name, separated by dots. The textual syntax of a generated model identifier is that of the model identifier of its main input model, followed by character, followed by the name of the compiler module to run on this input. For example, to name the model that is stored in the sub-package b of top-level package a and named c, the model identifier a.b.c 12

13 is used. It can be determined by the missing of character that this identifier is a defined model identifier. To name the model that results from running the compiler X on that model, and then running the compiler Y on the result of X, the model identifier a.b.c@x@y is used. By the existence of character it can be concluded that this is a generated model identifier, and the (immediate) input model and compiler name can be obtained by splitting a generated model identifier around the character it contains, in this case yielding the input model a.b.c@x (which is itself a generated model) and compiler module Y. Each model has a model type, no matter whether the model is defined or generated. Like the model data, the model type is defined by the user for defined models, and decided by a compiler for generated models. The model type defines for a model: The data structures (Java classes) used for model data Creation code for default instances Validator code to check model data for correctness and consistency The file format to store model data on disk Loader and saver code to convert between the two An editor for model data using Eclipse and SWT Additional actions which can be performed on models of that type It is not encoded into a model type by which compilers the model can be consumed; this information is specific to the compilers, not the types. Model types are also organized in a hierarchical folder structure, called the Model Type Registry, but solely to allow choosing a type by a hierarchical user interface (see Figure 7, The Model Type Registry ). In most places where model types are important, the absolute names of types are therefore used. The set of available model types can only be modified by restarting Eclipse with a different set of plug-ins providing them. Figure 7. The Model Type Registry Compiler modules, like types, are organized into a pseudo-hierarchy that exists solely for user interface purposes (see Figure 8, The Model Compiler Registry ). A compiler is referred to by the model identifiers of generated models, by the fact that such a model identifier contains the (absolute) name of that compiler module. A central concept of Modelflow now is that given the set of model types and compiler modules (both of which are built into the running system as Eclipse plug-ins), and a set of defined models supplied by the user, any model -- defined or generated -- can be requested by specifying its model identifier, without assuming a specific state of the system. Specifically, a generated model identifier refers to the result of the corresponding compiler invocation regardless of whether that invocation has already happened. The compiler is invoked automatically as soon as its 13

14 result is first needed, and invoked again when its input models have changed. This is in contrast to the traditional concept of intermediate files, where the name of an intermediate file only refers to that file after it has been generated, and upon source modification refers to outdated results until the compiler is invoked again. Note The full names of models, model types, and compilers must be used almost everywhere in the system. Especially, generated model identifiers contain the full name of the respective input model and compiler, including all model packages and compiler packages. However, figures in this text will sometimes abbreviate those identifiers when unambiguous to keep them readable. Figure 8. The Model Compiler Registry 3.2. Additional Features The basic concepts explained so far are now augmented to solve some practical problems. The first such problem is that models may be loaded and re-loaded excessively by compilers, depending on the way the compilers are used. Also, the user interface often refers to meta-information of a model (such as the model type) for displaying purposes. Taken together, these repeated accesses would put enormous stress on the storage mechanism that keeps defined and generated models -- currently, the file system. (Remember that Modelflow is not about abandoning the file system, but about shielding the engineer from its details). To solve these problems, the meta-information and, as soon as requested, the model data itself, are kept in a cache managed by the base system. This cache also presents the programming interface to request models. Figure 9, The Model Cache shows the model cache. Both defined and generated models can be cached. For implementation purposes, it is often useful to speak of two different caches, one for defined models and one for generated models. In this figure, all models but Helloworld.Util are present in the cache. 14

15 Figure 9. The Model Cache Another practical problem is that the model data, while usually syntactically correct through the means of structural editors, often contains semantic errors. This may be a model reference which refers to an unknown model, or to a model of wrong type. It may also be inconsistent information between two models. An engineer would want to get immediate feedback through the user interface about such problems, and not wait for compilation errors to result from them at some point in the future. Also, if several compilers can consume some model, it would be impractical to implement checks for such semantic errors in each of them. As a solution, each model type defines a validator, a piece of code which can check a model of that type and result either in success, or failure with a set of validation error messages. The results of the validator are used both for visual feedback and to inform compiler modules (upon request of the model) whether it is semantically valid (i.e. has passed some basic checks, which the compiler can then omit). Figure 10, Validating Models illustrates validation. Both defined and generated models are validated. The corresponding validator code is supplied by the respective model type of each model. Compilation fails if one of the input models is invalid. Even if all models are valid for themselves, the combined information from all input models taken together may be inconsistent, and compilation can still fail. What is not shown in this figure is how the result of the successful compilation is validated. 15

16 Figure 10. Validating Models. Invalid models are not even considered for compilation. A third problem that had to be solved in practice is that by invocation of a compiler on some main input model (and indirectly, on referenced models), only a single output model is generated. Depending on the situation, several output models are needed. For example, when generating documentation for some models automatically, the output documentation may need to comprise a model for an index HTML file, and others for section HTML files and yet others for images. A possible though impractical solution would be to define a new model type (called, for example, a web archive ) that is able to hold all that information. This approach would, however, waste the work that has already been accumulated into the model types for HTML pages and for images (assuming for the sake of this example that such model types already exist). Therefore, a mechanism has been added that allows generated models to be augmented by any number of parts in addition to its base model. Each part consists of a model type and model data, and can re-use existing model types. Figure 11, Generated Model Parts shows a compiler that produces an output model with parts (which is just Modelflow's version of several output models, as explained above). Each part has its own model identifier and model type. Also shown in that figure is that parts, like any other model, can be used as the input for a compiler. Figure 11. Generated Model Parts 16

17 Within a generated model, each part is identified by a part name. Parts are externally visible models, and the model identifier of a part consists of the model identifier of its base model, followed by a colon and then the part name. Revisiting the example described above, to identify the model that results from compiling the defined model a.b.c by compiler module X, taking part P, and compiling it with compiler module Z, the model identifier a.b.c@x:p@z has to be used. Omission of the colon and the part identifier selects the generated base model. Another problem was that while a set of models can easily be filtered by allowing only models of one specific type to pass, sometimes any of a set of types is acceptable. As an example, a model compiler may allow any one of a set of types as its inputs. Likewise, a reference contained within a model that refers to another model might be allowed to refer to any instance of any of a set of types. This could easily be allowed by specifying a set of types instead of a single type for filtering in each of those cases. However, the relevant type sets are often the same, and contain types with related meaning. For this, model supertypes were invented. Supertypes, like types, are defined in Modelflow extensions. Each supertype describes a named set of types. The term type filter means type or supertype, since both types and supertypes can be used as a filter to allow or reject types. In the compiler example mentioned above, a compiler now declares a type filter to specify what models it accepts as inputs Compatibility with other IDEs Since Modelflow introduces a new way to handle old problems, some older solutions to those problems become redundant, and do not work well together with Modelflow. In general, using two IDEs at the same time, in the same project, is likely to cause problems. Using them together with tools like make is similar. The reason is that every single of these tools was written to solve (partially) the same problems, to solve them well, and to solve them alone. Any effort to make such programs work together for some project is likely better spent in improving the best-fitting one for that project with the features missing in it. Note that some IDEs do play well with make since they are built specifically around it and are more of a front-end for it than a standalone IDE. The above statement was made with respect to IDEs that aren't. Modelflow has a similar aversion against other IDEs, and even more so because it treats some fundamental issues in a very different way: A project does not contain files, it contains defined models. Though the defined models can be mapped to files (and they are in the current implementation), this mapping occurs automatically without requiring user intervention or additional information. A project does not build (intermediate or resulting) files, it builds generated models. As described above, the set of model types, compilers, and defined models are the background against which a generated model identifier contains all information needed to build its model. Compilation parameters are traditionally handled separate from the actual source code and in a completely different way. In Modelflow, these settings are stored in models just like other source data is. Figure 12, Project Files in Traditional IDEs shows the data flow during compilation in a traditional IDE, and Figure 13, Project Models in Modelflow shows the same for Modelflow. The similarity is obvious, and illustrates that traditional tools could have treated project settings as just another type of regular source data, just like Modelflow does, if only the tool authors decided to do so. The obvious advantage of such a decision is that all input data flowing into compilation is treated in a uniform way. By this, the user avoids having to learn new compilation management mechanisms over and over again for different tools. 17

18 Figure 12. Project Files in Traditional IDEs Figure 13. Project Models in Modelflow 18

19 4. Implementation Issues Having explained the concepts upon which Modelflow is built, this section deals with how the concepts are realized in code, and how extensions to the base system can be created. As said before, all code is written in Java and builds on the framework provided by the Eclipse IDE. Extensions to Modelflow are plugged into the base system through the extension mechanism already present in Eclipse. Programming experience with Eclipse and its plugin architecture are helpful for writing Modelflow extensions. Note This version of Modelflow was developed for version of Eclipse Model Cache Modelflow maintains caches for both defined and generated models. The caches keep track of all existing models including their metadata, and can cache the model data in memory to increase performance. The cache also maps models to files. All interaction with model storage happens through the cache. For defined models, the set of cache entries corresponds exactly to the set of defined models. That is, every defined model has an associated cache entry; the only border case is when Eclipse's resource system has not recognized recent changes to the model files yet. Upon refreshing the resource manager, the changes are recognized and the cache is updated. The cache entry does not hold model data from the beginning; only when actual model data is first requested is the file read, the contents turned into instances of the model data classes (as implemented by the corresponding model type,) and those objects stored in the cache. In theory, data can be evicted at later times to free memory, but no guiding policy has been implemented yet. Changes in the cached model data are immediately saved to the file, i.e. the cache uses a write-through policy. The base system also detects modification of the backing file (as much as supported by Eclipse) and in that case evicts the cached model data without saving it, assuming that the modified file contains recent external modifications made by the user. For generated models, a cache entry is created when it is first requested. This is necessary since the set of all possible generated models is potentially infinite. Just as for defined models, the initial cache entry does not hold any model data. Data is created upon the first request by running the corresponding compiler module, and the result is both kept in the cache and stored to the model file. Whenever compilation dependencies are modified, the model is marked as outdated and its data evicted. The backing file in the case of generated models is intended to store the compilation results across different invocations of Eclipse/Modelflow. A developer would not want to re-build a whole project just because he or she closed and re-opened Eclipse. However, no such mechanism is implemented yet, although storing generated models to files is a key component in doing so. What is missing is similar storage for the state of a generated model, and in particular whether it is outdated (data must be produced by re-running the compiler) or not (data can be read from the file). Modelflow provides two classes which are useful for looking up models when needed: The CacheEntryRequest class looks up a cache entry and can be used whenever meta-data of a model is needed. This is typical for user interface code such as the model browser. The ModelLookup class goes further and also requests model data (loading or generating it when necessary). Both classes provide some support for error handling. The cache also supports notification of external entities of any events through a listener architecture Model Data Copy Policy Model data is not only stored and processed in Modelflow, it is also changed in editors and other tools. As always when modifications are allowed, this requires a certain policy that guides the use, copying, 19

20 sharing (aliasing), and modification of such data to avoid propagating changes in model data to some entity which does not expect so. To be more precise, one example for a problem due to copy policy violation is: Entity A obtains a data object by some means It passes the object to entity B as the input data for some operation B stores (part of) the input object for later reference A modifies the object for other purposes B uses the stored object as the input for another operation, assuming that it still contains the original data the latter operation produces incorrect or inconsistent output data This chain of events is illustrated in Figure 14, Errors due to Aliasing. In the end, entity B is not prepared to handle a modified data object. Since that entity is (unintelligent) code, it would simply perform an invalid operation and/or produce inconsistent result data. Figure 14. Errors due to Aliasing To avoid this chain of events, one of the links must be broken. This yields the following possible solutions: A could never reveal its internally used objects, and instead give a defensive copy of the object to B B never stores its input objects, but instead copies them defensively and only stores the copy A never modifies the copy. Instead, whenever modifications are intended, a copy is made. This leads to a programming style known from functional programming languages B is aware of possible modifications to the object, and can handle them There are other cases in which a similar problem occurs. In general, problems can occur whenever more than one entity has access to an object. Thus, a policy must be established for every shared object scenario in an object-oriented program. This section does so for the case of the model data structures Modifications by Editors An important observation to determine a useful policy is the sequence of actions an editor performs on the model data, since this is by far the most frequent source of modification to the model data. This 20

21 sequence could, for example, be: open, modify, save, modify again, save again. What is important is that the sequence could be stopped at any point, specifically between the modify (again) and save (again) actions. If that happens, and the editor is closed before saving the changes, then the whole modification shall be lost and have no effect on the stored model data. This implies two important rules for the policy: An editor must not modify its initial input data objects before the changes are saved, and that may never happen An editor must not modify the objects returned when saving before the subsequent changes are saved, and that may never happen What is not forbidden so far is to modify the original data objects once the modifications are requested to be saved through the user interface. However, saving a bunch of modifications to already-existing objects is complicated, if not plain impossible. For example, the original model data could contain an array of integer values, and the modified version from the editor could contain an array of different size. Java would not allow to re-size (and thus re-use) the original array to store the modifications. It is therefore sensible to add the following item to the policy, which concludes the two items mentioned above: An editor may never modify its initial input data objects, and it may never modify the objects produced during the save operation. The question whether editors may save their input objects for later reference (without modifying them) has not been answered yet Modifications by Other Tools Having examined editors a bit, we now turn to other possible data-changing entities. The first ones are compilers. A central rule in Modelflow is that compilers never modify their input data. They just use it to produce output data. Therefore, it can never happen that data modifications from a compiler propagate to other users of its inputs. Since a compiler produces its output data upon completion, it cannot propagate later modifications through its output data either. On the other hand, a compiler may always safely share its input data as part of the output data, regardless of the policy: Whenever the input data has changed, the output model is marked as outdated and the compiler is re-run. Figure 15, Indifference of Compilers About the Policy illustrates that the behavior of compilers does not give useful hints to decide about the copy policy. No matter whether compilers copy or share their input data objects, recompilation always restores a consistent state. 21

22 Figure 15. Indifference of Compilers About the Policy Importers and other data-generating tools do not cause any problems: They generate new models, or at most overwrite existing models. However, they do not modify existing data structures, and thus cannot cause unforeseen side effects in any entity sharing those structures. Also, importers usually do not take any input models. When they do, they usually do not share data structures of those models. Thus, importers (while still having to abide to the policy) are unimportant for deciding about a sensible policy. Refactoring tools are a different story. An example for such a tool could be a model re-namer: It would take a defined model and a new name, rename the model (changing its defined model identifier,) and then search through all defined models for the old identifier and replace it with the new one (including uses as the input of a generated model identifier, of course). This kind of operation would benefit from being able to change the model identifiers in place, that is, by modifying existing model data structures Analysis The observations made so far still allow two different policies (shown in Figure 16, In-Place vs. Out- Of-Place Modification ): 1. Model data structures may be modified by entities like the refactoring tool (editors may still not modify data objects in-place since the user might decide to close the editor without saving). Editors and other tools must not keep references to old input data objects unless they are ready to handle later modifications by other entities. Since the changes are applied to the original data objects without moving or replacing objects, we call this a policy of in-place modification. 2. Model data structures are not modified by any entities and can thus be shared freely. Entities like editors or the refactoring tool mentioned above have to create a modified copy and then request 22

23 that this copy is treated as the new up-to-date model data. Since the changes are applied elsewhere than the original data objects, we call this a policy of out-of-place modification. Figure 16. In-Place vs. Out-Of-Place Modification One argument to choose one of those two policies is that in-place modification is more efficient for certain refactoring tools (not all tools can exploit that advantage, for the same reasons editors cannot). On the other hand, such tools are not in use most of the time, so improvements to their efficiency have a lesser overall effect. Also, editors already make out-of-place modifications, so making refactoring tools behave similarly would yield a simpler and more consistent API. Focusing on that last point -- simplicity of the API -- some other useful observations can be made: In-place modifications need not only be made, they must also be signaled to the base system to indicate that each generated model that uses the modified model as one of its inputs must be rebuilt. This is prone to be forgotten by programmers, without immediately visible effects. With out-ofplace modification, application of the changes and signaling are done in a single operation, making it impossible to forget the signaling. With in-place modification, editors must always make a deep copy of their input data objects and never store the original beyond editor initialization, to avoid being affected by later external modification. Again, forgetting to do so is easy and does not produce immediately visible effects. With out-of-place modification, editors may keep their input data since it is known to be immutable. With in-place modification, refactoring tools that allow to copy part of a model's data into another model must take care to make a deep copy of the whole data sub-structure. Any shared sub-object again leads to interference when the corresponding part is modified in one of the models. With in-place modification, the API becomes inconsistent. First, refactoring tools may modify the data in-place, but editors must not. Second, compilers may re-use data objects despite possible later modifications (since they are restarted as soon as modifications occur,) but editors must not. With in-place modification, it is impossible to use the model data of a generated model independently from that model. Since a generated model could share data objects with its sources, any modification in the source model data could carry over to the generated model data due to the assumption that it is re-generated whenever those data objects change. However, this also means that the model data objects of the generated model before the modification become invalid as soon as the sources are changed, and the re-generated model data must be used instead. This can only be avoided by using a copy of the generated model data. On the other hand, out-of-place modification requires an unintuitive programming style in the case of the renaming tool. Conceptually, the renaming tool just replaces the old model identifier with the new one. However, all enclosing data structures have to be reconstructed one after another to contain the changes, from the leaf node containing the model identifier upwards to the root node of the model data. While this style is unintuitive, errors produce immediate effects and are therefore easy to find. It should be noted that some of the negative effects of in-place modification can be mitigated by defining that compilers must copy their input data and must not share any data objects with the output model. However, this would complicate the creation of compiler modules and introduce yet another possible source of hard-to-find errors. It would also have a negative impact on the performance of those compilers that could share objects. 23

24 Result A choice had to be made at this point: Either Modelflow requires an inefficient and unintuitive programming style as much as refactoring tools are affected, or it uses an inconsistent, complex API that allows hard-to-find errors. The final key argument is that while using immutable data objects is unintuitive for many programmers, there is little chance to make errors that are not immediately visible. Given that Modelflow was invented in the first place to make work easier for developers, and that it is run on high-end development machines, this choice can be answered in favor of simplicity, and we can finally establish the model data copy policy: Model data structures are immutable after creation. Whenever possible, their corresponding Java classes should disallow any later modifications, to have the policy enforced by Java's type system. Any entity can freely store references to data objects without having to deal with modifications. This includes compilers and editors, but also other tools. When model data is changed in an editor, those modifications must happen to internal data objects of the editor, never to the original input objects. The changes are not visible until the save operation of the editor is used to yield the modified model data, which is then used instead of the original data, while the original input objects are still unmodified. Also, later changes through the editor must not affect the saved version -- they must again only affect editor-internal objects Implementing Model Types Implementing a new model type is similar to implementing basic tools for a new file type. However, while most operating systems do not demand that any tools are present for a new file type, Modelflow demands that for any model type, the following meta-data and operations are implemented: Type identifier, used to refer to this type throughout the system Human-readable type name and description for user-interface purposes Default model file extension, to store the model in the file system The set of supertypes, used to group types with related meaning The Java class for the top-level data node, used for automated type-checking Creation of default model data which is syntactically correct but need not have any semantic meaning. This is used when creating a new model of this type. This operation may also open a GUI dialog to ask for additional parameters that influence the created model data. Such dialogs should be used sparingly, since they are of no use for further modifications once a model is created. Typically, asking for excessive parameters for model creation indicates the need for a higher-level model type or simply a better editor. Validation of model data for semantic correctness Reading and writing of model data to convert between Java objects and file contents A GUI editor for model data Any number of additional, type-specific operations that can be applied on existing models of that type The corresponding Java interfaces to implement are TypeImplementation as well as the interfaces to which that refers, such as ModelEditor for the editor implementation. It is however cumbersome to write all model type implementations manually and without re-using code from other model types. It is customary for different model types to use the same data classes for parts of their model data. A useful pattern is therefore to implement the data classes as well as the corresponding 24