How To Understand The History Of Science And Culture

Transcription

1 intersemiotic relations in multimodal text theory and methodology Sabine Bartsch Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft URI: The 4th Corpus Linguistics Conference University of Birmingham, UK July 2007 overview multimodal communication in science and engineering multimodal corpus studies intra- and intersemiosis in scientific documents conclusions and outlook Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 2 1

2 studies of the language of science and academia researchon languageforspecialpurposes(lsp) and language for academic purposes(lap) technical vocabulary and terminology hedging self-construal of the scientist in scientific papers lexicogrammar: nominalisation, grammatical metaphor etc. register profiling(biber et al.) expression of stance(biber 2004; Hyland 1988) Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 3 studies of the language of science and academia insights into the linguistic construal of scientific meaningand itsshiftin thecourseof thehistorical developmentof thesciences(e.g. Halliday1988; Halliday& Martin 1993; O Halloran 1999; 2004) registerstudiesof engineeringtexts(bartsch 2004; Bartsch et al. 2005) Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 4 2

3 multisemiosis in science and engineering interactive pdf: model of a turbine static pdf: expert paper Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 5 multisemiosis in science and engineering central aspects have been largely ignored by studies of scientific and academic text: visual artefacts such as images, schematic figures etc. symbolism(symbolic representations, formulae etc.) textual organisation, esp. including the role of other semiotic resources self-characterisation of a mechanical engineer: the drawing is the language of the engineer (Reiner Anderl, TU Darmstadt) Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 6 3

4 multisemiosis in science and engineering [ ] every linguistic act, spoken or written, takes place over more than one'mode' or channel of communication Thestudyof monomodal textstraditional in linguisticsand othertext-basedinvestigationsisthusrevealedas an abstraction whose validity is questionable. (Bateman et al. 2002) Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 7 research questions how does a discipline construe meaning? howisdomainknowledgeconstruedbymeansof different semiotic resources, i.e. by multisemiosis? how is multisemiosis instantiated in documents of the target domain? whatarethemechanismsthataccountforthejoint contributionof different semioticresourcesto themeaningof a multimodal document? Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 8 4

5 multisemiosis in science and engineering target domain: data processing in construction, a subdomain of mechanical engineering semiotic resources: natural language visual representations such as pictures, schematic figures, sketches etc. symbolic representations: formulae, diagrams etc. intermediary forms: tables, representations of data structures virtual models such as CAD (Computer Aided Design) models Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 9 multisemiosis in science and engineering domain content communicated semiotic resources (modes) photorealistic pictures animated visualisations schematic figures 3D CAD models symbols, formulae programming code natural language text realistic/ naturalistic scale of abstractness abstract domain knowledge Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 10 5

6 research issues characterisation of the individual semiotic resources characterisation of their individual and joint contribution to the meaning of the overall document characterisation of the interaction between them Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 11 research desiderates integrated multisemiotic view of scientific text studiescoveringlarger amountsand moretypesof multisemiotic texts theoretical and methodological foundation for multisemiotic register studies Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 12 6

7 theory Systemic Functional Grammar (SFG) (Halliday 2004) SF-MultimodalDiscourseAnalysis (SF-MDA) (O Halloran 2004b) -MDA Register linguistics(halliday& Hasan 1989; Biber et al. 1995; Martin 1985) Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 13 theory: SFG as a theory of intersemiosis metafunctions ideational: logical experiential interpersonal mode of meaning construeexperienceas meaning enactsocialrolesand relations Examples for language: (various ranks) taxonomic chains(lexis) lexis(word), transitivity(clause) mood(clause) textual modulateideational and interpersonal meaningas flow of information theme-rheme(clause), coherence(text) Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 14 7

8 methodology corpus and computational linguistics multilayer ML annotated corpora xlink/ xpointer to establish intersemiotic links tools working with standard ML encoding for annotation and query Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 15 multi-layer ML-encoded corpus field: data processing in construction a subdomain of mechanical engineering tenor: expert-to-expert, expert-to-learner, learner-to-teacher, expert-to-user/customer mode: written-to-be-read genres: research articles lab reports book chapters (instructional) web pages technology presentations Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 16 8

9 corpus semiotic resources: text (~ 1.7 mio. runningwords) visual representations(~ 780) tables(~ 300) diagrams virtual2d and 3D CAD models(~ 20) formulae, numerical data(unknown, yet) Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 17 corpus: overview expert-tolearner expert-toexpert expert-to-user student-toteacher research articles lab reports book chapters (instructional) web pages technology presentations Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 18 9

10 corpus: semiotic resources text visual representation symbolic representation intermediary types virtual models research articles lab reports book chapters instructional web pages technology presentations Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 19 levels of multisemiotic analysis intermodal organisation content structure genre rhetorical structure layout structure navigation structure intramodal structure form adaptedfromthegemmodel(bateman et al. 2001) Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 20 10

11 levels of multisemiotic organisation intrasemiotic organisation organisation of meaning within each semiotic resource intersemiotic organisation/ intersemiosis organisationof semioticresourceswithinmultimodal documents Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 21 multisemiotic text analysis: intrasemiotic organisation ideational metafunct. natural language tables programming code symbolism: symbols, formulae 2D and 3D models schematic figures photorealistic images experiential meaning lexicogrammar: vocabulary, terminology transitivity: processtypes, participants, circumstances under study under study symbols, numerical data participants and operators objects, process representations participants, processes/ actions, circumstances logicosemantic relations expansion, projection Various, Lemke 1998b under study expansion, projection Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 22 11

12 multisemiotic text analysis: intrasemiotic organisation natural language tables programming code symbolism: symbols, formulae 2D and 3D models schematic figures photorealistic images interpersonal metafunction mood, modality colour, shading, framing, highlighting commentary colour, shading, framing, highlighting textual metafunction lexical cohesion, grammatical cohesion, conjunctive relations arrangement indentation, object orientation (classes) visual cohesion Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 23 intrasemiotic organisation: linguistic text analysis situational parameters lexis field, tenor, mode terminology collocations grammar transitivity process types material, relational, mental, existential, behavioural, verbal participants circumstantial elements actor, goal etc. time, place, manner textual structure cohesion thematic structure rhetorical structure generic structure logico-semantic relations process type distribution Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 24 12

13 corpus annotation: linguistic annotation ideational features in language PoS-tagging (Stanford NLP Tools) syntactic parsing (Stanford NLP Tools) lexical analysis transitivity: ideational features (UAM CorpusTools; O Donnell 2007) textual features in language lexical cohesion Rhetorical structure (RSTTool; O Donnell 2001) automatic annotation semiautomatic + manual annotation some of these annotations are extensible to other semiotic resources: e.g. transitivity, lexical cohesion, RST Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 25 metadata for schematic figure: experiential description PARTICIPANT: AGENT: laser beam PROCESS: CHANGE-STATE fuse PARTICIPANT: powdered materials GOAL: into a solid object PARTICIPANT: AGENT: laser PROCESS: ACTION MOVEMENT: traces PARTICIPANT: the pattern, GOAL: sintering it together TRAJECTORY: vector, directionality Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 26 13

14 visual semiotic resource annotation Objects Actions Directionality PARTICIPANT: ACTOR PARTICIPANT: GOAL: CHANGE-STATE, ACTION MOVEMENT TRAJECTORY: vector, directionality laser powdered materials into a solid object fuse direction of object trajectory <unit id="u "> <object function type= participant-actor >laser </object> <action function type= change-state">fuse </action> <trajectory type= uni-directional </trajectory> </unit> Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 27 intersemiosis in science texts verbal intersemiosis visual ideational terminology, collocations processes, participants, (attributes), circumstances intersemiotic phoric links; intersemiotic cohesion; transitions(macro-and microtranisations) visual transitivity: actions objects, (attributes) interpersonal logico-semantic relations modality, modulation logico-semantic relations framing, highlightingof related participants or processes across srs logico-semantic relations color, shading, etc.; interactivity features textual lexical cohesion, grammatical cohesion, conjunctive relations intersemiotic reference, intersemioticrecurrence, repetition, substitution, ellipsis, rhetorical structure relations visual cohesion, (explicit) sequencing, interactional features: hyperlinks, sequencing, menues, buttons, clickable areas Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 28 14

15 intersemiosis: categories of a model status of semiotic resources: equal unequal logico-semantic relations expansion projection intersemiotic cohesion repetition of the same concept in different modes substitution of one mode for another, same concept paraphrasing of a concept in another semiotic resource semiotic transition micro-tarnsitions macro-transitions Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 29 inter-modal relations: status of modes image & text independent status equal image & text complementary image subordinate to text unequal text subordinate to image Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 30 15

16 inter-modal relations: logico-semantic relations logico-semantics: expansion: elaboration(elaborationon themeaningof anotherunitbya more detailed description of it) extension(extensionof themeaningof anotherunit/ mode by adding further, related information) enhancement(of a mode byqualifyingitin termsof time, place, cause, and other circumstantial meanings) projection possibility of extension of these relations to further discourseunitsand othermodessuch as images, tables, formulae, 3D models etc. Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 31 inter-modal relations: logico-semantic relations exposition: i&t same level of generality expansion elaboration extension enhancement exemplification temporal spatial conditional text more general image more general causal (reason purpose) comparative concessive Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 32 16

17 intersemiotic reference: explicit reference intermodal phoric links as shown in, as exemplified in,... as depictedin Figurex.x: Single-Point Cutting-Tool Geometry. Figure depicts the location of various angles of interest on a single-point cutting tool. usedratherunspecifically, no good indicatorof image function Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 33 intersemiotic reference: explicit reference Structures of interaction between modalities Links between text and visual modality Crossmodal referring expressions reference to document parts/ sections(e.g. Fig. 1 ) Multimodal referring expressions object reference by means of more than one modality Anaphoric referring expressions reference to previously introduced objects in abbreviated form cf. André, Rist 1994 Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 34 17

18 Intermodale Analyse multimodal referring expressions crossmodal referring expressions Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 35 intersemiotic reference: grammatical metaphor verbal indicative phrases in text vs. imperative phrases in picture photosensitive resin is sprayed on the build platform vs. spray resin grammatical metaphor Anaphoric referring expressions Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 36 18

19 intersemiotic cohesive chains and clusters: objects in different semiotic resources The technique [stereolithography] builds three-dimensional models from liquid photosensitive polymers that solidify when exposed to ultraviolet light. As shown in the figure below, the model is built upon a platform situated just below the surface in a vat of liquid epoxy or acrylate resin. A low-power highly focused UV laser traces out the first layer, solidifying the model's cross section while leaving excess areas liquid. Next, an elevator incrementally lowers the platform into the liquid polymer. A sweeper re-coats the solidified layer with liquid, and the laser traces the second layer atop the first. repetition text moregeneralthanimage Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 37 intersemiotic cohesive chains and clusters: processes in different semiotic resources The technique [stereolithography] builds three-dimensional models from liquid photosensitive polymers that solidify when exposed to ultraviolet light. As shown in the figure below, the model is built upon a platform situated complex just repetition below the surface in a vat of liquid epoxy or acrylate resin. A low-power highly focused UV laser traces out the first layer, solidifying the model's cross section while leaving excess areas liquid. Next, an elevator incrementally lowers the platform into the liquid polymer. A sweeper re-coats the solidified layer with liquid, and the laser traces the second layer atop the first. image more general and unspecific than text Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 38 19

20 intersemiotic cohesive chains and clusters: objects in different semiotic resources The technique [stereolithography] builds three-dimensional models from liquid photosensitive polymers that solidify when exposed to ultraviolet light. As shown in the figure below, the model is built upon a platform situated just below the surface in a vat of liquid epoxy or acrylate resin. A low-power highly focused UV laser traces out the first layer, solidifying the model's cross section while leaving excess areas liquid. Next, an elevator incrementally lowers the platform into the liquid polymer. A sweeper re-coats the solidified layer with liquid, and the laser traces the second layer atop the first. repetition hyponymy text moregeneralthanimage Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 39 intersemiosis: semiotic transition semiotic transition micro-tranistion: functional elements from one semiotic resource are contained within another semiotic resource example: symbols used in graphs, language used in formulae or schematic figures macro-transition: items consisting of one semiotic resource are replaced by items comprised of another semiotic resource example: language table visual image (graphs) symbolism(e.g. formula) O Halloran (2004a) Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 40 20

21 corpus annoation <unit id="u ">3.2 Laminated Object Manufacturing</unit> <unit ']" role="text-to-image" from= u to= u status= unequal-image-subord logsem= enhancement-temporal cohesion= complex-paraphrase />As shown in the figure below, a feeder/collector mechanism advances the sheet over the build platform, where a base has been constructed from paper and double-sided foam tape. </link></unit> <unit id="u " alt="bilder/figure2 objects= laser mirror platform collector processes= vertical_move horizontal_move /> <unit id="u "> <xlink:extended role="caption" title="figure caption"> <xlink:locator href="//gembase/unit[@id='u ']"/>Figure 2: Schematic diagram of laminated object manufacturing.</xlink:extended></unit> Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 41 conclusion& outlook multisemiosis is pervasive in scientific and engineering texts and will become even more central with the advent of new computational construction methods It can be described in terms of extended notions of cohesion and transition, but more work needs to be done especially with the advent of new forms of documents Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 42 21

22 Example: interactivity and cohesion in 3d-pdf implicit sequencing Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 43 references André, E., T. Rist Referring to World Objects with Text and Pictures. Coling Bateman, J., K.J. Rondhuis 'Coherencerelations': Towardsa generalspecification. Discourse Processes, Bateman, J., J. Delin, R. Henschel Multimodality and empiricism: methodological issues in the study of multimodal meaning-making. GeM Report 2002/1. Barthes, R Image Text Music. London: Routledge. Halliday, MAK, R. Hasan Cohesion in English. London, New York: Longman. Hoey, M Patterns of Lexisin Text. Oxford: OUP. Martinec, R., A. Salway A systemforimage-textrelations in old(and new) media. Visual Communication Vol 4(3): Mann, W.C., S.A. Thompson Rhetoricalstructuretheory: a theoryof text organization. Tech. rep. RS , USC/ISI, Los Angeles, CA. Martin, J.R English text: systems and structure. Amsterdam: Benjamins. O'Donnell, Mick Systemic Coder a Text MarkupTool. Version 4.5.URL: O'Halloran, K 'Towards a systemic functional analysis of multisemioticmathematics texts', Semiotica 124 1/2 (1999), O Halloran, K. 2004a. Multimodal Discourse Analysis. Systemic Functional Perspectives.London, New York: Continuum. O Halloran, K Mathematical Discourse. Language, Symbolism and Visual Images.London, New York: Continuum. Technische Universität Darmstadt Institut für Sprach- und Literaturwissenschaft 44 22