M GIS. A Short Introduction to Volunteered Geographic Information Presentation of the OpenStreetMap Project

Transcription

1 M GIS A Short Introduction to Volunteered Geographic Information Presentation of the OpenStreetMap Project Sylvain Bouveret LIG-STeamer / Université Grenoble-Alpes Quatrième École Thématique du GDR Magis. Sète, September 29 October 3, 2014

2 Sources Part of the presentation dedicated to OSM inspired from: An old joint presentation with N. Petersen and Ph. Genoud Nicolas Moyroud: Several talks from 3rd MAGIS summer school 2012 Released under licence CC-BY-SA and downloadable here: Guillaume Allègre: Cartographie libre du monde: OpenStreetMap Released under licence CC-BY-SA. Reference book about VGI [Sui et al., 2013] Other references cited throughout the presentation Sui, D. Z., Elwood, S., and Goodchild, M., editors (2013). Crowdsourcing geographic knowledge: Volunteered Geographic Information (VGI) in Theory and Practice. Springer. M GIS 2 / 107

3 Outline 1. Introduction to Volunteered Geographic Information 2. Presentation of the OpenStreetMap Project 3. Using OpenStreetMap Data 4. Using Volunteered Geographic Information M GIS 3 / 107

8 1 First part Introduction to Volunteered Geographic Information Beyond traditional GIS A new trend Some examples Features of participative datasets Volunteered vs Contributed Open vs Closed Sensing vs Thinking Volunteered Geographic Information

9 Introduction to Volunteered Geographic Information Beyond traditional GIS A new trend Some examples Features of participative datasets Volunteered vs Contributed Open vs Closed Sensing vs Thinking Volunteered Geographic Information M GIS 5 / 107

10 A very brief history of GIS First users of GIS (at least in France): local authorities, department of defense Early 90 s: paper maps (unprecise, to be regularly updated...) : Digital transposition of data 2000: Integration to enterprise IS (first spatial extensions to Oracle and Postgres) 2002: Geospatial webservers + OGC standards 2005: Mobility M GIS 6 / 107

11 A very brief history of GIS First users of GIS (at least in France): local authorities, department of defense Early 90 s: paper maps (unprecise, to be regularly updated...) : Digital transposition of data 2000: Integration to enterprise IS (first spatial extensions to Oracle and Postgres) 2002: Geospatial webservers + OGC standards 2005: Mobility 2004: Participative data, geographic crowdsourcing, volunteered geographic information, neogeographic datasets... M GIS 6 / 107

12 OpenStreetMap Project started on 2004 Open and collaborative geographical database of the world Content generated by users (about 1.8M registered users) Free license (initially CC-by-sa; ODbL since 2012) More about OpenStreetMap later M GIS 7 / 107

13 Wikimapia Project started on 2006 Aims at marking all geographical objects in the world and providing a useful description of them Mostly provides a way for users to give annotations about places in the world, (initially) using Google Maps as a base layer. Free license since 2010 (CC-by-SA). M GIS 8 / 107

14 Google Map Maker Project started on 2008 Equips Google Maps with a map edition interface Every registered user can submit modifications Modifications have to be approved before being published in Google Maps Data released under proprietary license M GIS 9 / 107

15 Ushahidi Free Software and platform for crisis management Crowdsourcing-based mapping Focuses on information flow (smartphones, SMS,...) Web platform Based on OpenStreetMap and Google Maps for Geocoding (source: Wikipedia). M GIS 10 / 107

16 Ligue de Protection des Oiseaux A network of participatory science websites dedicated to wildlife inventory Anyone can participate by adding observations to the database Requires some basic knowledge about different species In general, no verification is made, except for outliers Search engine and visualization tool (map) on the website M GIS 11 / 107

17 Other examples Geolabeled Flickr Images ( ) Foursquare ( ) UCrime ( )... M GIS 12 / 107

19 Features All these applications are examples of geographical crowdsourcing approaches M GIS 14 / 107

20 Features All these applications are examples of geographical crowdsourcing approaches Some strong common points... Data contributed on a voluntary basis by users Geospatial nature of data (or at list a part of it)...but very different features as well: Aims Geospatial as a first-class citizen or not Skills required Process for quality assessment (data verification) Data license M GIS 14 / 107

21 To volunteer or to contribute? In all these examples, data is jointly produced by users volunteering to contribute (geographical crowdsourcing) M GIS 15 / 107

22 To volunteer or to contribute? In all these examples, data is jointly produced by users volunteering to contribute (geographical crowdsourcing) But... What about geolocalized data transmitted by a smartphone, (more or less) unbeknownst to its user? M GIS 15 / 107

23 To volunteer or to contribute? In all these examples, data is jointly produced by users volunteering to contribute (geographical crowdsourcing) But... What about geolocalized data transmitted by a smartphone, (more or less) unbeknownst to its user? An example of crowdsourced geospatial data, assuredly not volunteered! M GIS 15 / 107

24 Opt-in vs opt-out Two approaches to crowdsourced geographic data [Harvey, 2013]: Volunteered Geographic Information (opt-in): Clarity about purposes Control over data collection Some guarantees about data reuse Contributed Geographic Information (opt-out): Unclear purposes No (or little) control over data collection No control over data reuse Harvey, F. (2013). To volunteer or to contribute locational information? Towards truth in labelling for crowdsourced geographic information. In Crowdsourcing Geographic Knowledge [...], chapter 3. Springer. M GIS 16 / 107

25 Data reuse licenses A key distinction between opt-in and opt-out: control over data reuse...raises the following crucial questions: Who owns the data jointly produced by users? Under which license this data can be used? M GIS 17 / 107

26 Data reuse licenses A key distinction between opt-in and opt-out: control over data reuse...raises the following crucial questions: Who owns the data jointly produced by users? Under which license this data can be used? Two antagonistic examples: Google Map Maker: Google owns the data, releases it under proprietary license, whose conditions can change whenever it wants (cf April 2011) irrespective of whether the user is a regular contributor or not OpenStreetMap: Data belongs to the contributors, and is released under a free license M GIS 17 / 107

27 Geomatics and open (free) licenses In geomatics, most datasets are released under proprietary licenses, even those who are funded by public money IGN, Ordnance Survey,... Even if it is free (cost-less), cannot be freely exploited (e.g for scientific projects). Some VGI projects like OSM explicitly aim at providing free and open geospatial data. M GIS 18 / 107

28 Geomatics and open (free) licenses In geomatics, most datasets are released under proprietary licenses, even those who are funded by public money IGN, Ordnance Survey,... Even if it is free (cost-less), cannot be freely exploited (e.g for scientific projects). Some VGI projects like OSM explicitly aim at providing free and open geospatial data. But... How to do it while still being compatible with author s right freely release data while protecting it and its authors? M GIS 18 / 107

29 Geomatics and open (free) licenses In geomatics, most datasets are released under proprietary licenses, even those who are funded by public money IGN, Ordnance Survey,... Even if it is free (cost-less), cannot be freely exploited (e.g for scientific projects). Some VGI projects like OSM explicitly aim at providing free and open geospatial data. But... How to do it while still being compatible with author s right freely release data while protecting it and its authors? The software community has a solution: use Free licenses Initially dedicated to software (like GNU/GPL) ill-suited for other kinds of intellectual stuff (music, books, pictures, information... ) M GIS 18 / 107

30 Data and open licenses: Creative Commons First option for geomatics: Creative Commons licenses (artworks) M GIS 19 / 107

31 Data and open licenses: Creative Commons First option for geomatics: Creative Commons licenses (artworks) 4 options Attribution (BY) the original author has to be credited Non Commercial (NC) no commercial profit allowed No Derivatives (ND) no derived work allowed Share Alike (SA) derivatives must be licensed under identical terms Six possible combinations CC-by, CC-by-sa, CC-by-nc, CC-by-nc-nd, CC-by-nc-sa, CC-by-nd M GIS 19 / 107

32 Data and open licenses: ODbL Initial OSM data was released under the terms of the CC-by-sa license. However: it turned out that it was not very adapted... Combining OSM data with other datasets Share-Alike only applies to rendered maps (tiles), not to data itself Attribution too many contributors! Uncertainty about derived work After two years of effort, OSM switched to ODbL in September 2012: Attribution, Share Alike, Redistribution (as long as one of the redistributed versions is kept open). All the past contributors have been contacted... most of them agreed with the new terms some of them explicitly disagreed ( data erased) some of them did not answer ( data erased) M GIS 20 / 107

33 Sensing vs Thinking Different applications different skills and levels of implication required from the users Haklay, M. (2013). Citizen science and volunteered geographic information: Overview and typology of participation. In Crowdsourcing Geographic Knowledge [...], chapter 7. Springer. M GIS 21 / 107

34 Sensing vs Thinking Different applications different skills and levels of implication required from the users VGI as geographic citizen science (even if not every VGI application falls into this category) Haklay, M. (2013). Citizen science and volunteered geographic information: Overview and typology of participation. In Crowdsourcing Geographic Knowledge [...], chapter 7. Springer. M GIS 21 / 107

35 Sensing vs Thinking Different applications different skills and levels of implication required from the users VGI as geographic citizen science (even if not every VGI application falls into this category) The ladder of implication according to Haklay [2013] Level 1: Crowdsourcing (citizens as sensors, volunteered computing) Level 2: Distributed intelligence (basic interpreters, volunteered thinking) Level 3: Participatory science (implication in problem definition and data collection) Level 4: Extreme citizen science (problem definition, data collection and analysis) Haklay, M. (2013). Citizen science and volunteered geographic information: Overview and typology of participation. In Crowdsourcing Geographic Knowledge [...], chapter 7. Springer. M GIS 21 / 107

37 What is VGI? As we have seen, crowdsourcing is just one feature of VGI M GIS 23 / 107

38 What is VGI? As we have seen, crowdsourcing is just one feature of VGI Volunteered Geographic Information [Goodchild, 2007] Volunteered geographic information is the harnessing of tools to create, assemble, and disseminate geographic data provided voluntarily by individuals. Goodchild, M. F. (2007). Citizens as sensors: the world of volunteered geography. GeoJournal, 69(4): M GIS 23 / 107

39 Neogeographic datasets Coote and Rackham [2008] propose the following (complementary) characterization of neogeographic datasets: Creation stimulated by lack of available data or restrictions, costs, limitations of conventional data sources Involve geographic information provided voluntarily by individuals Creation and management are not necessarily ruled by accepted standards Data licensed using open-source approach Coote, A. and Rackham, L. (2008). Neogeographic data quality is it an issue? In AGI Geocommunity conference, ConsultingWhere Ltd. M GIS 24 / 107

40 The big picture? M GIS 25 / 107

41 The big picture? Participative M GIS 25 / 107

42 The big picture? Participative Geospatial M GIS 25 / 107

43 The big picture? Participative Citizen science Geospatial M GIS 25 / 107

44 The big picture? Participative Citizen science Open data Geospatial M GIS 25 / 107

45 The big picture? Participative Citizen science VGI? Open data Geospatial M GIS 25 / 107

46 2 Second part Presentation of the OpenStreetMap project OSM: History and principles What is OSM? History Technical considerations Data model The OSM ontology Contributing

47 Presentation of the OpenStreetMap project OSM: History and principles What is OSM? History Technical considerations Data model The OSM ontology Contributing M GIS 27 / 107

48 OpenStreetMap: Genesis OpenStreetMap was created in July 2004 by Steve Coast, which was then studying at the University College of London. He did not understand why the Ordnance Survey created massive geographical datasets but did not freely distribute them to those who had paid to create them (i.e happy tax payers). NB: the same thing happens in almost every country in the world (except USA and the Netherlands) He then decided to start a mapping project whose aim would be to freely provide data to the users. M GIS 28 / 107

49 OpenStreetMap: principles Collaborative: mainly individual contributions the more contributors, the more complete the world coverage Open: data freely usable without restriction (ODbL). Data: OpenStreetMap is not a map, it is a database. Online map only provided for visualization purposes No airborne or satellite view M GIS 29 / 107

50 One dataset, several maps Mapnik standard style M GIS 30 / 107

51 One dataset, several maps Transport map M GIS 30 / 107

52 One dataset, several maps Cycle map M GIS 30 / 107

53 One dataset, several maps MapQuest M GIS 30 / 107

54 One dataset, several maps FranceTopo.fr (enriched with other public datasets such as Nasa SRTM) M GIS 30 / 107

55 One dataset, several maps 3D-OSM (XNavigator University of Bonn and Heidelberg) M GIS 30 / 107

56 History 1. Founding and Early History 9th August openstreetmap.org registered by Steve Coast 20th August Steve Coast presented his mapping idea at EuroFOO 2nd September First posting to the mailing list 17th July Map Limehouse the first Mapping Party 22th January Release of version 1.0 of the offline editor JOSM 20th August OpenStreetMap Foundation registered 10th November Mapnik rendered Slippy map makes its debut. 4th December Yahoo! aerial imagery sketching allowed M GIS 31 / 107

57 History 2. The Start of OSM s Current Technology Stack 5th May st version of the Potlatch editor. 14th-15th July First conference, State Of The Map 2007, held in Manchester. September TIGER data import for the US started 20th September AND Data for The Netherlands imported January The French Direction générale des finances publiques officially allows the OSM contributors to use the Cadastre as a source of data. M GIS 32 / 107

58 History 3. The Switch to API 0.6 and the explosion of User Growth 21st April Big switch to API version 0.6 1st April Ordnance Survey Opendata releases. OSM partly responsible for bringing this about. 30th November Use of Bing vertical aerial imagery allowed 25th November Association OpenStreetMap France registered 12th Sept License switched over to ODbL M GIS 33 / 107

59 Number of contributors 2e e+06 Today 1.6e+06 Number of contributors 1.4e e+06 1e / / / / / / / / /01 Date M GIS 34 / 107

60 Paris Paris in OpenStreetMap september 2006 october M GIS 35 / 107

110 Current state Current state of the database (25 th September, 2014) Number of users 1,800,453 Number of uploaded GPS points 4,218,137,961 Number of nodes 2,535,804,643 Number of ways 253,523,371 Number of relations 2,818,286 M GIS 36 / 107

111 Presentation of the OpenStreetMap project OSM: History and principles What is OSM? History Technical considerations Data model The OSM ontology Contributing M GIS 37 / 107

112 Basic OSM data model As in most GIS, each geographical entity is described in OSM using: Geographical information (geometries) Attributes ( semantics) M GIS 38 / 107

113 Basic OSM data model As in most GIS, each geographical entity is described in OSM using: Geographical information (geometries) Attributes ( semantics) More precisely, each OSM entity has: a numeric identifier: OSM ID a geometry a set of generic attributes present for every element uid, user: user id and user name timestamp: time of the last modification visible: if false then the element should only be returned by history calls version: edit version of the object (starts from 1) changeset: the changeset (group of edits made within a certain time by one user) in which the object was created or updated a set of tags (attributes): key-value pairs M GIS 38 / 107

114 Geometries in OpenStreetMap Three kinds of geometries: 1. nodes : basic element. Geographic point: latitude & longitude (WGS84) Point Of Interest (POIs) 2. ways : ordered interconnection of nodes open ways linear features (roads, railways...) closed ways areas 3. relations : group of any primitive with associated roles Relate nodes, ways and potentially other relations to each other, thereby forming complex objects (e.g. multipolygons). relationship between objects and abstract objects M GIS 39 / 107

115 Example of a way M GIS 40 / 107

116 Example of a relation M GIS 41 / 107

117 Tags in OpenStreetMap Attributes of geographical entities are described using tags M GIS 42 / 107

118 Tags in OpenStreetMap Attributes of geographical entities are described using tags A tag is a key-value pair A key broadly describes an element (e.g. highway, name) A value specifically describes its accompanying key M GIS 42 / 107

119 Tags in OpenStreetMap Attributes of geographical entities are described using tags A tag is a key-value pair A key broadly describes an element (e.g. highway, name) A value specifically describes its accompanying key Use of keys and values is unrestricted (free text) the data model is infinitely extensible anyone can define and use its own keys and values M GIS 42 / 107

120 Anarchy? Unrestricted tags M GIS 43 / 107

121 Anarchy? Unrestricted tags lead to uncontrolled data production process. M GIS 43 / 107

122 Anarchy? Unrestricted tags lead to uncontrolled data production process. Uncontrolled data production process leads to anarchy. M GIS 43 / 107

123 Anarchy? Unrestricted tags lead to uncontrolled data production process. Uncontrolled data production process leads to anarchy. Anarchy leads to useless data. M GIS 43 / 107

124 Anarchy? Unrestricted tags lead to uncontrolled data production process. Uncontrolled data production process leads to anarchy. Anarchy leads to useless data. Unrestricted tags are the path to the dark side. M GIS 43 / 107

125 Anarchy? Unrestricted tags lead to uncontrolled data production process. Uncontrolled data production process leads to anarchy. Anarchy leads to useless data. Unrestricted tags are the path to the dark side. Actually, tagging in OSM is governed by an agile self-organizing community process defining the basic ontology of OSM M GIS 43 / 107

126 Anarchy? Unrestricted tags lead to uncontrolled data production process. Uncontrolled data production process leads to anarchy. Anarchy leads to useless data. Unrestricted tags are the path to the dark side. Actually, tagging in OSM is governed by an agile self-organizing community process defining the basic ontology of OSM Resources to find an appropriate tag or explore tag usage: Map Features ( ) an extensive list of the most commonly used tags Taginfo ( ) a useful site to exploring current tag usage, including tag values that are not necessarily documented M GIS 43 / 107

127 Example of OSM tags Tags to describe real world objects building = church, hotel, school, university... highway = motorway, primary, secondary Tags to describe immaterial objects boundary = administrative, national_park Commons tags name = * source = * M GIS 44 / 107

128 OSM tags M GIS 45 / 107

129 A more formal OSM ontology? Simple tag structure and unrestricted tags are probably one reason for the success of OSM (easiness of contribution, flexibility, extensibility...) M GIS 46 / 107

130 A more formal OSM ontology? Simple tag structure and unrestricted tags are probably one reason for the success of OSM (easiness of contribution, flexibility, extensibility...) The price to pay is poor semantics harder to detect and fix logical inconsistencies less expressive queries link to other datasets? (Linked Data) Some attempts to provide a formal (ontological) backbone to the OSM ontology: e.g. LinkedGeoData M GIS 46 / 107

131 OSM Editing tools How to contribute? M GIS 47 / 107

132 OSM Editing tools How to contribute? Id ( ) JavaScript online editor, web application easy to use, recommended for beginners JOSM ( ) Java OpenStreetMap Editor desktop application written in Java, with a plugin architecture for advanced users, large set of features and tools M GIS 47 / 107

133 OSM Editing tools How to contribute? Id ( ) JavaScript online editor, web application easy to use, recommended for beginners JOSM ( ) Java OpenStreetMap Editor desktop application written in Java, with a plugin architecture for advanced users, large set of features and tools Potlatch ( ) written in Flash, can be used directly from a web browser has been made obsolete by Id Merkaartor ( desktop application C++,Qt (Windows, GNU/Linux, MacOSX) Plugins for: QGIS ( ) ArcGIS ( ) M GIS 47 / 107

134 The Id editor M GIS 48 / 107

138 Typical JOSM edition session M GIS 49 / 107

142 3 Third part Using OpenStreetMap data Exploiting the data Retrieving data (basic principles) OSM API Other querying tools End user applications Tutorial: create your own database Basic principles A step by step example

143 Using OpenStreetMap data Exploiting the data Retrieving data (basic principles) OSM API Other querying tools End user applications Tutorial: create your own database Basic principles A step by step example M GIS 51 / 107

144 OSM and server applications The OSM server cannot be queried for data directly (only a small amount). Instead, it can provide some dumps and regular diffs application servers have to instantiate the DB locally. The OSM server can provide tiles updated on a regular basis. planet.osm (first import) OSM server openstreetmap.org/ Regular tile generation (Mapnik) tiles regular diffs Application server (data) API or Web server Client (e.g OpenLayers) Client M GIS 52 / 107

145 Mapnik A (standalone) tile generator for OpenStreetMap: Mapnik OSM file PostGIS database OR Mapnik Tile (image) M GIS 53 / 107

146 OSM API v0.6 RESTFul API to consult and edit OSM entities requests take the form of HTTP GET, PUT, POST, and DELETE messages requests return or expect the data for the entities in a XML format Example : GET /api/0.6/[node way relation]/#id returns the XML representation of the entity M GIS 54 / 107

147 OSM API v0.6 <osm version =" 0.6 " generator =" OpenStreetMap server " copyright =" OpenStreetMap and contributors " attribution =" http: // www. openstreetmap. org / copyright " license =" http: // opendatacommons. org / licenses / odbl /1-0/ "> <way id=" " visible =" true " timestamp =" T09:00:14Z " version =" 11" changeset =" " user=" Liberal " uid=" " > <nd ref=" "/ > <nd ref=" "/ > [...] <nd ref=" "/ > <nd ref=" "/ > <tag k=" amenity " v=" university "/ > <tag k=" building " v=" yes "/ > <tag k=" contact:website " v=" http: // ensimag. grenoble - inp. fr/"/ > <tag k=" name " v=" Ensimag - D"/ > </ way > </ osm > M GIS 55 / 107

148 OSM API v0.6 <osm version =" 0.6 " generator =" OpenStreetMap server " copyright =" OpenStreetMap and contributors " attribution =" http: // www. openstreetmap. org / copyright " license =" http: // opendatacommons. org / licenses / odbl /1-0/ "> <node id=" " version ="3" changeset =" " lat=" " lon=" " user=" FredB " uid=" 1626 " visible =" true " timestamp =" T19:09:04Z " > <tag k=" source " v=" cadastre -dgi - fr source : Direction Générale des Imp^ots - Cadastre. Mise à jour : 2009 "/ > </ node > </ osm > M GIS 56 / 107

149 Overpass API an optimized read-only API that serves up custom selected parts of the OSM map data a powerful query language with search criteria like e.g. location, type of objects, tag properties, proximity, or combinations of them. M GIS 57 / 107

150 Overpass API: Example Overpass XML <query type =" node " > <has -kv k=" name " v=" Ensimag "/> <bbox - query e=" 5.78 " n=" " s=" " w=" 5.67 "/ > </ query > <union > <item / > < recurse type =" down "/ > </ union > <print / > Overpass QL relation [" name "=" Ensimag "] (45.13,5.67,45.22,5.78) ; (._; >; ); out OR Output in OSM XML or JSON M GIS 58 / 107

151 Overpass API: Example 13,5.67,45.22,5.78);(._;>;);out; <? xml version=" 1.0 " encoding="utf -8"?> <osm version ="0.6 " generator =" Overpass API "> <note >The data included in this document is from www. openstreetmap.org. The data is made available under ODbL.</ note > <meta osm_base=" T09:07:03Z "/> <node id=" " lat =" " lon =" "/> [...] <node id=" " lat =" " lon =" "/> <node id=" " lat =" " lon =" "/> <way id=" "> <nd ref =" "/> [...] <nd ref =" "/> <tag k=" amenity " v=" university "/> <tag k=" building " v="yes "/> <tag k=" contact:website " v=" http: // ensimag.grenoble -inp.fr/"/> <tag k="name " v=" Ensimag - D"/> </ way > [...] < relation id=" "> < member type ="way " ref =" " role =""/> [...] M GIS 59 / 107

152 Overpass API Applications Overpass Turbo: A web based graphical user interface for Overpass API M GIS 60 / 107

153 Overpass API Applications Public transport line generator M GIS 61 / 107

154 Overpass API Applications Public transport line generator width=1600&force-rows=1 M GIS 61 / 107

155 Overpass API Applications Public transport line generator width=1600&force-rows=1 M GIS 61 / 107

156 Nominatim Nominatim: tool to search OSM data by name and address. Accessible through the HTTP protocol with GET parameters can be queried with a standard web browser or with command-line tools. Three output formats: Standard HTML (a web page with embedded openlayers map) XML JSON Example: [sylvain@msnordlys]~ $ curl " de+la+passerelle%2c+saint-martin-d%27h%c3%a8res&polygon=1&format=xml" M GIS 62 / 107

157 Nominatim: result of the query <? xml version=" 1.0 " encoding="utf -8"?> <searchresults timestamp= Thu, 21 Mar :36: attribution= Data (c) OpenStreetMap contributors, ODbL 1.0. http: // www. openstreetmap.org / copyright querystring= rue de la Passerelle, Saint -Martin -d ' Hères polygon= true exclude_place_ids= , , more_url= http: // nominatim. openstreetmap.org / search? format = xml &amp ; exclude_place_ids = , , & amp ; polygon =1& amp ;q=rue+de+la+passerelle %2C+ Saint -Martin -d %27H%C3% A8res > <place place_id = osm_type = way osm_id = place_rank = 26 boundingbox =" , , , " lat= lon= display_name= Rue de la Passerelle, Saint - Martin -d ' Hères, Domaine Universitaire, Saint -Martin -d ' Hères, Grenoble, Isère, Rh^one - Alpes, 38402, France métropolitaine, European Union class= highway type= unclassified importance= 1.6 /> <place place_id = osm_type = way osm_id = place_rank = 26 boundingbox =" , , , " lat= lon= display_name= Rue de la Passerelle, Saint - Martin -d ' Hères, Domaine Universitaire, Saint -Martin -d ' Hères, Grenoble, Isère, Rh^one - Alpes, 38402, France métropolitaine class= highway type= unclassified importance= 1.6 /> </ searchresults > M GIS 63 / 107

158 End user applications MapOSMatic WheelMap OpenRouteService: Thematic maps: map of skiing/snowboarding pistes map of cycling routes map of sea navigation elements M GIS 64 / 107

159 Using OpenStreetMap data Exploiting the data Retrieving data (basic principles) OSM API Other querying tools End user applications Tutorial: create your own database Basic principles A step by step example M GIS 65 / 107

160 Data formats Where to download? : OSM files (XML format) by geographical area, city, country... : Garmin Map files, Shape files, TomTom POI, Adobe Illustrator,..., by geographical area, city,... : OSM files + changesets. What to download? OSM files: planet.osm for the entire planet (currently over 27GB compressed, over 300GB uncompressed), or regional extracts. Diffs: changesets for regular database updates (example: weekly changesets in ). M GIS 66 / 107

161 Create your own database OSM file PostGIS database Several possible tools and schemas... Osm2pgsql: Lossy translation (only converts some entities). Reconstructs polygons from relations (e.g for administrative boundaries) Osmosis: OSM data general purpose processing tool converts OSM data to Postgres/PostGIS DB (sticks to OSM datamodel) generates planet dumps from a DB applies or generates changesets extracts data... M GIS 67 / 107

162 A small Example The problem I want to develop a little application that displays a map showing the French departments and regions, and the French road network. My initial configuration: a PC running Ubuntu GNU/Linux OS 1st step. Install Postgresql: [sylvain@msnordlys]~ $ sudo apt-get install postgresql-9.1 M GIS 68 / 107

163 PostGIS installation 2nd step. Install PostGIS: $ sudo apt-get install postgis postgresql-9.1-postgis (A piece of cake... We have some packages for that in the distribution) M GIS 69 / 107

164 Database creation 3rd step. Login as postgres user and create a new user and a new database (and set this new user to be the database s owner): [sylvain@msnordlys]~ $ sudo -s Password: [root@msnordlys]~ # su postgres [postgresql@msnordlys] ~ $ psql postgres=# CREATE USER [username]; CREATE ROLE postgres=# ALTER USER [username] WITH ENCRYPTED PASSWORD [password] ; ALTER ROLE postgres=# CREATE DATABASE [dbname]; CREATE DATABASE postgres=# ALTER DATABASE [dbname] OWNER TO [username]; ALTER DATABASE M GIS 70 / 107

165 PostGIS Enabling 4th step. Enable PostGIS spatial functions into the database: $ cd /usr/share/postgresql/9.1/contrib/postgis-1.5 $ createlang plpgsql mydb [...] $ psql -d [dbname] -f postgis.sql [...] $ psql -d [dbname] -f spatial_ref_sys.sql [...] $ cd.. $ psql -d [dbname] -f postgis_comments.sql [...] M GIS 71 / 107

166 Data conversion 5th step. Run osm2pgsql to load Data into the database: $ osm2pgsql -U [username] -d [dbname] -r pbf --cache=4000 \ > -W france.osm.pbf Password: Using projection SRS (Spherical Mercator) Setting up table: planet_osm_point NOTICE: table "planet_osm_point_tmp" does not exist, skipping Setting up table: planet_osm_line [...] Completed planet_osm_polygon Osm2pgsql took 15962s overall M GIS 72 / 107

167 A test request We can now check whether the data has been correctly loaded... $ psql -U [username] -d [dbname] Password for user [username]: psql (9.1.8) Type "help" for help. [dbname]=# SELECT name, place, ST_XMin(way), ST_YMin(way) [dbname]-# FROM planet_osm_point WHERE name= Grenoble AND place= city ; name place st_xmin st_ymin Grenoble city (1 row) M GIS 73 / 107

168 Boundaries and roads The SQL requests inside the program... Retrieve administrative boundaries SELECT name, ST_SimplifyPreserveTopology (way,5000), admin_level FROM planet_osm_polygon WHERE boundary = administrative AND admin_level <= 6 ; Retrieve the road network SELECT ST_SimplifyPreserveTopology (way,5000), highway FROM planet_osm_line WHERE highway IN ( motorway, trunk, primary, secondary ); M GIS 74 / 107

169 The result M GIS 75 / 107

170 4 Fourth part Using Volunteered Geographic Information Authoritative data vs participative data Authoritative data, conventional data About metadata Can we trust OpenStreetMap? About data quality Some OSM data quality tools Case 1: Data quality for Location Based Services Case 2: Evaluating OSM data quality on the Department of Sarthe

171 Using Volunteered Geographic Information Authoritative data vs participative data Authoritative data, conventional data About metadata Can we trust OpenStreetMap? About data quality Some OSM data quality tools Case 1: Data quality for Location Based Services Case 2: Evaluating OSM data quality on the Department of Sarthe M GIS 77 / 107

172 VGI vs authoritative My project [put here the name of any serious research or industrial project] cannot use VGI data because we cannot control, trust, verify [put any other reason here] it... (Anonymous quotation) M GIS 78 / 107

173 VGI vs authoritative My project [put here the name of any serious research or industrial project] cannot use VGI data because we cannot control, trust, verify [put any other reason here] it... (Anonymous quotation) VGI is not a very important trend in GIS nowadays, so we do not consider this approach in our GIS department (Head of GIS department of a big IT group cited from memory) M GIS 78 / 107

174 VGI vs authoritative My project [put here the name of any serious research or industrial project] cannot use VGI data because we cannot control, trust, verify [put any other reason here] it... (Anonymous quotation) VGI is not a very important trend in GIS nowadays, so we do not consider this approach in our GIS department (Head of GIS department of a big IT group cited from memory) Are participative and authoritative data production so antagonistic? Can we trust participative data? M GIS 78 / 107

175 Authoritative data Authoritative data produced by professional mapping organizations M GIS 79 / 107

176 Authoritative data Authoritative data produced by professional mapping organizations According to Van der Molen and Wubbe [2007] (cited by Coleman [2013]): High-quality database Explicit guarantees about quality assurance Contains essential data about persons, institutions, issues, entities... Designated by law as the sole officially recognized register to be used by government agencies Coleman, D. J. (2013). Potential contributions and challenges of VGI for conventional topgraphic base-mapping programs. In Crowdsourcing Geographic Knowledge [...], chapter 14. Springer. Van Der Molen, P. and Wubbe, M. (2007). E-government and e-land administration-as an example: The netherlands. In 6th FIG Regional Conference, San Jose, Costa Rica, pages M GIS 79 / 107

177 Conventional data The term authoritative is a bit restrictive Coote and Rackham [2008] prefer the term conventional data: Collected for specific purpose and requirements Usually not free (charged) Use limited to some organizations or individuals Copyrighted data Managed by organizations established for the purpose Collected by professional staff, paid for this Based on standard and established methods and practices Quality assessment at different levels, guarantees provided to the user Coote, A. and Rackham, L. (2008). Neogeographic data quality is it an issue? In AGI Geocommunity conference, ConsultingWhere Ltd. M GIS 80 / 107

178 Example IGN, BD Topo Restricted access, specific license (free for a sample, charged for the rest) IGN keeps the ownership of data, and only gives utilization rights Collected by professional staff at IGN, standard production techniques Complete metadata following ISO standard Well-documented dataset, detailed quality assessment M GIS 81 / 107

179 About metadata One important aspect in Geographical datasets is metadata. M GIS 82 / 107

180 About metadata One important aspect in Geographical datasets is metadata. Metadata: data about the data (producer, owner, units, error bounds, geographical bounding box, production process, source of data...) Usually, metadata is separated from data (ISO 19115) Usually, metadata production is separated from data production M GIS 82 / 107

181 About metadata One important aspect in Geographical datasets is metadata. Metadata: data about the data (producer, owner, units, error bounds, geographical bounding box, production process, source of data...) Usually, metadata is separated from data (ISO 19115) Usually, metadata production is separated from data production But in the context of VGI... M GIS 82 / 107

182 About metadata One important aspect in Geographical datasets is metadata. Metadata: data about the data (producer, owner, units, error bounds, geographical bounding box, production process, source of data...) Usually, metadata is separated from data (ISO 19115) Usually, metadata production is separated from data production But in the context of VGI... Some metadata elements make sense for the dataset as a whole: owner (?), description of the dataset... But some don t really: bounding box, producer, production process... M GIS 82 / 107

183 Metadata embedded Let s have a look at OpenStreetMap... M GIS 83 / 107

184 Metadata embedded Let s have a look at OpenStreetMap... Reminder: Each OSM entity has: a numeric identifier: OSM ID a geometry a set of generic attributes present for every element uid, user: user id and user name timestamp: time of the last modification version: edit version of the object (starts from 1) changeset: the changeset (group of edits made within a certain time by one user) in which the object was created or updated comment: each changeset has an associated comment describing it a set of tags (attributes): key-value pairs M GIS 83 / 107

185 Metadata embedded Let s have a look at OpenStreetMap... Reminder: Each OSM entity has: a numeric identifier: OSM ID a geometry a set of generic attributes present for every element uid, user: user id and user name timestamp: time of the last modification version: edit version of the object (starts from 1) changeset: the changeset (group of edits made within a certain time by one user) in which the object was created or updated comment: each changeset has an associated comment describing it a set of tags (attributes): key-value pairs Metadata, embedded in the description of the entity itself M GIS 83 / 107

186 Metadata embedded Metadata embedded: user, timestamp, version, changeset, comment M GIS 84 / 107

187 Metadata embedded Metadata embedded: user, timestamp, version, changeset, comment But tags can also contain metadata, e.g.: source = cadastre-dgi-fr source : Direction Générale des Imp^ots - Cadastre. Mise à jour : 2009 tiger:source = tiger_import_dch_v0.6_ survey:date = M GIS 84 / 107

188 Metadata embedded Metadata embedded: user, timestamp, version, changeset, comment But tags can also contain metadata, e.g.: source = cadastre-dgi-fr source : Direction Générale des Imp^ots - Cadastre. Mise à jour : 2009 tiger:source = tiger_import_dch_v0.6_ survey:date = Actually, some metadata is common to the whole OSM dataset: license, conditions of use, and OSM Wiki ( ) M GIS 84 / 107

189 Centralized vs distributed approaches to metadata Traditional approach to metadata: centralized standards (ISO-19115) Clear standards, easy to navigate Heavy to produce sometimes incomplete even for authoritative data Does not always completely make sense Does not always completely reflects the production process Distributed user-centric approach: Can be messy Lightweight and flexible approach Consistent with the production process M GIS 85 / 107

190 VGI and metadata Conclusion about VGI and metadata [Poore and Wolf, 2013]: the process of producing metadata has to be rethought for VGI, because the dataset is: dynamically (continuously) generated generated locally by hundreds, thousands, or millions of users Production of metadata has to be integrated to the data production process Poore, B. S. and Wolf, E. B. (2013). Metadata squared: Enhancing its usability for volunteered geographic information and the geoweb. In Crowdsourcing Geographic Knowledge [...], chapter 4. Springer. M GIS 86 / 107

191 Using Volunteered Geographic Information Authoritative data vs participative data Authoritative data, conventional data About metadata Can we trust OpenStreetMap? About data quality Some OSM data quality tools Case 1: Data quality for Location Based Services Case 2: Evaluating OSM data quality on the Department of Sarthe M GIS 87 / 107