Using OpenStreetMap data. Nick Whitelegg Southampton Solent University, RM424

Transcription

1 Using OpenStreetMap data Nick Whitelegg Southampton Solent University, RM424

2 What is OpenStreetMap? OpenStreetMap[1] is a set of mapping data covering the whole world Liberally licenced (CC-by-SA) Anyone can contribute Rendered maps available, but the data is the important thing! See

3 Why OpenStreetMap? Existing mapping data was not free (as in beer or speech) Licencing fees necessary to use maps online Generally only the rendered maps were available, not the underlying data Data available to business, but at very high cost This restricted how you can use the data

4 It's the data! OSM has a default slippy map but access to the raw data is what makes OSM Developers can use this data in their own applications OSM has a web service API to allow access to the raw data Raw data also downloadable as XML files

5 Contributing Survey roads, footpaths and POIs with a GPS device Use freely-available data sources (Ordnance Survey out-of-copyright/opendata [2], Bing aerial imagery) Use an OSM editor (JOSM [3], Potlatch [4]) to draw highways on top of GPS traces or imagery

6 Overview of in-the-field mapping procedure Survey using a GPS device (e.g. smartphone, Garmin GPS unit) Load GPS track and waypoints into an editor (web or standalone) Download existing OSM data into the editor Draw new roads, paths, points of interest in OSM editor using your GPS track/waypoints Upload changes to OSM

7 Editing the data JOSM (josm.openstreetmap.de) Standalone editor, written in Java Powerful and flexible Communicates with API over HTTP Potlatch ( Online Flash Editor Easy to use id (ideditor.com) HTML5/JavaScript editor, in early development

8 Other sources of OSM data In addition to direct surveys, the following sources can be used to add data to OSM: Aerial imagery (Bing; special agreement) Out-of-copyright historical mapping (e.g. OS maps from 50+ years ago, i.e or before) Ordnance Survey OpenData or other liberallylicenced official data sources The editors can load in these data sources as a backdrop to the OSM data

9 The OpenStreetMap API API to allow developers to make use of OSM data REST-compliant web service providing data in OSM's own XML format Allows creation, retrieval and modification of map data Available at Examples of operations on following slides

10 Nodes and Ways OSM data consists of nodes and ways A node represents a point feature e.g. a point of interest, or a constituent point of a line feature such as a road A way represents a line or polygon feature e.g. a road, a stream, a wood, a lake Ways consist of two or more nodes

11 OpenStreetMap API - Examples Retreive map data in a given bounding box (OSM XML format) /map?bbox=-1.5,50.9,-1.4,51.0 Retrieve a given way (here, way with ID 223) /way/223 (GET request, retrieves OSM XML) Modify a way (here, way with ID 223) /way/223 (PUT request, send OSM XML to URL)

12 OpenStreetMap XML format <osm version="0.6"> <node lat="50.9" lon="-1.4" id="12345"> <tag k="place" v="city"/> <tag k="name" v="southampton"/> </node> <way id="223"> <nd id="1234" /> <nd id="1235" /> <tag k="highway" v="motorway" /> <tag k="ref" v="m27" /> </way>...

13 OpenStreetMap XML - Explanation The <node> tag represents a node It has an ID, latitude and longitude attributes and a series of free-form tags describing the point The <way> tag represents a way A way is made up of constituent nodes The XML consists of a series of node references, referenced by node ID (the <nd> tag) together with free-form tags describing the way

14 OSM tagging Properties of map objects in OSM are described by a series of tags Each tag has a key and value The key is the general class of property, the value is the actual value for that property

15 Examples of tags for linear ways highway: the class of road (motorway, primary, secondary etc) maxspeed: maximum speed in km/h ref: the road reference number oneway: is it a one way road? (boolean)

16 Examples of tags for nodes natural If it's a natural feature, indicates the type (hill summit, etc) man_made If it's a man made feature, indicates the type (water tower, mast etc) amenity If it's an amenity (e.g. a pub, restaurant, recycling centre etc) indicates the type

17 OSM way tagging example A main road could be described by the following tags: highway=primary Indicates it's a primary class road (A road in UK) name=high Street ref=a286 The road number maxspeed=64 Maximum speed in km/h

18 OSM node tagging examples Hill / Mountain: Pub: natural=peak; name=snowdon amenity=pub; name=the Crown Place: place=city; name=southampton Communications mast: man_made=mast

19 Examples of tags for areas Areas (polygons) are treated as ways (which just so happen to be closed) e.g. landuse for managed land landuse=forest or landuse=industrial or landuse=residential or natural for natural land cover natural=wood or natural=water or natural=heath

20 OSM planet files Requesting large amounts of data from API can place undue load on the OSM server API is really designed for adding and editing data, not retrieving So OSM also makes its data available as downloadable planet files in OSM XML format, containing OSM data for the entire planet Whole planet too large for most to use directly, so country or regional extracts available e.g. Geofabrik (

21 Extracting OSM data with osmosis Even with a country extract, chances are you are only interested in some of the data For example, a walkers' map might only need data of interest to walkers The osmosis tool (wiki.openstreetmap.org/wiki/osmosis) allows you to extract only the data you need

22 osmosis Osmosis allows you to (amongst other things): Extract data by bounding box or polygon Extract all nodes or ways containing a given key, or key/value pair Write OSM data to a MySQL database Read OSM data from a database and write to an OSM data file

23 osmosis Examples of osmosis commands: osmosis --rx file=uk.osm - bounding-box top=52 bottom=50 left=-2 right=1 -wx file=southern_england.osm This reads in uk.osm, extracts a bounding box defined by 50N, 52N, 2W and 1E, saving the results in southern_england.osm osmosis -rx file=uk.osm - tf reject-ways -tf accept-nodes place=* --wx file=ukplaces.osm This reads in uk.osm, rejects all ways and accepts only nodes with a place tag, writing the result to ukplaces.osm

24 General principles of osmosis Has a set of commands specified as commandline options, e.g. --rx (read XML), --tf (tag filter) Commands typically filter the input stream So the following command: osmosis -rx file=uk.osm -tf reject-ways -tf acceptnodes place=* --tf accept-nodes amenity=* --wx file=output.osm will only give nodes with both a place tag and an amenity tag If we want nodes with either a place tag or an amenity tag, we must use named pipes and stream merging, see the osmosis page for details

25 Overpass API An alternative to downloading the OSM data as a flat file The Overpass API is a third-party live web API which allows OSM data (a local copy of the OSM database) to be queried and returned as XML Uses an XML-based query language See

26 Setting up an OSM database You may wish to setup your own copy of the OSM database e.g. for rendering your own maps or developing a web service of some kind Typically, a PostgreSQL / PostGIS database is used for this OSM data is imported into two tables planet_osm_point and planet_osm_line

27 Setting up PostgreSQL / PostGIS PostgreSQL and its GIS-specific extensions, PostGIS, normally available as packages for Linux distributions Installation fairly straightforward on current Ubuntu or Debian distributions Detailed installation instructions available at: wiki.openstreetmap.org/index.php/mapnik/postgis

28 Importing OSM data Having set up the PostGIS database, OSM data must be imported into it A custom tool, osm2pgsql, is used for this This can be downloaded from OSM's github repository Osm2pgsql allows us to control which features are imported (by tag), minimising the size of the database See the osm2pgsql wiki page:

29 Map rendering OK, you've set up a database containing OSM data, but how are you going to generate your own maps? Two broad approaches: Server side rendering (e.g. Mapnik) Client side rendering (e.g kothic-js)

30 Mapnik: server-side rendering Mapnik (mapnik.org) is a C++ library (with Python wrappers) for rendering high-quality maps from geographical data Mapnik is able to read data from (amongst other data sources) PostGIS databases and uses an XML style file to control the look-and-feel of the final map Developers create programs in C++ or Python to render data using Mapnik from the PostGIS database

31 Helper tools for Mapnik Mapnik is powerful but can be tricky to setup and use So tools have been developed to ease creation of Mapnik maps One example is TileMill (mapbox.com/tilemill) An easily-installable package which allows you to design and render maps locally Uses a locally-running webserver

32 Creating web maps OK, we've got the maps rendered now, but what about creating a Google-style slippy map? JavaScript mapping libraries are used here Leaflet: OpenLayers: You need to ensure you generate tiles in the Google XYZ format

33 Brief intro to Google XYZ The tiling system used by most web mapping Map tiles (projected in Spherical Mercator ) described by an x,y,z coordinate scheme The x and y coordinates represent the tile's position in the world (x=0 is west; y=0 is north) The z coordinate represents the zoom level

34 Google XYZ zoom levels Zoom level 0 contains the whole world on one 256x256 pixel tile Zoom level 1 zooms in by a factor of two, so the whole world is contained by four 256x256 pixel tiles and so on for zoom levels 2, 3, etc See

35 Google XYZ zoom levels Zoom level 0 : whole world as 256x256 tile

36 Google XYZ zoom levels Zoom level 1: 4 tiles (2 rows, 2 cols) for world

37 Client-side rendering Server-side rendering can be computationally expensive, and requires large storage space (cache) once rendered Therefore not always the optimal solution Client-side, in-browser HTML5 canvas rendering possible with kothic-js (kothic.org/js)

38 kothic-js JavaScript library developed as a port of the Python renderer kothic Renders GeoJSON data client-side using HTML5 Canvas GeoJSON is a standard JSON-based format for representing geographical data, see geojson.org Server-side scripts must be written to generate the GeoJSON from a database using the Google XYZ tile system A Python implementation is available, produced by the kothic developers

39 Other uses of OSM data Location-based smartphone apps Apps can include vector maps Apps can find nearby points of interest downloaded from OSM Routing software OSM data can be converted to binary formats optimised for routing

40 Example site: Freemap Freemap ( is an example of a third-party site making use of OSM data OSM site geared for UK walkers, showing rights-of-way OSM data for parts of the UK loaded into PostGIS database with osm2pgsql Server-side scripts generate GeoJSON from the database kothic-js is used client-side to render the data OS LandForm-PANORAMA data[5] used for contours

41 Example mobile app: OpenTrail OpenTrail ( is Freemap's Android app It shows vector maps generated from OSM data using the Mapsforge library (code.google.com/p/mapsforge) It also allows users to find nearby POIs These are delivered from Freemap's web service, which is also used to generate the GeoJSON for Freemap itself

42 Novel uses of OSM data: example Augmented reality is an interesting concept in mobile development The general idea is that the real world (e.g. camera feed) is overlaid with virtual objects (e.g. POIs) Hikar (see is a researchware app which overlays OSM data (provided by Freemap's API) on the phone's camera feed and in doing so potentially offers a novel approach to navigation for walkers

43 References [1] OpenStreetMap, 2011 "OpenStreetMap" (online) Available: last accessed 6/6/11 [2] Ordnance Survey, 2011 "OS OpenData" (Online) Available last accessed 6/6/11 [3] OpenStreetMap, 2011 "JOSM" (online) Available: josm.openstreetmap.de, last accessed 6/6/11 [4] OpenStreetMap, 2011 "Potlatch 2" (online) Available: wiki.openstreetmap.org/wiki/potlatch_2, last accessed 6/6/11 [5] Ordnance Survey, 2011 "Land-Form PANORAMA - small-scale height data of Great Britain",(online) Available: last accessed 6/6/11

44 Thanks for listening Any questions?