OSM Tile Server 2012-06-25 1 Contents OSM Tile Server (Version 2) <frederik.ramm@geofabrik.de>, 2012-06-25 1 Task Description & Background...1 2 Basic System Setup...2 2.1 Disable IPv6...2 2.2 Check partitions...2 2.3 Configure rsyslogd...2 2.4 Repositories...3 3 Application Setup...3 3.1 Install PostGIS...3 3.2 Install Mapnik and osm2pgsql packages...4 3.3 Install Osmosis...4 3.4 Create OSM user and prepare initial data load...4 3.5 Install map style...5 3.6 Install Tirex rendering manager...6 3.7 Create tiles...7 3.8 Convert metatiles to tiles...7 3.9 Automating tile creation and conversion...7 3.10 Set up Apache to deliver tiles...7 4 Replication Setup...8 4.1 Initialiazing replication...8 4.2 Automating replication, database loading, and tile generation...9 4.3 Configuration in a VM/cloud environment...10 1 Task Description & Background Installing a tile server that covers Denmark and Southern Sweden. Due to the relatively small area covered and the pricing scheme of the cloud service (cloud.dk), a nightly full import with complete tile rebuild (i.e. no on demand tile rendering) is planned; this will initially be set up on one machine but later the build machine will probably only be fired up at night and a smaller tile distribution machine will be run 24/7 to publish tiles. The desired map is style Mike Migurski's solar (osm2pgsql based) as an initial example, but a custom style is planned for later, possibly even using imposm not osm2pgsql. Setup on cloud.dk VM 83.221.133.5 (Ubuntu 12.04, 6 cores, 16GB RAM and 250 GB disk). Low zoom levels will be covered world-wide by loading them from elsewhere.
OSM Tile Server 2012-06-25 2 2 Basic System Setup 2.1 Disable IPv6 The virtual machine has a problem with IPv6 (web downloads that resolve to IPv6 addresses fail) so we disable IPv6 altogether: echo "1" > /proc/sys/net/ipv6/conf/all/disable_ipv6 2.2 Check partitions We reserve 20 GB for PostGIS and 100 GB for tiles. Separate partitions are not required but since the basic VM only has a 20 GB partition created, we add a partition sda2 for PostGIS and sda3 for tiles using fdisk: Device Boot Start End Blocks Id System /dev/sda1 2048 41943039 20970496 83 Linux /dev/sda2 41943040 104857599 31457280 83 Linux /dev/sda3 104857600 314572799 104857600 83 Linux Format sda2 and sda3 with XFS: apt-get install xfsprogs mkfs.xfs /dev/sda2 mkfs.xfs /dev/sda3 create mount points mkdir /tiles mkdir /var/lib/postgresql and add to /etc/fstab: /dev/sda2 /var/lib/postgresql xfs noatime,nobarrier 1 1 /dev/sda3 /tiles xfs noatime,nobarrier 1 1 then run mount -a to activate. 2.3 Configure rsyslogd In /etc/rsyslogd/50-default.conf, uncomment this line: daemon.* -/var/log/daemon.log Then restart rsyslogd. 2.4 Repositories Add to /etc/apt/sources.list:
OSM Tile Server 2012-06-25 3 # for GIS libraries deb http://ppa.launchpad.net/sharpie/for-science/ubuntu precise main deb-src http://ppa.launchpad.net/sharpie/for-science/ubuntu precise main # for PostGIS 2.0 deb http://ppa.launchpad.net/sharpie/postgis-nightly/ubuntu precise main deb-src http://ppa.launchpad.net/sharpie/postgis-nightly/ubuntu precise main # for osm2pgsql deb http://ppa.launchpad.net/kakrueger/openstreetmap/ubuntu precise main deb-src http://ppa.launchpad.net/kakrueger/openstreetmap/ubuntu precise main # for mapnik deb http://ppa.launchpad.net/mapnik/nightly-trunk/ubuntu precise main deb-src http://ppa.launchpad.net/mapnik/nightly-trunk/ubuntu precise main then run apt-get -u update 3 Application Setup 3.1 Install PostGIS apt-get install postgresql-9.1 postgresql-9.1-postgis Do su postgres then: createuser -DRS osm createdb osm -O osm psql osm -c "create extension postgis;" psql osm -c "grant all on geometry_columns to osm;" psql osm -c "grant all on spatial_ref_sys to osm;" Back to user root, then modify /etc/postgresql/9.1/main/postgresql.conf so that: fsync = off synchronous_commit = off autovacuum = off work_mem = 16MB maintenance_work_mem = 1GB shared_buffers = 2GB and modify /etc/postgresql/9.1/main/pg_hba.conf so that: local osm osm trust As root, enter sysctl kernel.shmmax=2221539328 and add the line kernel.shmmax=2221539328 to /etc/sysctl.conf. Now /etc/init.d/postgresql restart
OSM Tile Server 2012-06-25 4 3.2 Install Mapnik and osm2pgsql packages The osm2pgsql importer is used to load osm data into the PostGIS database. Mapnik is the rendering engine that draws OSM data to bitmaps. Install the following standard packages: apt-get install ttf-dejavu unifont apache2-mpm-worker libmapnik python-mapnik \ libprotobuf-c0 mapnik-utils Also install the following package built from OSM SVN by Geofabrik: dpkg -i osm2pgsql_0.80.0-6~precise_amd64.deb 3.3 Install Osmosis Osmosis is required to load diff files from OpenStreetMap and keep the locally-held data file up to date. Without loading diff files, the process would have to rely on extracts made available by a third party (e.g. Geofabrik) which incurs a delay of several hours. Install the following standard package: apt-get install osmosis 3.4 Create OSM user and prepare initial data load Create a Unix user to run everything under, and assign the previously created /tiles partition to this user; adduser osm chown osm /tiles As user OSM, then cd ~osm mkdir var cd var wget http://download.geofabrik.de/osm/europe.osm.pbf We have to prepare the initial data load from the Europe file because we want a bit more than just Denmark. Go to www.openstreetmap.org and use the Export tab to define a bounding box. (Do not export anything just use it to find the box you want.) We select this box: left=7.64 bottom=54.33 right=13.37 top=57.98 Run the following command to extract this area from the just-downloaded europe.osm.pbf: osmosis --read-pbf europe.osm.pbf --bb left=7.64 bottom=54.33 right=13.37 top=57.98 \ --write-pbf denmark.osm.pbf (Ignore messages about output lacking metadata.)
OSM Tile Server 2012-06-25 5 Import denmark.osm.pbf into PostGIS: osm2pgsql -d osm -U osm -c -C8000 --number-processes=4 denmark.osm.pbf This process takes roughly 10 minutes. 3.5 Install map style Download the map style: cd ~osm mkdir style cd style wget https://github.com/migurski/osm-solar/raw/master/tile-style.xml This style does not require any extra symbols or include files but others frequently do. Since our database is called osm and not planet_osm which the style expects, we modify the style file as follows (other styles often have an include file where one can set the database name etc.): sed -i -e 's/<parameter name="dbname">planet_osm/<parameter name="dbname">osm/' tile-style.xml sed -i -e 's/<parameter name="host">.*//' tile-style.xml The style also requires a database table names coastlines where it expects polygons for landmass (commonly called processed_p in OSM). Since keeping this in the database makes our update process more difficult, we provide these polygons in a shapefile called denmark_coastline.shp and change the style from to <Layer name="layer 13" srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs +over"> <StyleName>polygon style 7</StyleName> <Datasource> <Parameter name="dbname">osm</parameter> <Parameter name="estimate_extent">false</parameter> <Parameter name="extent">-20037508,-20037508,20037508,20037508</parameter> <Parameter name="table">coastline</parameter> <Parameter name="type">postgis</parameter> <Parameter name="user">osm</parameter> </Datasource> </Layer> <Layer name="layer 13" srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs +over"> <StyleName>polygon style 7</StyleName> <Datasource> <Parameter name="type">shape</parameter> <Parameter name="file">denmark_coastline.shp</parameter> </Datasource> </Layer> Finally, the style is made for Mapnik 0.7 but we run Mapnik 2.0, so we have it automatically converted:
OSM Tile Server 2012-06-25 6 sed -i -e 's/>name</>[name]</g' tile-style.xml upgrade_map_xml.py tile-style.xml tile-style-mapnik2.xml 3.6 Install Tirex rendering manager Tirex is a program that queues and prioritizes render requests, feeding them to Mapnik one by one, and creating so-called meta tiles with the render results on disk. Tirex would normally be run in conjunction with mod_tile for live rendering of tiles but we don't need that part; we just need the render queue. We install ready-made Debian packages from OSM's SVN repository: apt-get install libjson-perl libipc-sharelite-perl libgd-gd2-perl dpkg -i tirex-core_0.3.1ubuntu1_amd64.deb tirex-backend-mapnik_0.3.1ubuntu1_amd64.deb mkdir /tiles/meta chown -R osm /tirex chown -R tirex /tiles/meta chgrp osm /tiles/meta chmod g+ws /tiles/meta Tirex is configured using a number of config files. In /etc/tirex.conf, drop the bucket=... lines that are there by default, and replace them with only one: bucket name=default minprio=1 maxproc=4 maxload=6 This means we'll be rendering at most 4 tiles at a time and only if the load is not greater than 6. In /etc/tirex/renderer/mapnik.conf, set procs=8 fontdir=/usr/share/fonts/truetype fontdir_recurse=1 Create /etc/tirex/renderer/mapnik/solar.conf with these contents: name=solar tiledir=/tiles/meta minz=0 maxz=18 mapfile=/home/osm/style/tile-style-mapnik2.xml (It is possible to define any number of different styles by simply dropping in the proper config snippets.) Check if /usr/lib/tirex/backends/mapnik displays an error about not being able to load the Mapnik library; if yes, do cd /usr/lib ln -s libmapnik.so.2.1 libmapnik.so.2.0 Now (re)start the tirex rendering backend and the queue master /etc/init.d/tirex-backend-manager start /etc/init.d/tirex-master restart
OSM Tile Server 2012-06-25 7 and check /var/log/daemon.log for possible error messages. 3.7 Create tiles We are now ready to run a (meta)tile creation job using the tirex-batch utility: tirex-batch z=9-17 bbox=7.64,54.33,13.37,57.98 map=solar The command terminates quickly but Tirex will be working for a while. With the current machine and the rectangle chosen, tiles for z9-13 are done within 5 minutes; z14 takes 10 minutes, z15 takes 40 minutes, z16 takes about 2 hours. As stated before, the render area should probably be reduced for the higher zoom levels. The tirex-status utiliy can be used to inspect the current queue status. 3.8 Convert metatiles to tiles Tirex creates metatiles which are suitable for publishing through e.g. mod_tile. To convert them into simple tiles, use the meta2tile utility. As root, do mkdir /tiles/plain chown osm /tiles/plain then as user osm, /home/osm/bin/meta2tile /tiles/meta /tiles/plain This finds all metatiles in /tiles/meta, and unpacks them into /tiles/plain into a standard directory structure. 3.9 Automating tile creation and conversion This requests all of Denmark to be rendered, waits until complete, then converts to normal tiles: rm -rf /tiles/meta/* tirex-batch z=9-17 bbox=7.64,54.33,13.37,57.98 map=solar while [ "`tirex-status --raw grep "size" tail -1 cut -d: -f2`" -ne "0" ] do sleep 30 done /home/osm/bin/meta2tile /tiles/meta /tiles/plain The same job can be run for any other area, e.g. 12.3,55.5,12.7,55.8 for just the Copenhagen area. 3.10 Set up Apache to deliver tiles This is just a temporary measure because in the long run we plan to deliver tiles via another machine. Modify /etc/apache2/sites-available/default to have /tiles point to our tile directory by adding: Alias /tiles /tiles/plain
OSM Tile Server 2012-06-25 8 Then restart Apache: apache2ctl restart Create a trivial slippy map in /var/www/index.html to showcase our tiles: <html> <script src="http://openlayers.org/api/openlayers.js"></script> <script type="text/javascript"> var lat=55.7; var lon=12.4; var zoom=11; var map; function init() { map = new OpenLayers.Map ("map", { controls:[ new OpenLayers.Control.Navigation(), new OpenLayers.Control.PanZoomBar(), new OpenLayers.Control.ScaleLine({geodesic: true}), new OpenLayers.Control.Permalink('permalink'), new OpenLayers.Control.MousePosition(), new OpenLayers.Control.Attribution()], maxextent: new OpenLayers.Bounds(-20037508.34,- 20037508.34,20037508.34,20037508.34), maxresolution: 156543.0339, numzoomlevels: 19, units: 'm', projection: new OpenLayers.Projection("EPSG:900913"), displayprojection: new OpenLayers.Projection("EPSG:4326") } ); newlayer = new OpenLayers.Layer.OSM("Solar Tiles", "/tiles/${z}/${x}/${y}.png", {numzoomlevels: 19}); map.addlayer(newlayer); layermapnik = new OpenLayers.Layer.OSM.Mapnik("Mapnik"); map.addlayer(layermapnik); var switchercontrol = new OpenLayers.Control.LayerSwitcher(); map.addcontrol(switchercontrol); switchercontrol.maximizecontrol(); } </script> </head> <body onload="init();"> <div style="width:100%; height:100%" id="map"></div> </body> </html> 4 Replication Setup 4.1 Initialiazing replication Since we will be keeping our own OSM extract continually updated, we need to find out how old our local copy of the data is. This is a tricky operation but it only has to be performed once after setup (or again if the initial data load from chapter 3.4 is repeated, e.g. to enhance the coverage
OSM Tile Server 2012-06-25 9 area). We need to find out the highest OSM node ID in the extract we are using. This is achieved with the command psql osm -U osm -c 'select max(osm_id) from planet_osm_point' In the current dataset, this yields 1787693805. We check the creation date of that node on the OSM web site http://www.openstreetmap.org/browse/node/1787693805 and find: 2012-06-14 19:27 GMT. We now go to http://planet.openstreetmap.org/redaction-period/hour-replicate/001/ and find the.osc.gz file in which the node was created, using the time as an indicator. We find that the node is created in http://planet.openstreetmap.org/redaction-period/hourreplicate/000/001/715.osc.gz and therefore we go back one hour in time and use the matching state file as our replication starting point, http://planet.openstreetmap.org/redaction-period/hourreplicate/000/001/714.state.txt. We set up the Osmosis replication: cd ~osm/var mkdir replication-status osmosis --rrii workingdirectory=replication-status cd replication-status wget -O state.txt http://planet.openstreetmap.org/redaction-period/hourreplicate/000/001/714.state.txt The configuration.txt file needs to be changed so that baseurl=http://planet.openstreetmap.org/redaction-period/hour-replicate maxinterval = 0 We are now ready to test our replication and call Osmosis to update our extract: cd ~osm/var osmosis --rri workingdirectory=replication-status --simc --read-pbf denmark.osm.pbf --ac \ --bb left=7.64 bottom=54.33 right=13.37 top=57.98 --write-pbf denmark-new.osm.pbf Note that the same bounding box parameter as before is used, to avoid object additions outside of the are of interest. This automatically updates the state.txt file with the latest replication status. 4.2 Automating replication, database loading, and tile generation The following shell script combines downloading new OSM data, reloading the PostGIS database, and creating new tiles: #!/bin/sh -e cd /home/osm/var mkdir -p log exec >> log/update-`date -I`.log 2>&1 echo "------------------------------------------- echo "update on `date`" osmosis --rri workingdirectory=replication-status --simc --read-pbf denmark.osm.pbf --ac \ --bb left=7.64 bottom=54.33 right=13.37 top=57.98 --write-pbf denmark-new.osm.pbf
OSM Tile Server 2012-06-25 10 mv denmark-new.osm.pbf denmark.osm.pbf osm2pgsql -d osm -U osm -c -C8000 --number-processes=4 denmark.osm.pbf # the "rm" just removes the meta tiles, old "plain" tiles # will remain in /tiles/plain and will only be overwritten # if newly generated. rm -rf /tiles/meta/* # for all of Denmark tirex-batch z=9-17 bbox=7.64,54.33,13.37,57.98 map=solar # for Copenhagen only #tirex-batch z=9-17 bbox=2.3,55.5,12.7,55.8 map=solar # wait util tirex is done while [ "`tirex-status --raw grep "size" tail -1 cut -d: -f2`" -ne "0" ] do sleep 30 done # generate normal tiles from meta tiles. /home/osm/bin/meta2tile /tiles/meta /tiles/plain This script could be run as a cron job, or as the default execution script on a newly started VM (see next section). This script is installed as /home/osm/bin/update.sh on the VM. A cron job is not currently active. 4.3 Configuration in a VM/cloud environment It is conceivable to set up the server so that it automatically starts the above update script once it is booted up. If the server distributing the tiles is a different machine, then either all generated tiles need to be copied to the new machine after creation, or instead of rendering to a /tiles partition one could render directory onto some kind of mounted network storage (NFS or similar) which is accessed by the tile delivery server. If it is important to create new tiles quickly, it is possible to have more than one instance of the tile generating server, e.g. one for computing z9-16 and one for z17/z18. If the cloud system used does not normally permit hibernating VMs to have state information (i.e. if when they boot up they are always reset to a clean state) then the important pieces in the above are the state.txt file and the denmark.osm.pbf file; these need to be preserved between runs. Everthing else (tile directory, PostGIS storage) is transient.