Managing update of tiles of dynamic data

Transcription

1 Managing update of tiles of dynamic data Rob Stekelenburg IDgis Boomkamp 16 Rijssen Netherlands

2 Dutch spatial plans project web: plans and growing public and professional use high performance

3 Dutch spatial plans project IDgis hometown Rijssen Example plan 5 layers (most common) NL.IMRO.1742.BPRC bp_gebiedsaanduiding_overzicht bp_grenzen_all bp_grenzen_def bp_svbp_all bo_svbp_def Topography pre-tiled

4 Dutch spatial plans project Plans are updated daily (roughly per day) Zoomlevels 0-14 (1:100 km 1:50 m) Most plans contain several layers Whole Netherlands: 5.5 million tiles Results in high demand on the WMS Tiling solution needed Only tile the changed parts of each layer Tiling ready before office hours

5 Tiling manager Static data is pre-tiled (topography) Dynamic data is tiled before office hours TM GWC GWC Tiling Manager GeoWebcache GeoWebcache (adapted) plans tiles WMS

6 Challenges monitoring progress of GWC tasks extending GWC REST interface generation of GWC configuration using a template configuration and plan data optimizing Wms performance (caching) transforming the 24 bit png to 8 bit up-to-date tiles

7 Tiling manager cycle plans TM layers generate GWC configuration READY tasks truncate (delete tiles) seed (make tiles) send configuration to GWC reset GWC

8 Tiling manager architecture System Software

9 TM database schema

10 TM configuration file (part) # host, port and db_name of the database containing ro-online data (imrosyn, imrojobs, imro2008 schema, etc) tiling-manager.url=jdbc:postgresql://[db_host]:[db_port]/[roo_db] tiling-manager.driverclassname=org.postgresql.driver # username and password of the account with read/write access to the above database tiling-manager.username=[db_user] tiling-manager.password=[db_password] # host and port of the server(s) hosting geowebcache tiling-manager.gwcurl= tiling-manager.gwcprogressurl= tiling-manager.gwcstopurl= # url's to WMS's used for seeding by GeoWebCache tiling-manager.vvvpwms= tiling-manager.basiswms= tiling-manager.ondergrondwms= # NOTE: More than one geowebcache instance is supported :

11 ?

12 Links IDgis Deegree (WMS) Geowebcache download bbu3fopceipbyt2/gwc.zip Tiling Manager download bbu3fopceipbyt2/tiling.zip

13 Managing update of tiles of dynamic data Rob Stekelenburg IDgis Boomkamp 16 Rijssen Netherlands IDgis is a small Geo company in the Netherlands with clients mainly in government organizations like provinces, councils, and police This presentation will concentrate on a national project we where involved in, where tiling was used. But because of several challenges we had to development special software to meet the project goals.

14 Dutch spatial plans project web: plans and growing public and professional use high performance TEXT The dutch governement has a national facility for presenting spatial plans on all levels (council, province, national) At the moment about plans are in the system and it is growing daily The website is publicly available and there is also a number of facilities for professional use. High performance therefore is very important. GRAPH FADE IN this is a view of the site's main page. Each blue dot is a plan NEXT GRAPH: users need to be able to navigate and zoom quickly, to find the information they look for.

15 Dutch spatial plans project IDgis hometown Rijssen Example plan 5 layers (most common) NL.IMRO.1742.BPRC bp_gebiedsaanduiding_overzicht bp_grenzen_all bp_grenzen_def bp_svbp_all bo_svbp_def Topography pre-tiled TEXT As an example I will show a plan from the home town of our company, Rijssen, in the east of The Netherlands. GRAPH fade-in These 5 layers including borders and overview layers are used by most plans. "bp_gebiedsaanduiding_overzicht" "bp_grenzen_all" "bp_grenzen_def" "bp_svbp_all" "bp_svbp_def" Other types of plans (1% of all plans) can have their own specific layers. "is_vv_gr" "vs_vv_gr" The topography you see around it is fairly static data and therefore pre-tiled on seperate hardware.

16 Dutch spatial plans project Plans are updated daily (roughly per day) Zoomlevels 0-14 (1:100 km 1:50 m) Most plans contain several layers Whole Netherlands: 5.5 million tiles Results in high demand on the WMS Tiling solution needed Only tile the changed parts of each layer Tiling ready before office hours TEXT A varying number of plans is updated, loaded or removed on a daily basis tiles must be generated over a wide zoom range for several different layers There are 700 unique layers of which 5 most common are used by most plans Example: The topography layer (whole NL, 15 zoomlevels) 5.5 million tiles. Takes 24+ hr to tile. Using tiling is the solution, but: Maybe a few % of a layer's area has actually changed Before office hours the layers have to refreshed, so that tiling does not stress the WMS in the daytime.

17 Tiling manager Static data is pre-tiled (topography) Dynamic data is tiled before office hours TM GWC GWC Tiling Manager GeoWebcache GeoWebcache (adapted) plans tiles WMS static data like topography is pretiled on other hardware dynamic data tiles cannot tile the main 5 layers for whole of the Netherlands every day layers must be refreshed within the bounding box of the plan itself (all in all few % of total surface area) NEXT using an adapted version of GWC and a tiling manager application that sits between the sites plan database and GWC The Tiling Manager software queries the plan database for updates, executes tiling tasks in collaboration with GeoWebCache and monitors progress. The plan database is updated each night, after that the TM receives a command to start tiling

18 Challenges monitoring progress of GWC tasks extending GWC REST interface generation of GWC configuration using a template configuration and plan data optimizing Wms performance (caching) transforming the 24 bit png to 8 bit up-to-date tiles Among the challenges we faced where monitoring GWC progress by extending the REST interface with progress command The TM has a simple information page to show progress. Each TM task is for a specific layer for a plan generating an up-to-date GWC configuration using templates, Webdav and tomcat restart request Optimizing WMS performance by using caching optimizing tiles size using pngquant to shrink from 24 to 8 bit keeping transparancy in tact. make sure the tiles are up to-date This is ensured by the tiling manager process cycle (next slide)

19 Tiling manager cycle plans TM layers generate GWC configuration READY tasks truncate (delete tiles) seed send configuration to GWC reset GWC (make tiles) Every night, after plans are updated in the database, the TM receives a refresh command It finds the layers of all relevant (updated) plans and makes a GWC task for each of them These tasks show up in the TM info page. The cycle starts with generation of a new GWC configuration based on the new list of plans then a truncate (deletion of tiles) is done for layers based upon the old configuration (day before). next the new configuration is send to GWC and it is reset so it reads ands uses this new configuration then the seeding of new tiles is started and all tasks are processed one by one when the list is empty we are ready

20 Tiling manager architecture System Software System architecture TM and GWC are deployed in tomcat Each TM can handle several GWC's communication is via http so TM and GWC can reside anywhere in a multi server system All is configured in a TM properties file TM will send each layer task to a different GWC All GWC use the same configuration though and will send Getmap request to same WMS. TM software architecture Tiling Manager application translates http requests to calls to the ManagerService. The ManagerService has a GWC pool that contains 1 or more GWC instances the status of a GWC instance is held in GWCtaskstatus

21 TM database schema tables: tile_refresh_tasks keeps the TM cycles gwc_seed_tasks keeps track of all individual GWC tasks plantype_tile_layers contains relation between plan types and layer names seed_bboxes is used for (pre)tiling topography to have an efficient set of bounding boxes around the country, instead of just 1

22 TM configuration file (part) # host, port and db_name of the database containing ro-online data (imrosyn, imrojobs, imro2008 schema, etc) tiling-manager.url=jdbc:postgresql://[db_host]:[db_port]/[roo_db] tiling-manager.driverclassname=org.postgresql.driver # username and password of the account with read/write access to the above database tiling-manager.username=[db_user] tiling-manager.password=[db_password] # host and port of the server(s) hosting geowebcache tiling-manager.gwcurl= tiling-manager.gwcprogressurl= tiling-manager.gwcstopurl= # url's to WMS's used for seeding by GeoWebCache tiling-manager.vvvpwms= tiling-manager.basiswms= tiling-manager.ondergrondwms= # NOTE: More than one geowebcache instance is supported : The Tiling Manager configuration is done with a single tiling-manager.properties file. This contains properties for : database connections GWC REST commands WMS in the system Webdav (send configuration to GWC) tomcat (GWC reload) refresh timings threadcount (# threads per GWC task)

23 ?

24 Links IDgis Deegree (WMS) Geowebcache download bbu3fopceipbyt2/gwc.zip Tiling Manager download bbu3fopceipbyt2/tiling.zip