While working with some DeepEarth map interfaces I stumbled across a really useful tool. Gdal2tiles is a cool python script that uses gdal to create tile pyramids from an image. This is the end result of a Google Summer of Code project and we have Klokan Petr Pridal to thank. In addition to creating the tile pyramid it can create a set of kml SuperOverlay files so that the resulting tile set can be served on top of Google. TMS tiles are also useful for other Tile interfaces like OpenLayers Flash and Silverlight DeepEarth.
As an experiment I downloaded the El Paso County Colorado NAIP from the USDA Geospatial Gateway site. This resulted in a .sid image of about 1.5Gb. Unfortunately the newer NAIP is not complete for El Paso County so I used the 2005 version at 1m resolution.
Now I downloaded the maptiler python wrapper: http://help.maptiler.org/betatest/ It is helpful to have the latest version of gdal2tiles, which for Windows, is easiest to get from the FWTools download page: http://fwtools.maptools.org/
Usage: gdal2tiles.py [options] input_file(s) [output]
–version show program’s version number and exit
-h, –help show this help message and exit
-p PROFILE, –profile=PROFILE
Tile cutting profile (mercator,geodetic,raster) -
default ‘mercator’ (Google Maps compatible)
-r RESAMPLING, –resampling=RESAMPLING
Resampling method (average,near,bilinear,cubic,cubicsp
line,lanczos,antialias) – default ‘average’
-s SRS, –s_srs=SRS The spatial reference system used for the source input
-z ZOOM, –zoom=ZOOM Zoom levels to render (format:’2-5′ or ’10′).
-v, –verbose Print status messages to stdout
KML (Google Earth) options:
Options for generated Google Earth SuperOverlay metadata
-k, –force-kml Generate KML for Google Earth – default for ‘geodetic’
profile and ‘raster’ in EPSG:4326. For a dataset with
different projection use with caution!
-n, –no-kml Avoid automatic generation of KML files for EPSG:4326
-u URL, –url=URL URL address where the generated tiles are going to be
Web viewer options:
Options for generated HTML viewers a la Google Maps
-w WEBVIEWER, –webviewer=WEBVIEWER
Web viewer to generate (all,google,openlayers,none) -
-t TITLE, –title=TITLE
Title of the map
-c COPYRIGHT, –copyright=COPYRIGHT
Copyright for the map
-g GOOGLEKEY, –googlekey=GOOGLEKEY
Google Maps API key from
-y YAHOOKEY, –yahookey=YAHOOKEY
Yahoo Application ID from
I didn’t explore all of these options, but used maptiler’s nice gui .py interface that queries for parameters to feed into the gdal2tiles pyramiding tool. The NAIP data is in UTM coordinates or EPSG:26913 for El Paso, CO. This means to be useful in Google’s kml or VE the image tiles needed to be warped to EPSG:900913. Lots of computation there.
. . . . . .
30 hours later, boy I really need to use that 64bit High cpu EC2 instance …..
I was troubled at first to see that gdal2tiles was only using one of my system’s cores, until I found Maptiler cluster. Klokan Petr Pridal is already moving on to address the time issue and exploring EC2 cluster possibilities. In theory a cluster of 8core EC2 instances could build tile pyramids in minutes rather than hours.
After completing, here is what I have ‘../ElPaso2005′ with a completed tile pyramid in the subdirectory tree levels 8 to 16. 435 subdirectories and 46,228 tiles. Each tile is a 256×256 jpg tile that meets OSGeo Tile Map Service TMS specification. The coordinates default to EPSG:900913/EPSG:3785 Mercator which matches the Google projection. The gdal2tile evidently calculates the optimal depth for my image resolution in a World Wide image. It then appears to do the complete tile division at the lowest level, in this particular case 16. Subsequent tile levels are simply mosaics of four tiles from the next level down so it is a relatively fast process to produce all the tiles above the bottom level. Total disk size for the jpeg files is only 1.7Gb which is very close to the original MrSId size.
Interfaces – DeepEarth, OpenLayers, Google Earth
OK nice, I have a local static tile set and I have some auto generated Google SuperOverlay and OpenLayer Flash interfaces, but since I’ve been playing with DeepEarth recently I next went to the DeepEarth codeplex project to grab the latest 1.0 release. This project has come a long way since I last played with it. I liked the interface from the beginning, but was stymied by the VirtualEarth TileProvider that was pulling hundreds of tiles from VE. A cost analysis showed this was going to be too expensive for common use. Now, however, the project has a number of other TileProviders such as OSM, Yahoo, and WMS, not yet Google. (Google restricts direct use of their tile servers) Google would share the VE disadvantage of cost but way less since the cost is per page view not tile downloaded. Of course really cool is being able to develop any xaml + TileProvider you want.
The project also includes an example showing how to set up a local tile set. The example uses 256×256 tiles but not in the OSGeo TMS directory structure. Here is an example using this DeepEarth local example TileProvider DeepEarth BlueMarble. You can see the tiles spin in from a local directory store on the server. The resolution is not all that great, but a full resolution BlueMarble isn’t that hard to get from BitTorrent. The alternative selection “Blue Marble Web” tiles are full 500m resolution hosted on Amazon S3 courtesy of ModestMaps.org. The Amazon S3 bucket is a flat structure, in other words the buckets don’t have an internal directory tree, which is why the tiles are not stored in a TMS directory tree.
The DeepEarth local TileProvider was easily adapted to suit the TMS directory so I could then directly pull in my El Paso tiles and show them with a DeepEarth interface. However, if I wished to take advantage of the high availability low latency S3 storage, I would need to flatten the tile tree. In S3 subdirectory trees are hiddeen inside buckets as file names. In the S3 case the tile names include the zoom level as a prefix: http://s3.amazonaws.com/com.modestmaps.bluemarble/5-r8-c24.jpg The 5 on the 5-r8-c24 nomenclature is the zoom level while row is 8 and column 24 at that level. TMS would encode this in a tree like this: ../bluemarble/5/24/8.jpg. The zoom level= subdirectory, row = subdirectory name, and column = name of tile. The beauty of TileProvider class in DeepEarth is that minor modifications can adapt a new TileProvider class to either of these encoding approaches.
Performance and reliability is a lot nicer on an Amazon S3 delivery, especially with heavy use. Once in S3 a map could also be promoted to a CloudFront edge cache without much difficulty. I imagine that would only make sense for the upper heavily used zoom levels, say 1-5 for BlueMarble. Once down in the level 6 part of the pyramid the number of tiles start escalating dramatically and repetitive hit rates are less likely.
Zoom Level Tile Count
I can see a common scenario moving toward this approach:
- Base layer tile sets from OSM, OpenAerial, VE, (not GE) etc
- Client specific base tile pyramids as an opacity overlay
- Dynamic layers as WMS from a middle tier WMS/WCS server like Geoserver
- Dynamic vectors coming from a spatial RDBMS like PostGIS behind Geoserver
···In this architecture map data is reused from open sources wherever possible and dynamic custom layers are coming out of OGC spec servers. Adding static client specific overlays to Tile storage provides the faster performance helpful for DeepZoom and Flash enabled interfaces.
Geospatial data is growing up and specialized data sources are multiplying. Now there are several background basemaps to choose from Virtual Earth, Yahoo, OpenStreetMap, Open Aerial. Google Earth is also available if you stick with the Google API in a Google container. The more sophisticated backgrounds require a commercial license, but there are also open sources that provide a great deal of capability. Projects like Flash OpenLayers and SIlverlight DeepEarth are improving the user experience substantially while giving developers a great deal of freedom.
A new Silverlight VE Map control is evidently on the way from Microsoft in the next couple of weeks. It is Silverlight so has some of the coolness factor of DeepEarth. I should be able to write some more on it next week.
I still like the approach of using a kml servlet to pull quads on demand from the seamless DRG service on Terraserver. Terraserver keeps chugging along.
Fig – Google Earth interface with DRG quads from Terraserver