Generalising geographic data

It often happens that the dataset you have is very detailed and hence very large. Displaying such volume of data in online mapping applications (eg. as KML on Google Map) is out of the question. The only alternative is to reduce the number of vertices to reduce the overall size of the file (ie. generalise the data). There is an excellent free online tool to do just that and surprisingly it works much better than commercial software that costs a fortune. It is called MapShaper.


It accepts spatial data in shapefile format only. It is an online service and you can upload files up to 80MB in size for processing. There are 3 generalisation algorithms to choose from. Activation of generalisation process itself is as simple as moving a slider “left to right” (from 0% reduction to 99%) – the result is visible on the screen instantaneously. Another great feature of the tool is that you can view original file underneath the generalised version to assess the quality of the process but also to pick the right generalisation level for your requirements.

Processed files can be downloaded back in either shp or EPS format (so it’s is also a great tool for straight conversion of files from shp format into EPS format, for use with vector graphics software!). There is simple edit functionality as well so you can add/delete vertices manually, if required. Generalisation process preserves topology so boundaries of adjoining polygons are generalised in an identical fashion – that is, they still “fit” perfectly. All in all, MapShaper is very straightforward to use.

File size still counts when performance of online mapping tool is the objective. For example, it will not be possible to display electoral boundaries as a single, full resolution file in Google Map application. But original 25MB shapefile can be reduced to just a few hundred KB using MapShaper tool, and then even further by converting shp to gzipped KMZ format. Here is an example of a Google Map application that uses both, full resolution data for close-ups and generalised version for whole of the country view: Australian elections map (work in progress!).

Related posts:
Free map service preview
Converting csv data into shapefile
Converting shapefile into KML
Colours for thematic mapping
Free GIS Tools – Google Map
Free Address Validation Tool
Manual geocoder for 70 countries

Converting shapefile into KML

In the inaugural post in the Free GIS Tools series I presented Google Map API as a very powerful online GIS alternative. Google Map has the capacity to display spatial data in various formats but probably the most attractive option is its native support for KML/KMZ. It takes only one line of code to display complex spatial information on a Google Map – as long as it is in that format. True, not all KLM features are supported in Google Map but you can still create great maps using a single KML file. Try aus-emaps.com free map service to test how well Google Map handles KMZ files: Victorian bushfires aftermath, current earthquakes.

KML/KMZ is the only spatial data format directly supported by free Google Map and Google Earth desktop application. However, a lot of free spatial data is distributed in another very popular GIS format: shapefile (shp). In order to display that data in Google Map or Google Earth, it has to be converted to KML/KMZ format.

There are many tools available for download, or that are bundled with commercial software to do that conversion, but I found only one that really work for me. And the best thing is that it is totally free! It took quite an effort to find so, by sharing this information I hope to save you many hours of fruitless trawling through the Internet. Here are the details.


ESRI Shapefile to KML Converter by reimers.dk

Simple but effective desktop tool (it works!) available for free download. Just point to the shp file you want to convert, select attributes to include in description field and press a button to run conversion. The only limitation is that you don't have a choice in selecting colours.




Here is an example of generated output: World Borders in KMZ (1.14MB kmz file draped on Google Map).

To reduce the size of KML file for distribution or deployment just gzip it. It is easy done using Google Earth “save as” function. That is, open your KML file in Google Earth and when you save it, it will be automatically converted to a compressed gzipped format with kmz extension.

For custom conversions from shp polygons to KMZ format, such as splitting a large file with many polygons into smaller files with individual polygons only (eg. to convert a single 60MB shp file with postcodes into 2,500 individual KMZ files), or grouping of polygons into custom areas (eg. to create sales or franchise territories by amalgamation of postcodes), or converting shp polygons into a thematic map layer (eg. by assigning colours to polygons based on specific attributes, like for example electorates by political party holding the seat), you will need more advanced tools. I can help you with those conversions for a small fee. Here is a sample of a thematic KMZ file using a selection of Sydney suburbs. I also have various polygon based data in KMZ format for immediate delivery (eg. postcode boundaries, as used in aus-emaps.com postcode finder service). Don’t hesitate to ask!

Use case example: Create outlines of polygons and show in an embedded map on a website (using collection of free tools and services available from aus-emaps.com)
Steps:

1. Define vertex coordinates of custom polygon area using simple geocoder tool from aus-emaps.com (select “Point” option and click on the map, reposition markers as required, copy to Notepad).

2. Convert a list of points in csv format to shp file format using instructions provided in a recent post titled “Converting csv files into shapefile”.

3. Convert shp file into KML using tools introduced in this post.

4. Convert KML file into KMZ format using Google Earth (open KML file and “Save As”…) [optional step]

5. Upload KML/KMZ file to your server and reference its URL address in an aus-emaps.com free map service.

6. Embed the map in you web page using HTML’s iframe element.

The process is a bit convoluted, I admit. You could take a shortcut by pasting latitude and longitude coordinates directly into KML “shell” but you will need the right template for this in the first place. Using Google Map’s MyMap drawing and embedding functionality is a simpler option but you will not be able to create complex polygons (eg multi-part and/or doughnut type).

I am currently working on a public tool to generate custom polygons from administrative boundaries in a few simple steps. Coming up in 2011! But I will also have to put some though into creating a dedicated KMZ generation tool so you can do it all from a single web page... added to the list of “Things to do!”.


Related posts:
Free map service preview
Converting csv data into shapefile
Colours for thematic mapping
Free GIS Tools – Google Map
Free Address Validation Tool
Manual geocoder for 70 countries

Free GPS navigation in Android phones in Oz

This week Google released its free turn-by-turn navigation app for Android smart phones in Australia. It is called Google Maps Navigation. It was only a matter of time since Google has already launched similar services in the US and UK. There is no peed limit information but otherwise, it is the “whole GPS package and more”… Smartphone based navigation applications are gaining rapidly over lower end traditional GPS tools.

Related Posts:
Google enters mobile GPS Navigation Market
Google Maps Navigation launch in UK
New Features on Google Map for mobiles
Free GPS navigation tool for iPhone
Pushing the limits of GPS navigation
Trends and opportunities in mobiles market
Mobile map giveaway from Nokia
Tom Tom GPS navigation in iPhone
Ban on use of phone GPS navigation in cars

Incorrect maps lead to war

Oh, and on the subject of spatial information and business, GIS is not perfect. Bad data and bad maps can cause a lot of troubles... As recently reported “…a Nicaraguan military commander caused an international incident by leading his soldiers into Costa Rican territory…” Apparently, Google Maps were wrong. There are many disputes around the World caused by less than accurate representation of national borders.

Although the whole premise of the article is now proven incorrect, nevertheless it highlights the importance of using correct spatial data for decision making. Even generally trusted and reliable sources can get things wrong. So, read the small print and disclaimers before embarking on the big decisions!

Bad location can send you broke

All spatial industry professionals know the importance of "location" in many aspects of business operations but the statement may be an eye opener for those not dealing with spatial information for business decisions on day to day basis. An article published earlier this week by smartcompany.com.au and titled: Poor site selection caused Kripsy Kreme crash, Sumo Salad founder Luke Baylis says well highlights the issues.

“Collapsed donut chain Krispy Kreme expanded too rapidly and selected extremely poor sites for its products… the market for ‘fat, greasy food’ remains large and it was Krispy Kreme's expansion strategy that led to its troubles, and not the healthiness of its food… they weren't overly clever in the way the expanded, in terms of their site selection and the positioning of their stores … in lower socio-economic demographics, their product would have had more appeal - people generally aren't as health conscious in those areas, according to research…”

Geodemographic analysis techniques and tools have been around since 1980’s but you don’t have to apply complex statistical models and methodologies to gain an insight into your customers or profile catchment areas of your stores or sales territories. Simple Census data from the Australian Bureau of Statistics and thematic maps are enough to get you stared and visualise relevant information. As they say, picture is worth 1,000 words! I will continue on this subject with a few follow up posts so, stay tuned. Meantime, here is a link to thematic maps with Census statistics for postcodes, with over 90 various demographic characteristics available for analysis.

Atlas of Living Australia

A new portal has been launched last week to provide rich information resource about Australian plants and animals. It is a "public face" of a $64.7 million initiative, funded over 6 years by the Australian Government and a group of fourteen contributors comprising scientific organisations, universities and museums. It aims to enable any user to quickly locate and access information across the Internet on all aspects of Australian biodiversity. And of course, as the name implies, the portal comes equipped with a mapping application – what better way to provide easy access to such an extensive and geographically diverse range of information!

The mapping application supporting Atlas of Living Australia is built with OpenLayers implementation of Google Map and has quite a few interesting features. In particular, I like the concept of local, regional and species "perspective" to access the information via a map driven interface. Local "your area" view gives a quick snapshot of all the species in visitor’s current location (but Google’s "IP address to location" web service is sometimes not very precise in determining user’s true location). Regions view starts with a long list of areas to choose which is a bit unfortunate but once you make your selection you have a choice to download a large selection of data in csv format.

Species Map is where the real fun starts. You can search for species using common names as well as scientific terminology – lookup feature is enabled so it shows suggestions as you type in the text. Occurrences are then mapped as points or as numbered cluster markers. Click on the map returns a query on how many occurrences of particular species are in 10km radius from a selected point.



I like how selected data layers are managed – they are added to the “current list” at the top of the navigation panel and then can be individually edited (colours, transparency, size of markers). Users can change order of layers with drag-and-drop feature.

The application comes with a long list of contextual layers – too many to mention! All well referenced to metadata and with auto generated legends for easy identification of what various colour scales mean. Created maps can be saved as images for further reuse. And for those in the know, there is an analytical module as well but I will not even try to describe what it can do – as most of such tools, it is designed for a specific purpose, not for average user.

Atlas of Living Australia is a good example of a very powerful online mapping and analysis application created with free OpenLayers implementation of Google Map and integrated with free GeoServer at the back end to manage access to the data for downloads and as WMS and WFS web services to display information dynamically on the map. It did crash on me a couple of times and some image tiles were not displaying but that are just teething problems of a newly deployed application. If something goes wrong you can always start again with a click on reset button. Very convenient feature! Access to all the data via web services is planned in the near future when the application is more stable.

Free map service preview

I am rushing the release of my latest free service offering from aus-emaps.com due to high level of interest in accessing postcode and other administrative boundaries data on a reference map that can be dynamically called from external sites. At this stage the free map service is more like a preview release rather than the real thing but it is good enough for a "show and tell". The full concept behind this service is rather a long story so I will leave it for a more appropriate occasion. For now I will just limit the description to a short statement that it is intended as a free reference map, for embedding into website or linking to, to share location specific information.

Currently it works only with kmz files that can be referenced from anywhere on the world wide web and postcode boundaries from aus-emaps.com. It accepts single and multiple kmz files, as per examples below.

Here are a couple of examples of kmz files with complex information: in this case, Victorian bushfires aftermath (public file extracted from Google MyMaps): http://www.aus-emaps.com/svs/ref/map.php?
kmzl=http://www.aus-emaps.com/data/VictorianBushfiresFeb2009.kmz



And here is an example of kmz file showing the latest world earthquakes from USGS: http://www.aus-emaps.com/svs/ref/map.php?
kmzl=http://earthquake.usgs.gov/earthquakes/catalogs/eqs7day-age.kmz



Finally, a simple example of referencing Australian postal boundaries map from aus-emaps.com:
http://www.aus-emaps.com/svs/ref/map.php?
kmzl=2601,2220,7000,3122,5000&svs=2,2,2,2,2



Width and height of the map can be easily controlled by adding the following parameter to the URL: &wh=500,400 (size in pixels). Map type selection is with &mt=0 parameter (valid numbers are 0 for street map, 1 for satellite overlay, 2 for hybrid map and 3 for terrain map). This is just a quick demo of basic capabilities to give you a taste of what is possible with a very basic deployment of Google Map API (this one is still running only on version 2). More on this service soon.