Wednesday, September 29, 2010

NSW traffic conditions map

Road Traffic Authority (RTA) in NSW has stared publishing up to the minute information about traffic conditions in Sydney as well as in NSW regional areas. The coverage and extent of information is quite comprehensive and it is served in two formats to suit individual preferences: as an interactive map with continuously updated information, and as a text list (with sort function to enable arranging information according to update time, type of incident or its location). A separate window lists 67 live traffic cameras strategically located across Sydney's road network, with images refreshed every few minutes.

I find map option the most informative as it displays all the data in geographic context. Information on the map is refreshed automatically with update time clearly marked. Auto update can be turned off if required and users can customise the list of information items for display on the map.

Live updates include information on accidents, scheduled road works, major events, fires, floods and Alpine conditions. There is an option to add traffic flow overlay for Sydney provided by Google as well as locations of live traffic cameras and temporary message signs.

Journey planner, utilising Google driving directions script, is part of the application. The map comes with a comprehensive list of points of interests that are very handy for planning a trip. These include: bike parking areas, heavy vehicles checking stations and rest areas, locations of Safe-T-Cams, general rest areas and RTA Motor Registries. Travel routes can be filtered to avoid incidents or road works on specific dates and can be saved for future reference.

RTA provides RSS feeds of the information on region by region basis. However, despite the push by Federal and State governments to liberate access to data generated by their agencies under creative commons licence, RTA is not letting developers to use its information for value added services and derived products. It is strictly for personal use although, embedding of the entire application in third party websites is allowed. A comprehensive widget customisation tool is provided as a part of the application, with easy to follow instructions.

Overall, this is quite a comprehensive information source, with attractive multi tab design and extensive range of features. The mapping functionality of the application is built with Google Map v2. Mobile version would be a great extension.

Tuesday, September 21, 2010

Australian Postcodes User Guide

There is a significant level of interest in postcodes as a convenient reference to locations because of perceived ease of linking them to information about individuals and businesses alike.

Over the years postcodes have been put to a wide range of uses in analysing and publishing social trends and population statistics as well as in defining sales, service, franchise or dealership areas.

Unfortunately, a misunderstanding of what postcode really is, resulting from a widely held belief about its value as a uniform referencing system, can cause many troubles for the unwary users. This article is a guide for all potential users of postcode boundary data.

Postcode Basics

Firstly, some facts about postcodes, from Australia Post site:

  • Postcodes were introduced in 1967 to facilitate the efficient processing and delivery of mail to customers.
  • Postcodes are only allocated to localities officially gazetted by State land agencies (usually, a postcode covers an area comprising of more than one locality).
  • The decision as to whether a new postcode or an existing postcode is to be allocated to a locality is based on operational efficiency.
  • Because the adoption of new or changed postcodes by customers is slow, changes are only made where significant reasons for change are established. A postcode change will only be considered if such a change leads to either enhanced service to Australia Post customers or operational efficiency to the organisation. Any such change will involve consultation with the local council/shire and residents.
Please note, the above holds true most of the time... but there are exceptions. It is also important to note that there are 3 types of postcodes: delivery areas, post office boxes and large volume receiver. Only delivery areas have meaningful reference to locations “on the ground”.


Sources of Postcode Information

Australia Post publishes a list of all postcodes from its database as a comma delimited text file. The list is updated every month and can be downloaded for free from Australia Post website.

Australian Bureau of Statistics publishes in 5 year intervals a set of Postal Area boundaries that are compiled using outlines of Census Collection Districts. They approximate official Australia Post postcode coverage areas at the time of publishing. These boundaries are available for free download in a range of popular GIS data formats. The next update of the data will be released in December 2010.

A number of private companies also produce and regularly update their own versions of postcode boundaries which are available for purchase. The two major suppliers include MapData Sciences (currently ESRI Australia) and Pitney Bowes (formerly MapInfo Australia).

There is also a number of smaller operators that may be a source of free or inexpensive information on postcodes, such as aus-emaps.com which genaralises and converts ABS postcode boundaries to KML format for use with Google Map and Google Earth and supplies large format static maps in PDF format for printing.

Postcode Finder app by aus-emaps.com


Common Problems with Postcodes

1. Changing Postcodes
Postcodes are changing over time due to evolving operational requirements of Australia Post. Changes include additions of new postcode numbers and deletions of old ones from the list as well as adjustments to composition of postcodes by adding or removing localities. This is especially the case with new, dynamically growing areas as well as some rural locations and is less of the issue for established metropolitan areas.

It means that postcodes are not a stable spatial reference. It is ok to use them as a snapshot of a particular point in time, but what often happens is that the attribution to “what area constituted that postcode X years ago” is lost from the supporting documentation and important facts can be misinterpreted by future users of the information.

This is a real problem for researchers of social trends - those who insist on using postcodes as the main location reference. As well, it may cause some legal headaches if postcodes are referenced in contracts for supply of services or franchise areas, etc. Postcodes were never meant to be used in this fashion!

2. Changing definitions of localities
On top of changes that are undertaken from time to time by Australia Post, there are also changes to boundaries defining localities which are implemented by State and local authorities. What was locality X in 2007 may now be split into locality X and Y. As the result, it is very difficult to maintain timely and consistent reference of postcode numbers to “what is actually on the ground”.

3. Imperfect procedures of referencing postcodes to localities
Where possible, Australia Post references postcodes to officially gazetted localities but localities are determined by State land agencies and boundaries are recommended by local councils. This process is not coordinated from end to end and sometimes it gets out of sync. Take for example postcode 3478 in Victoria. Australia Post lists Medlyn as a locality included in this postcode (June 2010 edition) yet this locality is not on Victoria’s register of gazetted locations. Referencing postcode numbers to localities is not a science and there can be inconsistencies.


Recommendations

If you must use postcodes, please consider the limitations outlined earlier as well as the following recommendations to avoid potential problems:

If you intend to match postcodes to official ABS statistics:
  • Your only choice is ABS version of postcodes as it will ensure consistency of definitions (that is, postcode X in the data table will correspond to postcode X depicted as an outline on the map). It is particularly relevant for Census of Population and Housing data.
  • If you need to combine those statistics with your own data (eg. client records), geocode individual addresses and then reference them to specific postcode boundaries (eg. using GIS software with “intersect” function capabilities) rather than just rely on postcode component of the address to match the records to boundaries. It is the only way to ensure a particular address/location is part of that specific postcode area.

If you intend to use postcode outlines to define custom areas:
  • Again, ABS version of postcodes is the most cost effective option as it is a free dataset.
  • Define your custom areas once and put effort in maintenance of that dataset over time. You can adjust a composition of custom areas if required (eg. add/ subtract postcodes or even adjust boundaries – but only if topological consistency can be maintained - that is, if changes to the boundary of one polygon can be reflected in the adjoining polygons).
  • It is important to acknowledge that this dataset becomes de facto your own version and that compatibility with “source” postcodes and/ or statistics published on postal area basis may be lost over time.
  • Always reference version of postcodes used in any legal documents to avoid future ambiguity as to what constituted “that” postcode at “this” particular point in time.
  • As in the previous case, if you need to reference those postcode outlines to your own data, run geocoding and then reference individual records to specific boundaries and do not rely on postcode details in the address record alone to match data with boundaries.

If you are relying on postcode boundaries from commercial operators:
  • There is really no point in aiming to always have “the latest” version of boundaries representing postcodes. After all, these are not compatible with ABS statistics (unless the company can assure they reprocess those stats “somehow” to a new representation of boundaries) and besides, what is the benefit of constantly having to reprocess your own data to accurately reference it to the ever changing representation of postcode boundaries? The only exception would be if the company supplies some other unique data that is available exclusively with their proprietary version of boundaries.
  • Although companies claim to have “the latest”, these data are rarely updated on continuous basis (ie. every month), rather in 3 or 6 monthly intervals so, you are still getting “dated” product.
  • Don’t assume you will be able to reference your address records to “the latest boundaries” using only postcode number unless your address details and postcode boundaries refer to the same time period. In most cases they don’t and you cannot avoid geocoding and then running GIS “intersect” processing of data to ensure reliability of information.


In conclusion, although postcodes appear to be well recognised spatial units for referencing locations, the complexity associated with accurate delineation of postal boundaries greatly diminishes their usefulness.

If you can, avoid using postcodes! If you can’t, be aware of all the limitations, especially when drawing conclusions with far reaching consequences.


Related posts:
Australian postcode boundaries 2016
Sales Area Management Tool 2015 Upgrade
Sales Area Management Tool with Postcodes 
Map of Australian Postcodes 
State postcode maps  
Free postcode search widget
Post code maps and population statistics

Wednesday, August 4, 2010

It's election time!

Google has just released Australian Federal Election 2010 special. It is a Google Map application with overlays showing current Commonwealth Electoral Boundaries, shaded by political parties holding the seats. The information can be filtered based on the size of the margin the seat is held by. The map is accompanied by featured links to a number of related applications, such as Google Trends – showing election related search trends, or Student Voice 2010 – an unofficial voting tool for under-18, or YouTube channels with political party videos.

It’s a very colourful map! Although, I am a bit disappointed with the quality of boundaries presented on the map – very rough and overlapping on a closer zoom. It requires a bit more effort but so much more can be done with kmz overlays supported in Google Maps...

Thursday, July 15, 2010

Apple buys mapping company

News has just started circulating that Apple Inc. has purchased Poly9, Canadian mapping company. It developed a cross-browser, cross-platform 3D globe application, similar to Google Earth and Bing’s 3D globes, but which apparently "...did not require special plug-ins to work". The list of users included Skype and LinkedIn. The site is barely working now so I am unable to verify how good it was in comparison to Google and Bing but it appears it is build with Flash! So, it requires a plug-in after all. Also, it makes it quite an interesting development given Apple’s recent history of resentment towards Adobe Flash…

This is actually the second acquisition of a mapping technology by Apple Inc. within a period of 12 months. Last year it bought map group Placebase – developer of Pushpin, an API used to layer commercial and other data sets, such as demographics and crime data, onto maps. It appears that Apple is trying to develop its own mapping platform to eliminate reliance on its main rival – Google!


via eweek.com

Tuesday, July 13, 2010

Converting csv data into shapefile

In the inaugural post of the new Free GIS Tools series I presented Google Map API, a very powerful and functional alternative to open source and commercial online GIS software. Then I wrote about a handy tools for selecting colour scheme for thematic maps. Today a quick overview of csv to shapefile converter tool which can be very handy if you need to convert latitude and longitude coordinates in decimal degrees to point, line or polygon features for display in desktop GIS software or for further conversion to complex KML/KMZ. These could be geocoded locations of your clients, or locations derived from geoRSS feeds, outlines of features from online digitalisation tools (eg. using aus-emaps.com simple geocoder point coordinates tool), or similar.

CSV to Shapefile Converter is the best tool for such tasks and it can be downloaded from wanderingidea.com for free. It is a Windows based application and requires .NET 2.0 framework to run - quite a large download as a bundle but really worth the effort. Here are a few tips on how to use it:


  • Conversion steps are almost self-explanatory: open csv file, select shape type, specify which field is latitude and which is longitude, and which fields to use as data attributes - all in a single form - then press a button to execute.
  • CSV file you want to convert has to have a header row, otherwise you will lose the first record.
  • When creating a polygon, the last record relating to a particular polygon has to be identical as the first record in order to “close the loop”.
  • This converter supports complex shape structures like multi-part polygons (ie. many polygons with the same attribute) and doughnut type polygons – just need to remember a convention that in shp file format points for outer boundaries go clockwise and inner boundary points are listed in anti-clockwise order.
Here is an example of a csv input files to generate a multi-part polygon and a doughnut type polygon (the result of CSV to Shapefile conversion is depicted on the images below):

doughnut polygon:

Shp,Prt,Lat,Lon
1,1,-28.323724553546, 153.028564453125
1,1,-28.297125824492888, 153.49273681640625
1,1,-28.77488163974644, 153.5064697265625
1,1,-28.74117220459353, 153.10821533203125
1,1,-28.323724553546, 153.028564453125
1,2,-28.359984824037397, 153.12469482421875
1,2,-28.58693349906796, 153.1329345703125
1,2,-28.596580064634804, 153.43231201171875
1,2,-28.32130676215295, 153.39111328125
1,2,-28.359984824037397, 153.12469482421875


multi-part polygon:

Shp,Prt,Lat,Lon
1,1, -28.01584726281309, 153.39008331298828
1,1,-28.030015853768017, 153.38913917541504
1,1,-28.03024314216738, 153.40621948242188
1,1,-28.0140286990148, 153.4053611755371
1,1, -28.01584726281309, 153.39008331298828
1,2,-28.0140286990148, 153.41609001159668
1,2,-28.029182458863605, 153.41548919677734
1,2,-28.028955168224194, 153.43093872070312
1,2,-28.012740531077274, 153.4310245513916
1,2,-28.0140286990148, 153.41609001159668

More Handy Tools:
Free GIS Tools: Google Map
Free Address Validation Tool
Manual geocoder for 70 countries

For more useful tips please also see posts listed under “How to…” category.

Monday, July 12, 2010

Colours for thematic mapping

Thematic maps, more formally known as choropleth maps, are great for conveying information about differences between areas. Using shaded and/or patterned polygons makes it easier to visualise geographic distribution of specific attributes. But picking colours to show those differences can be a problem – the bigger the more classes the data is divided into. Colours are especially important when you work with orderable, numerical data and want to pinpoint hostpots (either concentration or deficiency) of certain characteristics. Coming up with 4-colour scheme is easy, 6 is a stretch but 10 is a real challenge!

For clarity of information, it is best to avoid more than 5 classes because with more classes the difference in shades becomes too subtle for a human eye to easily distinguish a “hierarchy”. It is especially the case when using thematic overlay in combination with a multi-colour background (eg. as a transparent overlay on Google Map). Dividing the data into meaningful classes is a science on its own and I will come back to it in a separate post. Today I just wanted to focus on colour selection and to introduce a free online tool that will help in choosing the right combination.


The most commonly used colours for thematic mapping are red, blue, green, and purple, with white, yellow or grey as a secondary, “lighter” colours in the mix. “ColorBrewer 2.0” is a great colour selection tool that allows to experiment with the most popular colour swatches in various combinations. It caters for 3 different data types: qualitative, sequential and diverging, and depending on the option, allows to specify up to 12 colour classes (as RGB, CMYK or HEX values). An interactive “map panel” allows previewing your selection with different transparency settings and background colours. ScoreCard feature provides a guide to useability aspect of the selected combination - in terms of potential issues for colour blind, B&W photocopying, colour-printing and/or displaying on LCD monitors.

To present “nominal/ qualitative data” which is un-orderable, non-numerical, the colour variation should not present a pattern and should be totally random. ColorBrewer allows do define up to 12 classes.
To present “sequential data”, like for example counts by certain characteristics, it is best to use shades of a single colour. ColorBrewer allows do define up to 9 classes. If you are desperate for 10 class scheme, use YlorRD variant with white as 10th class.

For “diverging data”, that is data that oscillates around a certain “standardised value” (like mean or median), it is best to use 2 diverging colours with a neutral midrange shade. Again, ColorBrewer goes only to 9 classes but if you require 10 classes, it is easy to combine 6 class contrasting colours from sequential option (just ignore the first lightest class and merge remaining 5 shades from each colour scheme for your customised 10 colour swatch).



A final word of caution if you are working with KLM file format. It is a common convention to use colour sequence as Red–Green-Blue, then Alpha channel for transparency. However, KML standard specifies everything in reverse: Alpha-B-G-R! For example, if hex colour code for blue is 0x045A8D, in KLM it has to be written as ff8D5A04 (where the first two digits indicate alpha transparency of 100%, ie non transparent). And to save you a headache or two while trying to work out the right codes for different transparency settings in KML, here is a list of the most common values:
  • 100% = ff
  • 90% = e5
  • 85% = d9
  • 80% = cc
  • 75% = bf
  • 70% = b2
  • 65% = a6
  • 50% = 7f
  • 30% = 4c
And if you want to explore unlimited gamut of colours to define your totally custom swatches, try these tools as well:

http://www.sgrillo.net/kml_color/
http://www.zonums.com/gmaps/kml_color



Related Free GIS Tools posts:
Google Map API
Address Validation Tool
Simple Geocoder

Monday, July 5, 2010

Arts on the Map

Lovers of Australian art can now search interactively for… Arts on the Map. This map centric website was published by the Australian Council for the Arts and features nearly 200 companies currently funded by the Council. Visitors can use a very simple yet quite attractive interactive search widget to query a database of various artforms or art companies in each State and Territory. The results are displayed as a side list and interactive markers on the map. There is also an alternative search option for stories published by individual art companies. The mapping application comes in two flavours, as Google Map or Google Earth version.




It is good to see that Google Maps and similar free online mapping applications have a wide range of potential uses. The truth is that if it wasn’t for Google, there would be no such interactive applications on the Internet. Before Google launched its maps such functionality was reserved only for expensive and/or technically challenging software that many could not afford, or would never contemplate, to deploy on their websites.