East coast unanimously frees data

The NSW state government has joined QLD, Vic and ACT counterparts declaring support for open data initiative. It has just released a draft open data policy which aims to make state-owned data available to the public at no charge. Where is it not possible to provide data totally free of charge, it will be provided “at a reasonable price”. If the policy is approved, datasets will be available for download from www.data.nsw.gov.au portal.

If declarations of state and territory governments prove fruitful, we soon should have roads and cadastre/address data freely available for the eastern part of Australia - unless I totally misinterpret the intension of the open data policies. The move to free access to government data started 12 years ago with the release of Spatial Data Access and Pricing Policy however, the benefits have been slow coming. Hopefully, the momentum will now accelerate as more and more government organisations open up their data vaults.

Govhack 2013 initiative gives a taste of what can be done with all that free data. The winners have just been announced… 


First spotted on Spatialsource.com.au

Online map competition intensifies

There was a time that Google and its Map API was the only talk in town. Their revolutionary approach to serving spatial data online has broken the monopoly of specialist GIS companies on all things spatial. Many followed in Google footsteps with similar, simplified online products but were mostly overshadowed by the first mover in the field. For a moment it seemed that nothing else matters and that Google will dominate the market indefinitely. However, introduction of fees for high usage of Google Map API led to a big backlash against the company as many developers realised the vulnerability of their business to caprices of a key partner. Suddenly alternatives started to look very attractive - despite the limited functionality offered. And the competition in the online map market started to intensify again...

Acquisition of Waze by Facebook for a reported $1B is the flavour of the month (although not done deal yet). It demonstrates that the appetite of large companies for ready-made online mapping technologies is growing. Apple has its fingers firmly in the pie beefing up reliability of its offering (after a very unfortunate start) and optimising performance with vector graphics far beyond Google Map capabilities. Microsoft is surprisingly quiet…

The key enabler for “lower end” alternatives to Google Map was, and still is, Open Street Map project which provided developers with roads dataset all over the world (traditionally, the cost of using that data was so restrictive that only those with very deep pockets could attempt to do it on a world scale). CloudMade and MapQuest offered free access to their versions of OSM data and started building a variety of free services, enabling replicating key functionality of Google Map, such as geocoding or navigation. ESRI has also thrown in their support releasing free map layers and their GeoCommons platform (although I am not sure if the current licensing restrictions still allow unconstrained use of that data).

There are many viable alternatives to Google Map and, as a developer or end user, you are no longer forced to accept strict conditions imposed by Google. By switching to any of those alternatives you will have much more flexibility in developing services and applications and will take back control of your business. And you don’t have to compromise on functionality, at least not too much… My favourite amongst free alternatives is Leafletjs (originally developed by CloudMade). It is easily extendible with free services from MapQuest. If you are after a 3D kind, Cesium from Analytical Graphics, Inc looks very promising - supports also 2D mapping but unfortunately, it works only in WebGL enabled browsers (by the way, NICTA built on this platform a very impressive range of spatial applications which demonstrate its full potential).

Security standard for developers

I have been writing quite a lot about the standards recently - mostly about their deficiencies -  but today a few words about one standard that is really worth investing the time and effort into:  Application security  ISO/IEC27034.

It has been around for about 18 months so, it’s a relatively new concept but already adapted by the largest software development companies. Chances are it will become prominent enough to be demanded from contractors developing solutions for government and larger corporate clients.

From the introduction to the Standard:

ISO/IEC 27034:
a) applies to the underlying software of an application and to contributing factors that impact its security, such as data, technology, application development life cycle processes, supporting processes and actors; and
b) applies to all sizes and all types of organizations (e.g. commercial enterprises, government agencies, non-profit organizations) exposed to risks associated with applications.

The purpose of ISO/IEC 27034 is to assist organizations in integrating security seamlessly throughout the life cycle of their applications by:
   
• providing concepts, principles, frameworks, components and processes;

• providing process-oriented mechanisms for establishing security requirements, assessing security risks, assigning a Targeted Level of Trust and selecting corresponding security controls and verification measures;

• providing guidelines for establishing acceptance criteria to organizations outsourcing the development or operation of applications, and for organizations purchasing from third-party applications;

• providing process-oriented mechanisms for determining, generating and collecting the evidence needed to demonstrate that their applications can be used securely under a defined environment;

• supporting the general concepts specified in ISO/IEC 27001 and assisting with the satisfactory implementation of information security based on a risk management approach; and other standards.

A good starting point to learn about secure coding practices is a series of free courses/ video presentations offered by The Software Assurance Forum for Excellence in Code (SAFECode). You can read more about the Standard in a very interesting article by Lia Timson (The Age): If it's worth coding, it's worth securing.

Rift in GIS ranks

There is a troubling development in GIS community that threatens to flare up old animosities and split participants along the lines of big versus small as well as proprietary versus open source. In the centre of it are Open Geospatial Consortium (OGC) and its members, representing open source camp, minor proprietary solutions vendors and 800 pound gorilla in GIS – ESRI. Ultimately, the whole incident can undermine, one way or another, OGC stronghold on all things spatial…

Firstly, a short history of OGC, from my perspective. In the beginning OGC was just an obscure “public benefit” organisation focused on developing and promoting open and interoperable standards to facilitate exchange of spatial information. There was a real problem that required urgent attention – that is, finding an easy way to exchange vast amounts of valuable spatial data stored in “walled gardens” and in obscure proprietary formats.  So, many were prepared to put a lot of effort in their spare as well as “employer paid time” in order to come up with a solution to this problem. This early period attracted a lot of prominent participants from government organisations and academia across the world, as well as support (often financial) from several industry sectors hoping for significant cost savings in the future. It was a community wide initiative with roots firmly established in an open source movement.

When OGC standards started gaining traction and wider acceptance amongst key buyers of GIS technologies the organisation attracted to the cause members from a proprietary solutions camp as well. OGC reached then the peak of its relevance.  A new era started where a united GIS community worked together towards a common goal – achieving full interoperability of spatial data, regardless whether data originated in proprietary or open source formats, or using proprietary or open source tools.

However, the relevance of OGC and its initiatives is under threat now - from outsiders as rapid technological advancement demands more flexible, easier to implement and performance focused standards (like those developed by organisations and individuals operating on fringes of GIS  - Open Street Map initiative and GeoJSON are the two most prominent examples), but also from within its own ranks as the organisation is forced to deal with conflicting interests of its members.

In particular, ESRI had put forward a new GeoSpatial REST API standard for endorsement by OGC. It covers a wide set of web services including catalogue, map, feature, imagery, geometry, geoprocessing, and geocoding web services.  However, a significant number of OGC members lodged an objection to it and demand the documentation not to be adopted as an official OGC standard.

The essence of the objection is that the proposed standard is not a community designed scheme but rather just a documentation of an implementation of proprietary web services by a single commercial vendor.  No other vendors will be able to build solutions against that standard, and if endorsed, it will dilute relevance of existing OGC standards.  When it becomes an OGC endorsed standard there is a danger that buyers will demand that all the vendors of spatial solutions support that standard in their products, hence giving an unfair advantage in the market to vendors of ESRI solutions. The objections are fully documented on OSGeo website.

Unless a workable compromise can be found, then regardless whichever way OGC decision goes it will alienate a significant group of its members and ultimately, it will split GIS community. That is, it will upset either open source and non-ESRI vendors or ESRI and thousands of its affiliates whose livelihood is almost exclusively dependent on ESRI’s fortunes.  If things get rough ESRI is dominant enough in the market to go its own way, without the blessing from OGC. But will OGC be able to retain its relevance without such a prominent member as ESRI? Potentially some troubling times ahead for GIS industry...

[Update: ESRI has withdrawn the GeoSpatial REST API standard from OGC, potentially defusing the situation. ]

Geoscience Australia new catalogue

Geoscience Australia has just released a new, map based catalogue to assist visitors to the site in the discovery and access to a diverse range of data, products and services available from the organisation.  Discovery and Delivery System is in beta release for public testing so, unfortunately not everything is working properly in Internet Explorer. However, you will still be able to find a lot of goodies there like, for example, over 100,000 historic Landsat-7 and Landsat-5 images over Australia captured between 2000 and 2010. These can be viewed as WMS overlays on the map or can be downloaded as data (in NetCDF format, but be careful, each scene is more than a gigabyte in size).

Mapping functionality of the catalogue is built with Google Map API. Backdrop map is a standard satellite overlay but users have a choice of several custom layers provided by GA, such as topographic map, land cover map, geology map or gravity map. 

The catalogue lists publications as well as data for use with GIS software. Advanced search option allows refining search criteria by specific theme, geographic location and time frame. Links provided in search results take users directly to a download page. The catalogue is a convenient way to discover what is available from Geoscience Australia although, at first, the choice may appear a bit overwhelming!

Related Posts:
Australian floods mapped
Google unlocks Landsat archives
Landslides in Australia
Continental reference image

Maps for Excel

The idea of integrating maps and spreadsheets is not new - both Microsoft Excel and Lotus 123 (for those who remember!) had maps embedded as far back as 1990’s. Yet, the usage of those maps never gained widespread popularity amongst spreadsheet users. On the other hand those who really needed to map their data were prepared to pay good money for the functionality. Some integrators did very well out of this, selling their plug-ins for up to $10,000 a user. Now Microsoft is making another attempt to integrate Excel and maps in 2013 version of the product. Microsoft is positioning this as part of its BI solution. Reportedly, it will be able to handle “big data” to the tune of "one million rows of data from an Excel workbook." The new functionality will allow users to geocode data as well as view geospatial and temporal data by creating thematic and heat maps.

[image courtesy of Excel Blog]

Google is already offering integrated spreadsheet, fusion tables and map solution through their Docs application so it is just a catch up for Microsoft. If history is anything to judge by, mapping data in spreadsheets will not become a mainstream use of Excel – simply because users will need to geocode all those “millions of records” in the first place - no indication so far as to how much this service will cost. Also, all those who deal with geocoding records know that it is not as simple as “pressing a button”. Not to mention that challenges with “processing of spatial data for 1 million points” have not been quite solved yet without a quite complex server setup so chances of doing anything useful with such vast amount of information, even within a desktop version of Bing Map, appear rather limited (in fact, some users have already commented about the performance). Spatial education within general population is another factor – general knowledge is increasing but is still not enough to deal with issues of “projections and datums” for spatial data and more advanced forms of analysis, beyond simple presentation of points on a map…

In my opinion Microsoft would have much better chances of success developing specific tools in support of specific business activities, with narrowly focused spatial functionality, rather than providing a generic DIY mapping capability. This strategy failed in the past but it seems to be a preferred approach by big players, including Google. Time will show if this time the outcome will be different.

Revisiting thematic map theory

There has been a long standing view that choropleth maps (from the Greek words choros - space, and pleth – value, or simply thematic maps) should not be used to present absolute values like, for example, counts of persons per postcode, unless mapped areas are of similar size. Ideally, only normalised values should be mapped (for example, proportions such as density of people per postcode area). People have been conditioned to perceive that “bigger means more” so, when we present information in a spatial content the effect of size of spatial units should be eliminated for a meaningful comparison.

For example, two postcodes may have exactly the same number of people residing in them, so both would be coloured the same way on an thematic map, but if one polygon is significantly larger than the other, it would give an impression of a greater importance (ie. larger size = more important). The argument goes that using ratios rather than absolute values eliminates the influence of area, so that the map becomes meaningful by portraying accurately the distribution of features within each area. In our example, if we use ratio of counts to area size, larger polygon would have a smaller density value hance would be coloured in a lighter shade, “demoting” the importance of that area in relation to the other.

[Source: Wikipedia]

There are all sorts of other concerns about inadequacy of thematic mapping for presentation of spatial information, including the more recent one that maps in Mercator projection, ie Google and other online maps, are particularly bad because they introduce extra distortion of areas the further you move from the equator. Not to mention there is a general concern that since users “get clues” as to hierarchical importance from both, the value attached to a polygon (shown with colour intensity) AND the size of respective polygon, they may not be able to interpret correctly the “hierarchy” of information being presented. That is, users are potentially unable to rank the order from “the smallest” to “the largest” because of conflicting messages presented on a map: large polygons with small absolute values (light colours) and small polygons with large absolute values (dark colours), or vice versa. Yet, many novice cartographers are either unaware of these issues or are totally ignoring “experts”, happily mapping absolute values on Google maps. 

However, as it turns out, there is some validity in this non-expert approach to thematic mapping. In particular, “…psychological studies have shown that size and color (or at least, size/value and size/hue) are a ‘separable’ combination: that is, variations in the size of graphic elements do not considerably interfere with our ability to determine their color [sequence], or vise-versa. So, theoretically, there should be no problem with using choropleths on a Mercator projection: the distorted areas shouldn’t mess up our ability to determine a region’s color, which is what choropleth mapping is all about.”

So, there you go – since only colour counts in thematic mapping and users are able to distinguish colour hierarchy independent of size of polygons,  it should ok to use absolute values with unequal polygons after all! Happy mapping!