Archive

Posts Tagged ‘osm’

OSM user history heatmap

January 16, 2011 1 comment

In the past few months I’ve been contributing to the OpenStreetMap database. From the day I started I’ve had privacy concerns on my mind. If a criminal goes around committing crimes, leaving behind a signature and always travelling x km from home to commit the crime, you get a nice circle around their home. So I’m interested to see what kind of pattern I’m leaving behind. Someone else has probably got an existing solution out there, but I decided to experiment and put together something myself.

I already had all my osmChange files (i.e. the ones from http://www.openstreetmap.org/api/0.6/changeset/${changeset_id}/download). I pull these daily from http://api.openstreetmap.org/api/0.6/changesets?display_name=aharvey find the changeset ids and download the osmChange files for all my new changesets. I do this as I try to keep a local copy of things I upload into the cloud, as a backup in case the service I uploaded to ever decides to cut my access. I know I should really be saving a copy of what I upload, rather than uploading then asking the cloud server for a copy, but JOSM uploads a full osm file, not an osmChange file. Scripts here.

Anyway, from these osmChange files I pull out the coordinates of all the nodes I created or modified, script here.

From this list of points I could use any heat map program. gheat which is based on http://blog.corunet.com/the-definitive-heatmap/ looks good. The problem is I ran out of time setting it up– it wasn’t in the Debian repository. Instead I used http://www.sethoscope.net/heatmap/, which I combined with a wrapper script to render to tiles in the OSM Slippy Map format. However it is really slow– perhaps I should try to speed it up.

Now that I had the transparent heat map tiles, the next step was to get these on top of an OpenStreetMap map. Given they are in the standard format, it should be really easy to add this layer to any software that lets you view OSM tiles. Support for overlays like this in libchamplain is on my todo list, in the meantime I gave Polymaps a try, HTML with JavaScript embedded here.

Heat map image of my created or modified nodes in OpenStreetMap

Heat map image of my created or modified nodes in OpenStreetMap

The results don’t come as much of a surprise, I already knew which areas I edit and how much. This map is also biased towards nodes, and doesn’t take into account editing an existing way or relation. One surprise was some cosmetic work along creeks shows up as nice lines (tracing a creek adds a lot more nodes than adding a single toilet node)…

Postscript

While I was working on this I once again needed a bbox string to test with. This has come up a lot in the past and I didn’t know any easily way to use a GUI to position the map, and then just copy and paste the bounding box of that view. So I made a modification to one of the libchamplain demos to return a bounding box from the current view.

Advertisements
Categories: Uncategorized Tags: ,

Old Maps Online

November 28, 2010 5 comments

I should have made a post about this a while ago, but I didn’t want a half complete post, and the scope of my project kept expanding!

Part 1: Scraping

I found two huge repositories of old digitised maps of Australia, many of which are in the public domain. The National Library of Australia and Parish Maps from the Department of Lands NSW. Unfortunately they didn’t really have a nice documented RESTfull API for the use of the images and metadata. My first step was to extract as much information as I could and convert it into an intermediate format. Most of my code and documentation for doing this is at https://github.com/andrewharvey/govscrape in those two respective folders. Unfortunately it’s not as easy as running one command from my repo to download and parse all the data. My goal was to get the data to my machine, not write a robust system that anyone could run to get a clone of the nla and pmap repositories.

Part 2: Georeferencing

It would be great if I could push out an easy to use API for the data I collected from the scrape stage, but I don’t have the resources (let me know if you are willing to help out with server resources to host these old public domain maps). Even without a nice interface to the data, I could still play around with it and to see what use I could make of it. I dabbled into using these maps as a source of data for OpenStreetMap. I only got through a few of the maps, I put this on hold as I figured it would be easier (especially for others) to do this if they were georeferenced. I tried out both http://warper.geothings.net/ and QuantumGIS, but both had way to much lagging. So I rolled out my own solution which was just a bunch of scripts which used Inkscape and a hacked libchamplain demo as the GUI. The code and documentation for this is at https://github.com/andrewharvey/georeferencing-scripts.

The georeferencing data that I have made so far (it’s a big task!) is at https://github.com/andrewharvey/georeferencing-data.

Part 3: Sharing

From the data and code from the last step, I’m able to push out these old maps in several formats. I used gdalwarp to convert the maps into Transverse Mercator (well actually I don’t really know what they are, but this seems to work), from here I can use gdal2tiles.py (…finally understanding the difference between OSM Slippy map tilesnames and the OGC TMS… take note that gdal2tiles.py produces TMS format tiles which differs from OSM style as it has the y axis going bottom to top, see http://groups.google.com/group/maptiler/browse_thread/thread/aa89fc726b8f7261/8bdc39d7829cc80c) to push out an OSM slippy map like tile directory, I can push out a KML GroundOverlay, or you could probably use a WMS server to push it out through WMS. I really wanted to leave it open.

Overlay from public domain map, http://nla.gov.au/nla.map-rm2795. Background CC BY-SA 2.0 OpenStreetMap Contributors, http://www.openstreetmap.org/

Parishmap as backgrop in JOSM.

Parishmap as backgrop in JOSM. Data CC BY-SA 2.0 OpenStreetMap Contributors, http://www.openstreetmap.org/. Background public domain map PMapMN04/14015601.

Parish map overlaid on OSM mapnik layer.

Overlay from public domain map, PMapMN04/14015601. Background CC BY-SA 2.0 OpenStreetMap Contributors, http://www.openstreetmap.org/

I would post a Google Earth one too, but its too much effort to get a free background in there for the screenshot. I’m not convinced that this display of the data is user friendly. Having control of the transparency of the overlay is a must. Maybe one day, someone will crop out all the non-map parts of the parish maps so we can get a single whole of NSW parish map slippy map.

I suppose now I need to focus on the infrastructure. It should be really easy for a user to browse the available maps and view them either as a KML, an OpenLayers overlay. I should also plug this into the meta-data I scraped and have stored in CSV like files.

The problem I have with distribution right now is that many of the maps need warping and that means I need to host the warped image somewhere. Some could probably be georeferenced from their source image using just translate, scale and rotate, and hence should be able to use the source image from the government server to serve the georeferenced imagery. But the work flow I’ve set up so far, relies on using gdalwarp, and hence having access to the warped image.

Categories: Uncategorized Tags: , ,

Slippy Map Tilenames For Non-North [Isometric] Web Maps

November 28, 2010 1 comment

I made this image to help my understand a slippy map tile naming system for maps that don’t always point North (it turns out nothing changes in terms of the tile numbering, you just have to apply additional view space transformations). This is how Nearmap does it, and when I get around to putting up http://osm.kyblsoft.cz/3dmapa/ like tiles of Sydney I will endeavor to use the same system (in other words you put your code to view the other views in your map viewer application, rather than just change the tile numbering so that you can use existing code for all the views). It doesn’t really change anything here if we have square or non-square tiles, the tile numbers and true coordinates don’t change because of this. Keep in mind that all the points of tile z/y/x for any of these views will be the geographic location, the views just have a different view space translation.

Tile names for Noth, South, East and West views

Diagram CC BY-SA. Map tiles CC BY-SA based on CC BY-SA data from http://www.openstreetmap.org.

see also: http://wiki.openstreetmap.org/wiki/Slippy_map_tilenames

Categories: Uncategorized Tags: ,

Commentary on NearMap’s Licenses

November 12, 2010 3 comments

I’ve taken some time to look at the NearMap licenses (Community License, Free Commercial License) more closely. Here is some of my commentary of them. I’ll use the terminology used in the licences so please excuse me for using their language and jargon. When I say free I mean free as in “free software”, “free culture” and “free as in freedom”. Also this is just my interpretation from reading them, I am not a lawyer. Hopefully I’ve interpreted them as NearMap intended.

Derived Works

The licenses create a distinction between a PhotoMap or modified PhotoMap, and a work made using information derived or observed from the PhotoMaps. So they separate out works like vector information about roads derived or observed from their PhotoMap from other derived works (what they call modified works) like someone photoshoping fake roads into the imagery. My preconceptions of the Australian Copyright Act were that both of these things are “derived works” which are subject to the copyright of the original work, but despite this I really like the separation made in NearMap’s licenses here. I do think there is a difference and whatever their motives, they are essentially saying “information is free” so you must distribute any information you observe under the CC BY-SA license. Of course if information and facts were really free then the copyright law would say so (and you wouldn’t have to be in doubt due to the Telstra phonebook cases) and you wouldn’t need to add your defence of this freedom via the SA clause… to quote Nina Paley,

“ShareAlike is an imperfect solution to copyright restrictions, as it imposes one restriction of its own: a restriction against imposing any further restrictions. It’s an attempt to use copyright against itself. As long as we live in a world wherein everything is copyrighted by default, I will use ShareAlike or some other Copyleft equivalent to attempt to maintain a “copyright-free zone” around my works. In a better world, there would be no automatic copyright and thus no need for me to use any license at all. Should that Utopia come about, I will remove all licenses from all my work. Meanwhile I attempt to limit other peoples’ freedom to limit other peoples’ freedom.”

http://questioncopyright.org/CC-branding-confusion

Another important observation that I previously overlooked is the fact that,

“You will own all Derived Works that you create. However, you may only distribute Derived Works to others on the terms of a Creative Commons Attribution Share Alike (CC-BY-SA) licence (and you may use any version of that licence you wish, whether localised for a particular country or not). For example, you may Use the Licensed PhotoMaps or Modified PhotoMaps to obtain information which you can then use, under the Creative Commons Attribution Share Alike (CC-BY-SA) licence, to populate or update community street mapping projects.”

http://www.nearmap.com/products/community-licence#clause5

Previously I had thought that any derived works that came from NearMap PhotoMaps used in OpenStreetMap needed to be attributed to NearMap. I guess I just incorrectly thought that all the information was CC BY-SA licensed by NearMap, but that is not the case. These works actually need to be attributed to the user who observed that information and turned it into a work by, for example, adding it into the OSM database. The works do not need to be attributed in any way to NearMap.1 This also means that any copyright that arises from any creativity in deciding what to trace, and any copyright that arises from the tracing being a derivative work can be treated as CC BY-SA licensed by that person or user. That person is the copyright holder but they are only allowed to distribute the work under a CC BY-SA license. This is a good thing! I’m glad that NearMap have not chosen to change the wording to make it compatible with the public-domain-like OpenStreetMap contributor terms, as (unless of course one has some other license from NearMap) it guarantees that this information remains free2.

The discussion and use of Yahoo imagery as a source of tracing for OpenStreetMap was before my time, but from http://wiki.openstreetmap.org/wiki/Yahoo it seems that strong legal foundations are lacking. In this respect I feel much safer tracing from NearMap. I know that my contributions can be licensed under the free CC BY-SA license and nothing can be done to unfree these works. Whereas the legalities of Yahoo imagery is, at least from my reading, very questionable and potentially a huge problem in the future.

Modified Works

Another reason I took a closer look at the licenses was to make sure that the works I posted earlier which were modified PhotoMaps complied with all licensing requirements, both the NearMap and OpenStreetMap requirements. I have come to the conclusion that unfortunately I am most likely unable to satisfy both the Mapnik share-alike requirement and the NearMap share-alike requirement in coexistence.

The NearMap “free community licence” gives me the “right to use, copy, modify and distribute our PhotoMaps”. The distribution to others clause says that I must give NearMap attribution for the distribution of any original or modified PhotoMaps, however the license also says “You may sublicense your rights to the Licensed PhotoMaps, Modified PhotoMaps or APIs to others on the same terms as this licence or our free commercial licence.” I interpret this as if I modify a PhotoMap I need release it under the “NearMap free community license”, that is it is share-alike.

On the other hand though, I also used the default OpenStreetMap Mapnik-style map images. My understanding is that like the data used to create these maps, the actual map images are copyrighted by all the OpenStreetMap contributors and released under the CC BY-SA license. The share-alike means that any derivative works (like overlaying NearMap terrain maps) must the released under a CC BY-SA compatible license, so you cannot impose non-commercial or non-government on it. However although the NearMap free community licence plus the NearMap free commercial license almost allow anyone to use or modify the work they don’t meet CC BY-SA because they exclude government and exclude commercial use made in a “competing manner” and use that is “material to their business”. This leaves me to believe that I cannot legally distribute any work that is a mash-up of OSM data/maps and NearMap PhotoMaps. Unless of course that it is only the default Mapnik tiles that are CC BY-SA, and that anyone can copyright map images made from OSM data. Because NearMap uses OSM data to create Mapnik tiles using their own map style. I assume then that it is only the OSM data that is CC BY-SA and someone is free to make a non-free map using their own style from this data. Then they would own the copyright to that map and hence you would be free to combine this with NearMap’s PhotoMaps and release the product under their free community license. This could also explain why NearMap can overlay their transparent tiles based on OSM data over their non CC BY-SA imagery.

It is a shame, but I can totally understand NearMap restricting use of their PhotoMaps in a specific field of endeavor, namely the government. The government is central to their current revenue stream, without it they probably could not produce the volume of work they currently do under their almost free, community license. It is almost CC BY-SA, except they exclude three fields of endeavor, “Competing Manner”, “Material to their business” and “Government Entities that use our PhotoMaps for their own governmental purposes”. The first two exclusions make the PhotoMaps near CC BY-NC-SA, but the last clause means they cannot be compatible with any of the Creative Commons licenses.

Licensed PhotoMaps

Let me use the example case of distribution of original NearMap PhotoMaps. For instance say I download a bunch of imagery tiles and distribute them through BitTorrent, the key question here is do I need to enforce that this distribution is to non-government entities. If I am only allowed to distribute it to non-government, I cannot do that, so the freedoms that the license grants are not as broad as I thought. If on the other hand if I can distribute the works to government entities along with the free community license as a LICENSE file, but leave the responsibility and liability on the government to not use the works I make available, then this would be much better. Hopefully the latter is the case. This was almost touched on here, but which party the liability lies on was not mentioned.

Termination

This is why I hate reading all this legal jargon, every word is important but has different interpretations. Code on the other hand has just one interpretation, and that is defined in the compiler… Anyway, at first I thought this termination clause meant NearMap could terminate the license grant at any time, however I missed the words “if the other party breaches this licence”. I view this to mean that NearMap cannot terminate the license grant unless you breach the license. But even if such a case arose, derived information is safe. So NearMap can do nothing to prevent the CC BY-SA distribution of derived information. Although it appears all the other parts of the license grant can be subject to this clause.

1 However I still think that one should attribute NearMap regardless. In the OSM case, attribution using the source tag should be done for other reasons as well; like so people know where the data came from, hinting some clues of the quality of the data.

2 Without turning this into a copyleft v permissive debate, I think that CC BY-SA is the most common license in use (there is a whole OSM database under it), so to avoid incompatibilities with other works under different share-alike licenses, CC BY-SA is probably the best choice at the moment. Some people say CC BY-SA is not for data, this is true, but that doesn’t stop the Australian government releasing data under the license, but more to the point, CC BY-SA has a clause that allows derivative works to also be licensed under a “Creative Commons Compatible License” that is listed at http://creativecommons.org/compatiblelicenses. One would hope that if someone creates an attribution share-alike license suited for data that would protect that data in jurisdictions that copyright data, and leave it free to use without restriction in jurisdictions that don’t copyright data (so don’t try to force attribution share-alike in these jurisdictions with contracts), Creative Commons could add this as a Creative Commons Compatible License, and all works like the current CC BY-SA OSM database could be relicensed under that data attribution share-alike license.

Categories: Uncategorized Tags: , , ,

A Mashup of Nearmap Shaded Relief Terrain Maps with OpenStreetMap Mapnik Tiles

October 28, 2010 9 comments

Some eye candy from my experiments of combining Nearmap terrain tiles with OpenStreetMap tiles. These images (which are hyperlinked to full sized ones) are derived works from OpenStreetMap.org default mapnik tiles (which is based on data from OpenStreetMap contributors licensed CC BY-SA 2.0), and Nearmap.com shaded relief terrain maps licensed under the Nearmap Community License.

Nearmap Terrain Maps with OpenStreetMap Mapnik Map

Nearmap Terrain Maps with OpenStreetMap Mapnik Map. Based on images (c) OpenStreetMap contributors, CC-BY-SA, and (c) Nearmap, Nearmap Community License.

Nearmap Terrain Maps with OpenStreetMap Mapnik Map.

Nearmap Terrain Maps with OpenStreetMap Mapnik Map. Based on images (c) OpenStreetMap contributors, CC-BY-SA, and (c) Nearmap, Nearmap Community License.

Nearmap Terrain Maps with OpenStreetMap Mapnik Map.

Nearmap Terrain Maps with OpenStreetMap Mapnik Map. Based on images (c) OpenStreetMap contributors, CC-BY-SA, and (c) Nearmap, Nearmap Community License.

Nearmap Terrain Maps with OpenStreetMap Mapnik Map.

Nearmap Terrain Maps with OpenStreetMap Mapnik Map. Based on images (c) OpenStreetMap contributors, CC-BY-SA, and (c) Nearmap, Nearmap Community License.

Nearmap Terrain Maps with OpenStreetMap Mapnik Map.

Nearmap Terrain Maps with OpenStreetMap Mapnik Map. Based on images (c) OpenStreetMap contributors, CC-BY-SA, and (c) Nearmap, Nearmap Community License.

My approach is based on http://wiki.openstreetmap.org/wiki/TopOSM/Details. I used two mapnik stylesheets, one for everything minus the labels, and one with just the labels. From here I used this bash script,

for f in nearmap-dem/*/*/*
do
f=`echo $f|sed 's/[^\/]*\///'` #get rid of the nearmap-dem part
d=`dirname $f`
mkdir -p "grayshaded/$d"
#convert "nearmap-dem/$f" -colorspace gray "grayshaded/$f" #just convert to gray
#...alternatively, extract the Lightness channel...
convert "nearmap-dem/$f" -separate -channel Lightness -colorspace gray "grayshaded/$f"
lastdir=`dirname "grayshaded/$f"`
lastfile=`basename "grayshaded/$f" .png`
rm -f "$lastdir/$lastfile-0.png" "$lastdir/$lastfile-2.png"
mv "$lastdir/$lastfile-1.png" "$lastdir/$lastfile.png"
done

to convert a directory of Nearmap Terrain tiles into something suitable to be passed in as the hillshade layer to the combine script at http://wiki.openstreetmap.org/wiki/TopOSM/Details#Combining_images_into_a_final_composite, and then I used combine script to merge the layers together for each tile.

Update: The Mapnik stylesheets I used for labels and nolabels are at http://gist.github.com/653184

Categories: Uncategorized Tags: , ,

Geoscience Australia

October 25, 2010 Leave a comment

So a while back Geoscience Australia stuck this notice on their website,

Unless otherwise noted, all Geoscience Australia material on this website is licensed under the Creative Commons Attribution 3.0 Australia Licence.

Great, another source of free government data. So today when I find a use for some data held on their site I was very dissapointed to see,

“Please note: Any organisation or individual wanting to use the Gazetteer data in a similar capacity to the Online Place Name Search or any other online application, will require an Internet user licence. (See the Licence Fees and Order Form below).”

So much for the CC-BY license. The latter notice is an “otherwise noted” which says, this data is NOT CC-BY because if you want to use it for such an such purpose you need to get another license to allow you to do that.

Categories: Uncategorized Tags: , ,

Some experiments with OpenStreetMap Overlays and NearMap Oblique Views

October 11, 2010 Leave a comment

…I was thinking, short of having a high resolution accurate computer model of the world (i.e. digital terrain model with an orthophoto on the terrain + accurate 3D models of buildings, bridges, trees, etc), one can use these “oblique” aerial views to similate other low angle views.

 

Sydney, North Oblique View. Tiles from NearMap. Used under the http://www.nearmap.com/products/community-licence license.

Sydney North Oblique view. Image tiles from NearMap. Used under the http://www.nearmap.com/products/community-licence license.

 

What I mean is that Google Earth uses orthophotos as the ground texture. For some places they also have rough building models and these make low angles (ie. when you are not looking straight down towards the earth’s surface) in Google Earth look a little bit more real. An alternative, when we lack a high resolution accurate model, for an interactive Google Earth like application is to use overhead orthorectified imagery when the user is looking from above, and use the oblique views (called MultiView on NearMap) when the angle between the direction of view and the ellipsoid surface normal gets larger. For a 2D web interface when the user changes from the overhead imagery to the oblqiue views, the imagery could fade from one to the other.

 

Sydney North Oblqiue viewed in Google Earth from a lower angle (not straight above). Image tiles from NearMap. Used under the http://www.nearmap.com/products/community-licence license.

Sydney North Oblique viewed in Google Earth from a lower angle (not straight above). Image tiles from NearMap. Used under the http://www.nearmap.com/products/community-licence license.

 

I hope to also get libchamplain to overlay transpanent map tiles on top of imagery (more on that in the next post), because I really think that oblique views make map reading much easier for the user (although the South, East and West can be a little confusing when viewed at high zooms on a 2D map, just like looking at the world map upside down can be a little confusing at first you just have to think a little bit harder to work out what your looking at, see South and East views further below).

 

Sydney North Oblique Imagery from NearMap with OpenStreetMap tiles on top. OSM tiles and data CC-BY-SA 2.0 http://www.openstreetmap.org/, NearMap imagery http://www.nearmap.com/products/community-licence.

Sydney North Oblique Imagery from NearMap with OpenStreetMap tiles on top. OSM tiles and data CC-BY-SA 2.0 http://www.openstreetmap.org/, NearMap imagery http://www.nearmap.com/products/community-licence.

 

 

North Oblique Imagery from NearMap with OpenStreetMap tiles on top. OSM tiles and data CC-BY-SA 2.0 http://www.openstreetmap.org/, NearMap imagery http://www.nearmap.com/products/community-licence.

North Oblique Imagery from NearMap with OpenStreetMap tiles on top. OSM tiles and data CC-BY-SA 2.0 http://www.openstreetmap.org/, NearMap imagery http://www.nearmap.com/products/community-licence.

 

 

Sydney South Oblique View. Image tiles from NearMap. Used under the http://www.nearmap.com/products/community-licence license.

 

 

Sydney East Oblique View. Image tiles from NearMap. Used under the http://www.nearmap.com/products/community-licence license.

Sydney East Oblique View. Image tiles from NearMap. Used under the http://www.nearmap.com/products/community-licence license.

 

Soon I’ll post up some stuff about NearMap in libchamplain, and my plans to set up some kind of object recognition to find objects like zebra crossings, the yellow school zone rectangle, cars… from imagery for integration into OpenStreetMap.

Categories: Uncategorized Tags: , ,