Archive

Posts Tagged ‘osm’

A free and open source SRTM shaded relief and contour map

October 9, 2012 Leave a comment

I’ve recently pushed out a new srtm3-stylesheets repository which contains shell scripts for working with NASA SRTM DEM data, gdaldem based stylesheets for shaded relief maps, Mapnik stylesheets for contours and a TileStache configuration for sandwich those styles together into a single map.

This was spurred on by the fact that I simply needed a map which showed hills. I’ve used Andy Allan’s OpenCycleMap in the past which has color relief and contours. Unfortunately it is closed source.

So I put my head down and hacked together repeatable scripts to get the source data up and running and some basic stylesheets to produce a usable and pleasant looking map. All released as free and open source software under the CC0 license.

Contour Map

I want to avoid adding things like streets etc, such maps could be built as separated layers based upon this style and sandwiched together, for example, with the TileStache sandwich provider.

I hope to build upon the lessons learnt here to produce a map like the Stamen Terrain map, except with the source code released under a free and open source license. Perhaps just with hill and slope shading applied to landuse with other map features placed on top.

Contour Map

Contour Map

I’ve rendered NSW (only server resource prevent worldwide!) as a slippy map here.

Categories: Uncategorized Tags: , ,

Development Update (Static OSM Tiles, git rebase, Map Labelling)

December 11, 2011 Leave a comment

Map Labelling Suburbs and Cities

Yesterday I pushed a bunch of changes to my OSM/Aerial Imagery Hybrid Style, see the (demo). I spent a bit of time on suburb and city labels as they are a really important feature of a map aimed at non-experts.

I’m finding making a non-trivial style like this has defiantly made me realise the difficulties of such a task.

There are a lot of improvements that can be made (either by modifying or making new components) to the OSM Mapping conventions, osm2pgsql/imposm, mapnik library, carto language stack. That said, the current form is still great and you can still make great maps. But,

I would like to be able to but this needs
have labels for large bays at lower zooms than small bays need bays mapped as closed ways covering their area, rather than a point in the centre
define a linear function for the size of icons. i.e. at z10 the icon is 10px, at z20 the icon is 20px, now linearly interpolate all sizes for zooms in between either build this functionality into the carto language, or make another higher level macro like language which you can code this in which is then compiled into carto
render spread text inside a riverbank needs functionality in the mapnik rendering engine

Static OSM Tiles

I’m still yet to find a tile server which is fast and works well with lighttpd (nginx would probably suffice too). As an experiment I decided to pre-render a bunch of tiles for my hybrid style sheet. This tile layer doesn’t need to be minutely updated, anyone who needs that can use the normal mapnik layer. Also static tiles server straight from the webserver should be pretty fast (maybe on memcached tiles would be faster) and I wanted my tiles to be fast.

Next up how can I generate these static tiles? There is the popular generate_tiles.py, but that won’t render meta tiles, seems like such a waste to render every tile with a buffer of 128px when I would render a 5 by 5 meta tile of the same buffer for only 36% of the total pixels rendered. The larger the meta tiles the larger the latency, but if I’m pre-rendering them all than latency doesn’t matter any more.

So I wrote a C++ program as my replacement for generate_tiles.py. I also programmed it to render from a list of meta tiles rather than a bbox. This means for my demo I can only render high zooms where there is nearmap coverage. This is where https://github.com/andrewharvey/OSMTileListFromGeometry/ came in, which pulls nearmap coverage areas from an osm2pgsql database, and generates a list of meta tiles.

Using this method I rendered up to and including zoom 17, composed of 22444 5 by 5 meta tiles or 561100 regular tiles in a time of,

real    198m17.021s
user    103m49.985s
sys     37m36.117s

with disk usage,

<1M   0-8
2.0M  9
6.8M  10
26M   11
98M   12
13M   13
36M   14
114M  15
411M  16
1.5G  17
========
2.2G  total

(Updated with results when using “png” rather than “png256″)

real    228m17.082s
user    160m24.725s
sys     31m42.653s

with disk usage,

<1M   0-8
3.1M  9
8.6M  10
28M   11
121M  12
34M   13
83M   14
218M  15
663M  16
2.0G  17
========
3.1G total

I think that time could still be sped up with,

  • tuning of the mapnik stylesheet queries
  • tuning of the postgres database
  • spanning more rendering threads
  • using larger meta tiles
  • using a machine with more that 512MB of RAM
  • not running minutely updates in the background

Git Rebase

As I use git more and more I’m slowing learning more of the features it has to offer (and thanks to the free hosting by github). One such feature which I think is awesome is git rebase. As an example the other day I did git commit -ammend instead of git commit --amend (I knew there was a duplicate character somewhere but because I seem to have a tint of dyslexia I confused the duplicate — with mm).

Of course this resulted in committing all files which had changed with a message of “mend” as a new commit. I did this twice, and only noticed after I had already made a bunch of correct commits afterwards. With git rebase I could pop some commits of the commit stack, remove the two “mend” commits fix the commit which I should have been amended to and pop my other commits back on top of the stack.

Thanks to http://stackoverflow.com/a/180085.

Next Up

  • build a daily OSM planet extract by pushing minute-replicate files into the osmosis psql simple schema, then dumping to an OSM file
  • use this OSM file as a basis for monav and an OSRM service for fosm data
  • run a name finder service for fosm data
  • more updates to the hybrid map style
  • render the hybrid style tiles as oblique images for nearmap multiview
Categories: Uncategorized Tags: , ,

OSMT

October 9, 2011 Leave a comment

I feel the map style I’ve been working on recently is in good enough shape to announce. It is written in carto and designed to be used with OpenStreetMap data. It is an overlay layer for aerial imagery, though in my example I use NearMap imagery.

Take a look at http://tianjara.net/osmt.html.

The source is at https://github.com/andrewharvey/osm-hybrid-carto/, licensed CC-BY-SA.

There is still lots of work to do on it though.

I also have a lightweight cycle-overlay too (view example at http://tianjara.net/leaflet.html and tick on the cycle overlay or view the code at https://gist.github.com/1216315)

Categories: Uncategorized Tags: ,

All Roads Lead to Packaging for Debian

September 12, 2011 1 comment

Just a quick update to say that I have exhausted all my options to build a latest .osm file from fosm.org minute-replicate osc files + a base .osm file from osm.org. Currently fosm.org is down, but I still want to edit on top of the latest data so I can submit my changesets when it comes back online. This is proving to be more difficult that I thought. Although really it is a good thing as even though I’ve tried to ensure I have everything I need to rebuild fosm.org if it were to go down for good, I’m still not 100% sure as I haven’t tried, well now I get to try.

My first method was to use osmconvert with the aid of the workflow given in https://github.com/rrankin/osmconvert/blob/e1fbb7319f92f338e0023d110a3098f5a979fd64/update_osm.sh. It was quite fast and it almost worked, but due to limitation of osmconvert it seems that it expects the objects to be sorted in the order given by the object IDs. We can either patch osmconvert to work around this, or sort the osc files…

Putting the first method on hold, I gave installing the rails port that runs the API at osm.org a go with the idea that I can load the changesets or osc files directly into the API or DB. I got it installed and got to the step of loading in my starting OSM .osm file from before the FOSM fork. http://wiki.openstreetmap.org/wiki/The_Rails_Port#Populating_the_database even gives the command line to run, but osmosis didn’t like this. I believe the issue is fixed with later versions of osmosis though.

This leads me to require a later version of osmosis. I don’t really want to install osmosis from source using the upstream project’s method as it will pull in a bunch of 3rd party libraries using maven, so the only real option is to work to upgrade the debian package for osmosis. This isn’t easy either as I don’t know that much about ant/maven/ivy. I could spend I whole weekend just trying to update the debian package of osmosis and still get nowhere, all the time I’m just getting further away from my original goal which was to make some FOSM changesets from my latest trip while my memory is still fresh…

Categories: Uncategorized Tags: , ,

Using TileMill to make OpenStreetMap Map Stylesheets

June 6, 2011 1 comment

A short time ago TileMill got support for PostGIS data sources. I’m really excited by this as it means you can use TileMill, which is very easy to use, having a gentle learning curve, to construct your own OpenStreetMap stylesheets and see the data being rendered using your style. This is great! e.g. in less than an hour I made this,

Map of Sydney made using TileMill from OSM Data.

Map data © OpenStreetMap contributors, CC-BY-SA

Want a map which just shows the drainage network (which you can see from low zooms)? Easy.

Sydney Drainage Map

Map data © OpenStreetMap contributors, CC-BY-SA

I’m sure I’ll be publishing lots of new styles in the near future, one of which will be a style designed as an aerial image (NearMap) overlay. Exciting times ahead…

Categories: Uncategorized Tags: ,

Using ABS ASGS Data in OpenStreetMap

March 17, 2011 3 comments

A little while ago Marcus Blake from the Australian Bureau of Statistics asked the OSM community about the potential use of some ABS data. As I mentioned on the list I think it is good that at least some government departments are making their data available under free licenses and that they engage with with the community to sort out any technical details about the data.

As described by the ABS, the ASGS is essentially data describing geographical areas.

ASGS ABS Structures - CC-BY 2.5 AU Australian Bureau of Statistics

ASGS Non-ABS Structures - CC-BY 2.5 AU Australian Bureau of Statistics

Working out which if any structures should be incorporated into OSM and how needs careful consideration, and I’ve posted some of my thoughts to the list. In the mean time, since the data does contain some landuse information I’ve been looking into how best to use this information to aid in mapping. A blind import is not an option in my opinion, but I thought it would be handy to see the data as a base map when mapping.

I did try using ogr2osm to convert the data to the OSM xml format to load into JOSM (I even got the translateAttributes function for ogr2osm working for this dataset), but due to the nature of the data, I think a simple raster underlay works well. I tried two approaches in parallel.

  1. shp -> osm (using ogr2osm) -> postgres (using osm2pgsql) -> raster tiles (using mapnik).
  2. Using GeoServer to serve a WMS which can be loaded into JOSM.

Option 2 seemed to require less set up time. Simply used the GUI to load the shapefile, and apply a style. Then load the WMS into JOSM.

ABS Mech Blocks with landuse styling in JOSM, with NearMap underlay.

ABS Mech Blocks with landuse styling in JOSM, with NearMap underlay.

One caveat, if you want to load a WMS services from GeoServer into JOSM, I found the URL should look something like:

http://localhost:8080/geoserver/wms?service=WMS&version=1.1.0&request=GetMap&layers=workspace_name:layer_name&styles=&format=image/png&

Slightly unrelated but if you are using tomcat or jetty locally but only occasionally (like I do), I find it is best to use sysv-rc-conf (eg. sudo sysv-rc-conf tomcat6 off) to disable the tomcat or jetty daemon from running at boot, whilst still allowing you to start it (sudo service tomcat6 start) when you need it.

If anyone is interested in getting such data in JOSM and would like more details, just let me know.

Categories: Uncategorized Tags: , ,

Using XSLT to Transform XML data into OSM format

February 14, 2011 2 comments

For a while I used to think that all there was to XML was <blah attribute=”value”>inner</blah>, but of course there is much more. I’m now digging into the real stuff like XPath, XSLT and XML Schemas.

I’ve come across a data set of bus stops (as well as live info on where buses are, and their status). The bus stop data set (http://nswbusdata.info/ptipslivedata/getptipslivedata?filename=stopdescriptions.zip, no longer active so I’m hosting my original copies at http://tianjara.net/data/nsw-buses/ for preservation) is in an XML format,

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<StopDescriptionList license="http://creativecommons.org/licenses/by-nc-nd/3.0/au/" copyright="NSW Roads and Traffic Authority">
    <stop longitude="151.17832" latitude="-33.81852" tsndescription="Osborne Rd nr Ronald Av" TSN="206699"/>
    <stop longitude="151.17359" latitude="-33.8082" tsndescription="Ralston St nr Murray St" TSN="2066138"/>
    <stop longitude="151.17764" latitude="-33.82054" tsndescription="Second Av nr Osborne Rd" TSN="206698"/>
    <stop longitude="151.17629" latitude="-33.81926" tsndescription="Fourth Av nr Second Av" TSN="206697"/>
  ...

Although because of the license, I cannot use this data in OpenStreetMap, I was still interested in converting it into an a .osm file. The perfect job for XSLT!

It turned out to be quite a simple task with a neat solution. My XSLT stylesheet used to do the translation:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output method="xml" indent="yes"/> 

    <xsl:template match="/StopDescriptionList">
        <osm version='0.6' generator='XSLT'>
            <xsl:apply-templates select="stop"/>
        </osm>
    </xsl:template>

    <xsl:template match="stop">
        <xsl:variable name="count">
            <xsl:number/>
        </xsl:variable>

        <node id='-{$count}' lat="{@latitude}" lon="{@longitude}">
            <tag k='ref:tsn' v='{@TSN}' />
            <tag k='fixme' v='{@tsndescription}' />
        </node>
    </xsl:template>

</xsl:stylesheet>

Then it was a just a simple,

xsltproc -o busses.osm busses-stylesheet.xslt stopdescription.xml

The data is CC BY-NC-ND 3.0, but they sneak in some additional terms in the fine print, which in addition to the NC-ND would further lead to incompatibilities with the OSM license, and would under my definition of free data, make this data set non-free. For interest the first three additional terms are,

  1. You must not use the Data in any way that could create false or misleading outcomes or interpretations, or bring the RTA into ridicule or disrepute. You must not use the Data in conjunction with the promotion of alcohol or unsafe road practices.
  2. You must ensure that the Data used is current, and provide details as to the date and time of sourcing the Data from the RTA in all reproductions of the Data (including in any software applications incorporating the Data).
  3. In all reproductions of the Data (including in any software applications incorporating the Data), the following disclaimer must be provided: “The accuracy or suitability of the Data is not verified and it is provided on an “as is” basis.”
Categories: Uncategorized Tags: ,
Follow

Get every new post delivered to your Inbox.