Yesterday I pushed a bunch of changes to my OSM/Aerial Imagery Hybrid Style, see the (demo). I spent a bit of time on suburb and city labels as they are a really important feature of a map aimed at non-experts.

I’m finding making a non-trivial style like this has defiantly made me realise the difficulties of such a task.

There are a lot of improvements that can be made (either by modifying or making new components) to the OSM Mapping conventions, osm2pgsql/imposm, mapnik library, carto language stack. That said, the current form is still great and you can still make great maps. But,

Static OSM Tiles

I’m still yet to find a tile server which is fast and works well with lighttpd (nginx would probably suffice too). As an experiment I decided to pre-render a bunch of tiles for my hybrid style sheet. This tile layer doesn’t need to be minutely updated, anyone who needs that can use the normal mapnik layer. Also static tiles server straight from the webserver should be pretty fast (maybe on memcached tiles would be faster) and I wanted my tiles to be fast.

Next up how can I generate these static tiles? There is the popular generate_tiles.py, but that won’t render meta tiles, seems like such a waste to render every tile with a buffer of 128px when I would render a 5 by 5 meta tile of the same buffer for only 36% of the total pixels rendered. The larger the meta tiles the larger the latency, but if I’m pre-rendering them all than latency doesn’t matter any more.

So I wrote a C++ program as my replacement for generate_tiles.py. I also programmed it to render from a list of meta tiles rather than a bbox. This means for my demo I can only render high zooms where there is nearmap coverage. This is where https://github.com/andrewharvey/OSMTileListFromGeometry/ came in, which pulls nearmap coverage areas from an osm2pgsql database, and generates a list of meta tiles.

Using this method I rendered up to and including zoom 17, composed of 22444 5 by 5 meta tiles or 561100 regular tiles in a time of,

real    198m17.021s
user    103m49.985s
sys     37m36.117s

with disk usage,

<1M   0-8
2.0M  9
6.8M  10
26M   11
98M   12
13M   13
36M   14
114M  15
411M  16
1.5G  17
========
2.2G  total

(Updated with results when using “png” rather than “png256″)

real    228m17.082s
user    160m24.725s
sys     31m42.653s

with disk usage,

<1M   0-8
3.1M  9
8.6M  10
28M   11
121M  12
34M   13
83M   14
218M  15
663M  16
2.0G  17
========
3.1G total

I think that time could still be sped up with,

• tuning of the mapnik stylesheet queries
• tuning of the postgres database
• spanning more rendering threads
• using larger meta tiles
• using a machine with more that 512MB of RAM
• not running minutely updates in the background

Git Rebase

As I use git more and more I’m slowing learning more of the features it has to offer (and thanks to the free hosting by github). One such feature which I think is awesome is git rebase. As an example the other day I did git commit -ammend instead of git commit --amend (I knew there was a duplicate character somewhere but because I seem to have a tint of dyslexia I confused the duplicate — with mm).

Of course this resulted in committing all files which had changed with a message of “mend” as a new commit. I did this twice, and only noticed after I had already made a bunch of correct commits afterwards. With git rebase I could pop some commits of the commit stack, remove the two “mend” commits fix the commit which I should have been amended to and pop my other commits back on top of the stack.

Thanks to http://stackoverflow.com/a/180085.

Next Up

• build a daily OSM planet extract by pushing minute-replicate files into the osmosis psql simple schema, then dumping to an OSM file
• use this OSM file as a basis for monav and an OSRM service for fosm data
• run a name finder service for fosm data
• more updates to the hybrid map style
• render the hybrid style tiles as oblique images for nearmap multiview

Take a look at http://tianjara.net/osmt.html.

The source is at https://github.com/andrewharvey/osm-hybrid-carto/, licensed CC-BY-SA.

There is still lots of work to do on it though.

I also have a lightweight cycle-overlay too (view example at http://tianjara.net/leaflet.html and tick on the cycle overlay or view the code at https://gist.github.com/1216315)

I think this is a poor choice, my reasoning is because now you have to use modifier+Del for both send to trash and skip trash. It makes sense to have Del as safe delete (send to trash) and a modified+Del for hard delete (no trash) that is logical and it is easy to remember the difference between the two. But now in this new situation a large majority of users (me included) won’t be able to remember which modifier does what. Is it Ctrl or Shift and will just pick one at random, giving you a 0.5 chance of doing the actual type of delete you wanted.

Also I don’t see any reason to add more safety to the straight delete key, after all it only goes to the trash so you can recover it if you accidentally hit the key. If you want add a sound upon sending something to the trash as an audio cue.

The number one thing I learned in my Human Computer Interaction course at uni was get feedback from the users (this isn’t just asking for feedback, but also observing how they use the software). This is my feedback as a user, and if Gnome developers want to improve usability then they should take in the feedback, i.e. read this.

My first method was to use osmconvert with the aid of the workflow given in https://github.com/rrankin/osmconvert/blob/e1fbb7319f92f338e0023d110a3098f5a979fd64/update_osm.sh. It was quite fast and it almost worked, but due to limitation of osmconvert it seems that it expects the objects to be sorted in the order given by the object IDs. We can either patch osmconvert to work around this, or sort the osc files…

Putting the first method on hold, I gave installing the rails port that runs the API at osm.org a go with the idea that I can load the changesets or osc files directly into the API or DB. I got it installed and got to the step of loading in my starting OSM .osm file from before the FOSM fork. http://wiki.openstreetmap.org/wiki/The_Rails_Port#Populating_the_database even gives the command line to run, but osmosis didn’t like this. I believe the issue is fixed with later versions of osmosis though.

This leads me to require a later version of osmosis. I don’t really want to install osmosis from source using the upstream project’s method as it will pull in a bunch of 3rd party libraries using maven, so the only real option is to work to upgrade the debian package for osmosis. This isn’t easy either as I don’t know that much about ant/maven/ivy. I could spend I whole weekend just trying to update the debian package of osmosis and still get nowhere, all the time I’m just getting further away from my original goal which was to make some FOSM changesets from my latest trip while my memory is still fresh…

Just viewing the obliques with NearMap’s front end at nearmap.com, you will often find all the buildings are slanted on an angle, with my viewer however you can correct this to make them straight and looking nice again by tweaking the camera rotation. If nothing else came out of this experiment I think that just the mere ability to easily correct this artefact to produce nicer looking obliques made it a worthwhile endeavour.

You could push the concept further and add perspective (like Google Earth, rather than my simple parallel projection) and have each tile pick the optimal {Vert|N|S|E|W} layer, although with LIDAR progressing we can now capture true 3D so we don’t need to fake it with 2D images any more…

I think I would have preferred just buying a new dedicated machine myself, however unless you’re in the CBD you can’t get upload speeds much more than 1.5Mbps around here (if you are lucky), which is not really fast enough, hence I’ve gone with the 3rd party hosting option. Perhaps if this NBN the government keeps taking about is turned on and not too expensive I’ll switch. In the mean time it seems silly that Australians need to host content that is mainly consumed by Australians overseas just because of the price difference.

For example if A wants to host content to Australians, but it is cheaper to do so from overseas, then A will mostly likely host their content overseas. This just means Australia has an even greater disturbance in the overseas traffic up/down ratio (i.e. we pull more than we push), which in turns means our local ISPs have to pay more compared to the overseas ISPs to lay the cables, this in turn leads to higher internet costs for Australians compared to overseas ISPs.

At any time our local ISPs could provide incentives for keeping traffic local inside the country by making overseas traffic more expensive than local traffic (for both people hosting and consuming), which would help Australians host locally and in turn even up the balance. It also means that the overseas pipes won’t need to be as thick (because for sites which mostly get Australian visitors, hosting locally means the total amount of cross country traffic for that site is less, hence more efficient).

There is not much I can do about this, so I’m giving in and hosting overseas with the hope that one day things will return to sanity, but in the meantime I haven’t been left off-line.

Anyway, back to Linode, I having a lot of fun at the moment setting it up and getting things working.

I’m running Debian 6.0, with unattended-upgrades, lighttpd, ufw, etckeeper (and soon either exim or postfix and possibly I’ll migrate this wordpress blog across). I have run through http://www.debian.org/doc/manuals/securing-debian-howto/ it’s a bit dated and I’m by no means following everything, but none the less it’s still a nice read.

I’ve set up ssh with protocol 1 disabled (securing debian howto says it has some design flaws, so why enable it if I don’t need it?), publickey authentication only, fail2ban, and only accepting traffic to the ssh port from IP ranges I know I’ll be connecting from (I know if I’ve got security via publickey auth, I don’t really need this, but it can’t hurt… at least my logs don’t fill up with as many break in attempts.).

However since one can always log into the machine via the console at linode.com, security here comes down to the weakest link of web based username/password and ssh publickey auth (ignoring all the other threats like compromised VM separation, compromised VM host, physical security, etc.. stuff I have no control over).

I have set up lighttpd as RAM is limited and I read that this (and nginx) are better than Apache httpd in this regard. I have also set up munin and munin-node (and adapted /etc/munin/apache.conf for lighttpd). I would prefer the munin stats not be appreciable to the world, so I just expose it to localhost only and access it via an SSH tunnel.

Additionally, while trying to debug lighttpd configuration I noticed that it wasn’t responding to nc. Apparently lighttpd will only respond to my requests when lines end with \r\n but just typing away in the terminal as input to nc and using the Enter key I only end lines with \n. Apparently this is per the HTTP spec and it only works with Apache because it isn’t as strict about this.

Next I have to see about getting a domain name. This is a huge problem in itself, but it since,

• the IP can change at any time (and will change if I move away from Linode),
• you can’t have sub-domains to separate parts of the site, and
• you can’t really do email

I don’t really have a choice. Unfortunately the current ICANN DNS is the only real option at the moment, so I’m just going to have to pay up, and try to avoid having some details which I don’t want listed, listed in WHOIS. At the moment I’ll probably go for a .id.au domain.

I’ll probably move this blog across once I set up a domain name, so more news to come on this later.

Update: I’ve registered with gandi.net. My site is now available at http://tianjara.net/

Map data © OpenStreetMap contributors, CC-BY-SA

Want a map which just shows the drainage network (which you can see from low zooms)? Easy.

Map data © OpenStreetMap contributors, CC-BY-SA

I’m sure I’ll be publishing lots of new styles in the near future, one of which will be a style designed as an aerial image (NearMap) overlay. Exciting times ahead…

As described by the ABS, the ASGS is essentially data describing geographical areas.

ASGS ABS Structures - CC-BY 2.5 AU Australian Bureau of Statistics

ASGS Non-ABS Structures - CC-BY 2.5 AU Australian Bureau of Statistics

Working out which if any structures should be incorporated into OSM and how needs careful consideration, and I’ve posted some of my thoughts to the list. In the mean time, since the data does contain some landuse information I’ve been looking into how best to use this information to aid in mapping. A blind import is not an option in my opinion, but I thought it would be handy to see the data as a base map when mapping.

I did try using ogr2osm to convert the data to the OSM xml format to load into JOSM (I even got the translateAttributes function for ogr2osm working for this dataset), but due to the nature of the data, I think a simple raster underlay works well. I tried two approaches in parallel.

1. shp -> osm (using ogr2osm) -> postgres (using osm2pgsql) -> raster tiles (using mapnik).
2. Using GeoServer to serve a WMS which can be loaded into JOSM.

Option 2 seemed to require less set up time. Simply used the GUI to load the shapefile, and apply a style. Then load the WMS into JOSM.

ABS Mech Blocks with landuse styling in JOSM, with NearMap underlay.

One caveat, if you want to load a WMS services from GeoServer into JOSM, I found the URL should look something like:

http://localhost:8080/geoserver/wms?service=WMS&version=1.1.0&request=GetMap&layers=workspace_name:layer_name&styles=&format=image/png&

Slightly unrelated but if you are using tomcat or jetty locally but only occasionally (like I do), I find it is best to use sysv-rc-conf (eg. sudo sysv-rc-conf tomcat6 off) to disable the tomcat or jetty daemon from running at boot, whilst still allowing you to start it (sudo service tomcat6 start) when you need it.

If anyone is interested in getting such data in JOSM and would like more details, just let me know.

First impression is compiz seems to be much slicker (ie. animations are smoother) on nouveau rather than the nvidia proprietary driver. I like it.

I had a couple of hiccups in the switch. I needed to clean /etc/X11/xorg.conf down to the 4 line minimal version to get a sane display resolution, and I needed to properly remove the nvidia proprietary driver (apt-get remove worked, but I needed to remove all the non-free nvidia packages). I also needed to install libgl1-mesa-dri-experimental to get direct rendering for compiz and basic OpenGL programs to work. Compiz seems to run fine now. The only noticeable difference in SuperTuxKart is a white flicker…