Sydney Gets a new (and improved) Weather Radar

I just realised that the new capabilities of the Sydney BOM radar aren’t just something that I had missed before, rather they are indeed new. The Buero brought online their new Terry Hills radar (on 9/9/09). Here is their media release (from Wednesday 9 September 2009).

New Sydney weather radar now online

Sydney has a new state-of-the-art weather radar that will help forecasters and the community see in greater detail upcoming weather – including severe weather such as thunderstorms.

The Doppler weather radar was launched today at Terrey Hills by Australia’s Director of Meteorology, Dr Greg Ayers.

The new radar is more sensitive and uses the latest technology to provide clearer radar images at a higher resolution than previously possible. It replaces one that has been operating at Appin, south of Sydney since 1992.

Commissioning the radar, Dr Ayers noted the significance of its range which spans Sydney and surrounding areas – “an area that is home to more than a quarter of the Australian population.”

“Weather radars play an important role in helping the community prepare for and manage the threats posed by extreme weather.”

The Bureau’s Regional Director for New South Wales, Barry Hanstrum said the improved capability offered by this new radar will help forecasters “more easily detect and track thunderstorms in the Sydney area and detect dangerous wind changes during the fire season.”

The new radar animation, covering the previous half hour’s weather, now consists of six images, one every six minutes where as previously it was just four images. “This upgrade will certainly provide a new level of detail about weather in the Sydney area,” Mr Hanstrum said.

“The greater sensitivity of the radar will also assist in better detection of drizzle and light shower activity over Sydney.”

This new Doppler weather radar is the fifth of six new high resolution radars to come online as part of the Australian Government’s $62.2 million dollar radar upgrade project.

Recently the Government announced a further $48 million program to install four new radars around Australia as well as invest in the underlying science to integrate this technology into the current radar network. Imagery from the Bureau’s network of weather radars is available on the Bureau of Meteorology website www.bom.gov.au

–http://www.bom.gov.au/announcements/media_releases/nsw/20090909.shtml, © Copyright Commonwealth of Australia 2009, Bureau of Meteorology (ABN 92 637 533 532). Information is presented with the permission of the Bureau.

This is great news! Higher resolution radar images, new images every 6 minutes, a Doppler wind map. They have some nice documentation which I’ll read when I get a chance. The only thing I don’t like is the radar loop only shows the last 30 minutes. This is nothing to do with the radar, just the web interface. But the situation is not too bad, they have all the radar maps/data available through their FTP site, and they have a permissible licence which allows republication of this data, so anyone is free to build their own interface which could allow you to loop through more than just the last 30 minutes (which is something I have on my TODO list). But, I think they only keep something like the last hour or 1h42min of radar images on their HTTP/FTP servers so you would need an always on machine to ensure you have all the data.

Channel TEN’s Online Video

After someone commented on one of my other posts I had another look around the Channel TEN Video site, this is what I gathered.

So it appears that the whole Channel 10 Video section has been outsourced to http://www.kit-digital.com/ using their vx.roo.com hosting. Thus the vx.roo.com host has to allow for other clients not just Ten. vx.roo.com appears to have assigned Ten’s “SiteIdGuid” as “666b8363-97e9-4c40-b665-53846db95ad0″. Here was my first port of call http://publish.flashapi.vx.roo.com/666b8363-97e9-4c40-b665-53846db95ad0-4883/PlaylistInfoService.asmx. From there, http://publish.flashapi.vx.roo.com/666b8363-97e9-4c40-b665-53846db95ad0-4883/PlaylistInfoService.asmx?op=GetPlaylistXML is a good place to go. From there they tell you how to make SOAP, HTTP GET and HTTP POST requests. I’ll use HTTP GET for now because its easiest for you to follow along.

As I mentioned the first field is SiteIdGuid which for Channel TEN is 666b8363-97e9-4c40-b665-53846db95ad0. The second field is Channel. Some channel codes are listed on Ten’s video page, but we want the vxChannel. For example in the URL http://www.australianidol.com.au/video.htm?vxSiteId=666b8363-97e9-4c40-b665-53846db95ad0&vxChannel=S7CUTV%3AAuditions. You can usually find it in the URL somewhere.

Using the vxSiteId and vxChannel you can formulate the RSS feed for that channel,

http://publish.flashapi.vx.roo.com/xmlgenerators/video/$vxSiteId/RSSGenerator.aspx?siteId=$vxSiteId&channel=$vxChannel

From that RSS feed you will usually get a list of clips. For example in the RSS feed, http://publish.flashapi.vx.roo.com/xmlgenerators/video/666b8363-97e9-4c40-b665-53846db95ad0-4882/RSSGenerator.aspx?siteId=666b8363-97e9-4c40-b665-53846db95ad0&channel=S7CUTV:Auditions, the fist item is a link to http://www.australianidol.com.au/video.htm?channel=S7CUTV:Auditions&clipid=2692_030TT070809&bitrate=300&format=flash. This gives you the clipid (and the bitrate and format, but those usually 300 or 700 and flash). Using this you can fill out the details for the fields on this page, http://publish.flashapi.vx.roo.com/666b8363-97e9-4c40-b665-53846db95ad0-4883/PlaylistInfoService.asmx?op=GetPlaylistXML. But that invoke button doesn’t work, so just make your own HTTP POST request, something like, http://publish.flashapi.vx.roo.com/PlaylistInfoService.asmx/GetPlaylistXML?SiteIdGuid=666b8363-97e9-4c40-b665-53846db95ad0&Channel=S7CUTV%3AAuditions&Bitrate=300&Format=flash&ThumbnailTypeCode=square_large&RowCount=1&StartPosition=0&ClipId=2692_030TT070809&Artist=&Album=&Criteria=&RelatedLinksKeyName=

It appears you need all those other arguments but they mostly stay the same. Anyway, this gives you an XML file with details/metadata for that clip, but it also gives you the URL of the FLV.

Also I just noticed some things going on over at the forums on whirlpool.net.au, http://forums.whirlpool.net.au/forum-replies.cfm?t=1212283. And this stuff is actually useful. The other day one of my lecturers mentioned a show that had been shown on ABC (but originally aired in the UK) about the history of law in England, which is what we looked at in one of my gen-ed classes at uni.

Anyhow, I hope this to be the last of my posts on this kind of thing, so I can try to do more unswcourse posts.

Facebook Video Updates as an RSS Feed (Using a Shell Script)

We are finally learning common Unix tools at uni. Gosh I wish we had done these earlier because they are so useful! (yes I could have learnt them myself, and I did a bit. But I ended up just learning the parts to get the job done. This didn’t always work because I had very little understanding of why things worked (and why they didn’t) and thus things turned into trial and error).

So anyway I wanted an RSS feed for videos uploaded on Facebook to public pages. (For example http://www.facebook.com/video/?id=20916311640). So I put my newly learnt skills to good use and wrote a shell script.

#!/bin/sh
wget http://www.facebook.com/video/?id=$1 -q -O - -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv: 1.8.0.3) Gecko/20060523 Ubuntu/dapper Firefox/1.5.0.3' | grep 'http://www.facebook.com/video/video.php?v=' | sed -e 's/http:\/\/www.facebook.com\/video\/video.php?v=[0-9]*/\n&\n/g' | grep 'http://www.facebook.com/video/video.php?v=' | uniq | sed -e 's/.*/<item><title>&<\/title><link>&<\/link><\/item>/' | sed "1 s/^/<?xml version=\"1.0\"?><rss version=\"2.0\"><channel><title>Facebook Video Feed<\/title><link>http:\/\/www.facebook.com\/video\/?id=$1<\/link><description>Facebook Videos for ID $1<\/description><language>en-us<\/language>/" | sed '$ s/$/<\/channel><\/rss>/'

UPDATED: (links on the page from facebook no longer have the domain etc in the link)

#!/bin/sh
wget http://www.facebook.com/video/?id=$1 -q -O - -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv: 1.8.0.3) Gecko/20060523 Ubuntu/dapper Firefox/1.5.0.3' | grep '/video/video.php?v=' | sed -e 's/\/video\/video.php?v=[0-9]*/\n&\n/g' | grep '/video/video.php?v=' | uniq | sed -e 's/.*/<item><title>http:\/\/www.facebook.com&<\/title><link>http:\/\/www.facebook.com&<\/link><\/item>/' | sed "1 s/^/<?xml version=\"1.0\"?><rss version=\"2.0\"><channel><title>Facebook Video Feed<\/title><link>http:\/\/www.facebook.com\/video\/?id=$1<\/link><description>Facebook Videos for ID $1<\/description><language>en-us<\/language>/" | sed '$ s/$/<\/channel><\/rss>/'

Facebook will actually check the user agent and refuse to serve users it doesn’t like so I had to spoof it. So anyway the pipeline will grab the html page and find all the links to individual videos and feed these out, one line for each (this is up to just after the uniq). Next I add some text to turn this list into a basic RSS file. I don’t worry about making it fancy with the video title, thumbnail etc. because honestly I don’t care about that for my use.

To actually use it I can use cron, (actually I think its easiest to make another shell script and put this in /etc/cron.daily/ or /etc/cron.hourly/) to run the command,
./fbvidrss.sh 20916311640 > /var/www/fbvid_20916311640.xml

SBS Playlist to RSS Feed Perl Script v2

I have made some changes to my original script. This new perl script will scrape info from sbs.com.au and give an RSS feed of the items in the specified playlist. I only know of two playlists (94 = Latest Full Episodes, 95 = Preview Clips). Only one line needs to be changed to use the script to give the RSS feed of a different playlist. The major improvement is the items that are only available over RTMP now have the correct URL which was previously incorrect (but now the script runs slower as it has to grab more pages from the web). I use the url, http://player.sbs.com.au/video/smil/index/standalone/$item_code/ to find out the url details.

FLVStreamer appears to do a good job of downloading media over the RTMP protocol. Just use ./flvstreamer -r rtmp://file.flv > file.flv. Mozilla has an article on how to add protocol’s to firefox here. But I didn’t bother with that as the command is simple as it is, and building an app with a save as dialogue is beyond me for now, but I hope to learn that soon.

[Update: It seems that you also need to have the --swfUrl argument set ('http://player.sbs.com.au/web/flash/standalone_video_player_application.swf' works.). Also the perl script doesn't get the file name correctly (it uses the thumbnail image url, rather it should be using the url's given at the /video/smil/index pages).]

For local use the current format will probably be what you want, but in a production environment you probably want to have the script save the RSS file to disk and have people hit that RSS file with the requests. Just set the perl script to run every now and then. Unfortunately I can’t seem to upload .pl files to WordPress (I’ve put a link, but that will expire eventually)… I really need to get my own site.. There is so much customisation I would like to do and many experiments to try out on a live server, but the $$$’s are too much…

On another note I tried out EPIC (Eclipse Perl Integration), which was fairly simple to install. It seems much nicer than using a plain text editor and command line, especially the debugging abilities that it adds.

SBSPlaylistToRSSv0.2.1.pl

SBS Latest Online Video RSS Feed

[An updated (but more complex) script can be found in this post]

I needed an excuse to practice some Perl. So this was my first try.

The Perl script below will convert http://www.sbs.com.au/shows/ajax/getplaylist/playlistId/94/ to an RSS feed. That 94 playlist is a list recent episodes from the TV broadcaster SBS available online. This may not work if the source file’s structure changes.

#!/usr/bin/perl

# This script will download the ajax xml file containing the latest full episode videos added to the SBS.com.au site.

#Adapted from the code at http://www.perl.com/pub/a/2001/11/15/creatingrss.html by Chris Ball.

# I declar this code to be in the public domain.

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

use strict;
use warnings;

use LWP::Simple;
use HTML::TokeParser;
use XML::RSS;
use Date::Format;

# Constants
my $playlisturl = "http://www.sbs.com.au/shows/ajax/getplaylist/playlistId/94/"; # Latest Full Ep
#my $playlisturl = "http://www.sbs.com.au/shows/ajax/getplaylist/playlistId/95/"; # Latest Sneek Peek

# LWP::Simple Download the xml file using get();.
my $content = get( $playlisturl ) or die $!;

# Create a TokeParser object, using our downloaded HTML.
my $stream = HTML::TokeParser->new( \$content ) or die $!;

# Create the RSS object.
my $rss = XML::RSS->new( version => '2.0' );

# Prep the RSS.
$rss->channel(
 title            => "SBS Latest Full Episodes",
 link             => $playlisturl,
 language         => 'en',
 lastBulidDate    => time2str("%a, %d %b %Y %T GMT", time),
 description      => "Gives the most recent full episodes avaliable from SBS.com.au"
 );

$rss->image(
 title    => "sbs.com.au Latest Full Episodes",
 url    => "http://www.sbs.com.au/web/images/sbslogo_footer.jpg",
 link    => $playlisturl
 );

# Declare variables.
my ($tag);

# vars from sbs xml
my ($eptitle, $epthumb, $eptime, $baseurl, $img, $url128, $url300, $url1000, $code1char, $code2char, $code1);

#get_tag skips forward in the HTML from our current position to the tag specified, and
#get_trimmed_text  will grab plaintext from the current position to the end position specified. 

# Find an <a> tag.
while ( $tag = $stream->get_tag("a") ) {
 # Inside this loop, $tag is at a <a> tag.
 # But do we have a "title" token, too?
 if ($tag->[1]{title}) {
 # We do!
 $eptitle = $tag->[1]{title};
 #print $eptitle."\n";

 # The next step is an <img></img> set.
 $tag = $stream->get_tag('img');
 $epthumb = $tag->[1]{src};

 #get the flv urls from the img url
 #eg. http://videocdn.sbs.com.au/u/thumbnails/SRS_FE_Global_Village_Ep_19_44_48467.jpg
 #print $epthumb."\n";
 $baseurl = substr($epthumb, 40, length($epthumb)-40-4);
 $url128 = "http://videocdn.sbs.com.au/u/video/".$baseurl."_128K.flv";
 $url300 = "http://videocdn.sbs.com.au/u/video/".$baseurl."_300K.flv";
 $url1000 = "http://videocdn.sbs.com.au/u/video/".$baseurl."_1000K.flv";

 #SRS|DOC|MOV
 $code1char = substr($baseurl,0,3);
 #SP|FE
 $code2char = substr($baseurl,4,2);

 my %epcode_hash = (
 'DOC'    => 'Documentary',
 'MOV'    => 'Movie',
 'SRS'    => 'Series',
 );
 $code1 = $epcode_hash{$code1char};

 $stream->get_tag('a');
 $tag = $stream->get_tag('p');

 # Now we can grab $eptime, by using get_trimmed_text
 # up to the close of the <p> tag.
 $eptime = $stream->get_trimmed_text('/p');

 # We need to escape ampersands, as they start entity references in XML.
 $eptime =~ s/&/&amp;/g;

 # Add the item to the RSS feed.
 $rss->add_item(
 title         => $eptitle,
 permaLink     => $url1000,
 enclosure    => { url=>$url1000, type=>"video/x-flv"},
 description     => "<![CDATA[<img src=\"$epthumb\" width=\"100\" height=\"56\" /><br>
 $eptitle<br>
 $eptime<br>
 Links: <a href=\"$url128\">128k</a>, <a href=\"$url300\">300k</a>, <a href=\"$url1000\">1000k</a><br>
 Type: $code1<br>]]>");

 }
}
print "Content-Type: application/xml; charset=ISO-8859-1"; # To help your browser display the feed better in your browser.
#$rss->save("sbslatestfullep.rss"); #this will save the RSS XML feed to a file when you run the script.
print $rss->as_string; #this will send the RSS XML feed to stdout when you run the script.
 

The Cause of Slow Loading WordPress Pages Over https

For as long as I could remember, loading pages in my wordpress.com blog dashboard was really slow. I should have realised what was happening sooner but I never took the time to investigate. Whether I went to the edit posts page (/wp-admin/edit.php), new post (/wp-admin/post-new.php) there seemed to be numerous connections back to wordpress.com once parts of the page were loaded. These requests were to s-ssl.wordpress.com. Taking a look at the source, all the css and js files linked to from the html of the page were over the https protocol, and rightly so because I always go over the https protocol. What I didn’t realise is that Firefox will not cache files from https by default. So if I go to about:config and change browser.cache.disk_cache_ssl to true then these static css and js files will be cached. I restart my browers and all of a sudden pages load much faster and much more tolerable. The only problem is that its not just css and js files transfered over https that are cached but html files as well. I’m not sure how to get Firefox just to cache css and js files from https, but I have to leave that for another day.

Re: 7.30 Report “Uncertain future for newspapers”

Going through the backlog. Just a few comments on the 7.30 Report story (transcript) (video sorry for the format but its all I could see, I don’t know where I got my m4v one from).

Very interesting stuff here, but I have a couple of points. Mind you I don’t have much experience here, I’m not a journalist or an economist…

1. Government Subsidiary for Quality Journalism

One of my main concerns here is that you have two negative forces. On one hand you have the government paying for investigative journalism, but those journalists are having to fight the government to get the story to break. Would there be a need for investigative journalism in the government arena if the government was more open? You wouldn’t need the journalist filing freedom of information requests if the department put this info into the public domain by default. The solution is for the government to be more open and transparent, something which they seem to do a lot of talk about (and are doing some things that make them open), but not nearly enough.

But the government or the government departments are probably unwilling to put information out there that may embarras them. Unfortuantly you probably need so driving for that motivates government departments to be more open. I see an online “village pump” where the community can gather and build up in numbers to support certain movements (such as access to certain statistics that may be part of a journalists investigation). Those numbers are a force that could provide pressure for an unwilling government department.

2. Coverage

Nick Davies said that “what you haven’t got is citizen journalists covering the courts or the government departments or the police or the hospitals or the schools or doing investigations.” He quoted lack of “skills, time or resources” as the reason for this. I don’t believe this. A large chuck of the feeds I subscribe to are citizens blogging about copyright decisions made in court. Perhaps it is lack of cooperation of the government department. For example they charge huge unreasonable fees for your FOI application. I don’t see the solution as get a big company who can pay the fees, rather use some other methods to pressure the department into providing the information needed for investigative journalism free or charge, free for all.

Channel TEN’s contractor/outsourcer/… Posts TV Show Online Before It Airs

It seems that channel ten (or at least their contractor/outsourcer/whatever) decided to release episodes of a show its airing (Merlin) on the interent before their have aired on TV. Its not such a big deal because the show has already aired in the UK, but I would still say good on you channel 10 for going that extra mile and supporting the fans who want to watch the show through legal methods.

The links are (its a 13 episode season),

???
???
http://flash.vx.roo.com/streamingVX/19056/1395/geo/ausonly/2009/Q2/MERL-3fullep-210509_700.flv
http://flash.vx.roo.com/streamingVX/19056/1395/geo/ausonly/2009/Q2/merlin_ep4_220509_700.flv
http://flash.vx.roo.com/streamingVX/19056/1395/geo/ausonly/2009/Q2/MERL-5fullep-270509_700.flv
http://flash.vx.roo.com/streamingVX/19056/1395/geo/ausonly/2009/Q2/MERL-6fullep-270509_700.flv
http://flash.vx.roo.com/streamingVX/19056/1395/geo/ausonly/2009/Q2/MERL-7fullep-270509_700.flv
http://flash.vx.roo.com/streamingVX/19056/1395/geo/ausonly/2009/Q2/MERL-8fullep-270509_700.flv
http://flash.vx.roo.com/streamingVX/19056/1395/geo/ausonly/2009/Q2/MERL-9fullep-280509_700.flv
???
http://flash.vx.roo.com/streamingVX/19056/1395/geo/ausonly/2009/Q2/MERL-11fullep-280509_700.flv
http://flash.vx.roo.com/streamingVX/19056/1395/geo/ausonly/2009/Q2/MERL-12fullep-270509_700.flv
http://flash.vx.roo.com/streamingVX/19056/1395/geo/ausonly/2009/Q2/MERL-13fullep-280509_700.flv

But they are sure to change, esp the dates (but they were valid when I posted them (wget -S –spider filename, will verify if a file exists for you)), so you may need to change them (the dates) (ps. DownThemAll is great at trying many different file names to see which ones get a hit). On ten’s web site they only list the most recent episode but if you look at your HTTP logs you can see the flv URL. There is no key in the file so you should be able to just change the episode number and then try all dates in the near months.

PS. In the odd chance than someone from TEN reads this you should note that Australia has poor interent (in terms of IP quota) as such we have a greater need to be able to download shows and watch later (so the download can be done off peak, or from a location with more generous IP quota). You should consider promoting friendly file formats straight for download. Perhaps then you will have a larger uptake of online viewers (I would find a short advertisement at the start and/or end of the video acceptable).

PSS. I’m not one of those people who thinks posting links to pirated material should be (or is) illegal or immoral. Among many other reasons, how is the person who posts the link supposed to know if the material they are linking to is infringing?

PSSS. This is not a vunrability and as such I don’t think I am required under my own ethics to let them know. But regardless this is obviously intentional, and dispite all this I cannot report it to TEN anyway as there is no email listed on their contact us page.

NT Government Refuses To Allow Republication of Hansard

It’s sad but I’m not surprised. Not only does this reinforce that the Government does not want you republishing what they say in Parliament sessions (as per their copyright statement on their web site), but they are not even willing to grant specific rights to specific groups for specific intentions.

From http://tickets.openaustralia.org/browse/OA-237,

from Katherine Szuminska <kat[at]openaustralia.org>
to steve.stokes[at]nt.gov.au
date 6 May 2009 22:18
subject Hansard Copyright request NT
mailed-by openaustralia.org

Hi Steve

I am writing to you from OpenAustralia in your capacity as the contact for the NT Parliamentary Hansard as per http://www.nt.gov.au/lant/hansard/hansard.shtml

We are a group of volunteers who run a website, http://www.openaustralia.org which republishes the Hansard from Federal Parliament in a user friendly searchable format. We also support email alerts and rss feeds by keyword.

In future, as well as the Senate and the House of Representatives, we’re also intending to republish the State and Territory Parliament Hansards in the same format, making it even easier for all Australians to have access to their elected Representatives and be easily informed of Parliamentary proceedings at all levels.

Specifically in this case we’d like to republish the NT Parliamentary Hansard at www.openaustralia.org

—————————————————————————————————————————————-

On 27 May 2009, at 15:56, Steve Stokes wrote:

Hi Katherine,

Please note that authority has been given for you to link only to our Hansard & Legislation page or more specifically the Parliamentary Record databases on that page as per the Hansard link below. Unfortunately, authorisation has not been given to republish Northern Territory debates in another format.

Should you require any further information, please feel free to contact me.

Regards,

Steven Stokes
Table Office Manager
Chamber Support
Ph: (08) 8946 1447

Bye Bye UNSW Email

No time to discuss the ethical dilemmas or my opinion but I’ve worked out why my UNSW email stopped working. In short UNSW no longer provides students with email accounts. They mealy provide an address (@student.unsw.edu.au) that will redirect any mail to an email address of your choice. No webmail, no POP access, no IMAP access, who knows if they actually store any emails at all. An if you want to send mail from your UNSW email well you cannot if you are not at uni and I don’t even know if you can at uni. But you would need to have access to a computer with a mail program (so don’t bother with the library) and try their SMTP outgoing mail server, I haven’t tested this. Sure you can fake your mail to be sent by your unimail address but this is quite pointless.

All is not lost, CSE students get a CSE email so I can send mail from that account.

Oh and what about zmail you say? Its just a prepackaged Microsoft email thing. ie. its just like all your other service based emails like Gmail, Yahoo Mail, Hotmail. You just tell the uni servers to forward your mail to Microsoft. All the outgoing and POP/IMAP servers are in the outlook.com and exchangelabs.com and hotmail.com domains (bye bye internet quota, yes the uni now has to (presumably) pay more $$$).

I don’t like this. I’ll post more after the session is over.