Ch-ch-ch-changes (new job)

As usual, it's been a while, but people seem to be getting used to the idea of it being OK to blog only occasionally, now that (insert social networking site of choice) has cornered the market in real-time link sharing and minutiae.

I'm moving jobs soon, and it seems right to mark such tumultuous events with a bit of prose. So, soon I'll be leaving my position of Senior Product Manager within the Online Technology Group (OTG, formerly Online Media Group (OMG), formerly Digital Media Technologies (DMT)) of BBC Future Media & Technology (FM&T) to become Lead Engineer in the Prototyping team of BBC Research & Development (R&D).

WTF? You bet. None of my family really know what I do, because it takes too damn long to even explain where I work, never mind all the TLDs.

Within OTG and its former guises, I've been closely involved in the development of the huge success story that is BBC iPlayer, mainly making the audience consumption aspects of its various streaming & download platforms measurable and reportable on. No small task, given both the size and the complexity of the iPlayer offering - although primarily about streaming audio or video to your browser via Flash, there's many more additional ways to consume its content (on mobiles, games consoles, Freesat and other set top boxes, iPads, integrated televisions..) and they all need to be measured in some way so we can see how they're being used, and how popular they're becoming with audiences.

To cut a long story short, I (and subsequently the team I built) designed and built a suite of data munging & measurement products known internally as iStats, and it's those systems that ingest and produce all the data on which iPlayer's audience and business reporting is based. You can see some of the data from iStats in the press packs released here. As well as external and regulatory reporting, the data has been used for all sorts of purposes - measuring how new features perform (such as Series Catchup), debugging content delivery issues across different CDNs and ISPs, informing which new platforms to target, and feeding into product UX decisions (such as the new iPlayer v3 beta).

I'm very proud to have played a small part in the big success that is iPlayer, and I think I've progressed hugely as both a software engineer and manager over the last three years (I'm one of those weird sub-Spolsky types who likes doing both). It's time for a new set of interesting challenges, however, and I've always wanted to work across different areas of the BBC, so I have hopes that my new role will bring that. The team work on all sorts of diverse, fascinating, forward-looking things (like P2P AV delivery, real-time services, and all manner of other stuff I don't quite grok yet) across hardware and software so I'm hugely excited to be a part of it.

Of course, there's no point doing things in halves, so I'm naturally buying a house and preparing to move at the same time as all of this going on. Hmm.

This blog has moved

This blog is now located at http://blog.70cities.net/.
You will be automatically redirected in 30 seconds or you may click here.

For feed subscribers, please update your feed subscriptions to
http://blog.70cities.net/feeds/posts/default.

Thoughts on Scalecamp

The other Friday I attended Scalecamp, one of these 'unconference' jobbies - organised by the Guardian. I'm not a very conference-y person, but given that I'm in the big data game and have come up against (and overcome) many a scaling problem in the past, I thought it'd be good to pop along and see what was happening.

It was certainly very well-organised, with everyone knowing what they were doing - obviously quite a few of the organisers/attendees had done this sort of thing before, whereas I hadn't. I enjoyed a few of the talks and sessions, particularly the ones that touched on Hadoop and Memcached, because they're very useful/interesting to us. Quite a lot of NoSQL data warehousing dogma floating around the industry already, which I did my best to puncture - I probably had my most interesting discussions here.

I was also quite taken aback at how few Perl hackers I met there. Bar one or two exceptions, most of whom I knew or I've worked with already, it's all about Python, Ruby, PHP and Java these days. I felt about 5,000 years old.

Any negatives? Not really - I had to skip out when the talks had finished and so missed the sponsored beers, which was a bit of a pain. Also, some nerds really need to find where their shower is. Seriously - it's not difficult. BO is very much the elephant in the room (almost literally) at any gathering of geeks. In fact, I might do a lightning talk on it at some point in the future*.

In other news, I'm worried that I've developed an allergy to certain premium brand lagers. No, really..

* This is a joke.

It's a numbers game

I seldom talk/blog/tweet/etc about work directly, mostly because I never have the time, but I'm feeling quite proud that I can point out a couple of things I've been directly involved in in one way or another:

Jo Hamilton, Head of Audience Measurement writes on the BBC Internet Blog and shares a data pack with insights about BBC iPlayer's usage in October 2009 - a record month across all platforms.

And John Linwood (BBC FM&T CTO, AFAIK) has been sharing similar traffic and audience statistics with a great infographic.

All of the audience-oriented statistics are derived from the internal iStats data collation, warehousing and analytics system I designed, built and now product manage as part of my role in the BBC's Future Media & Technology division.

The public release of this data above is, hopefully, a step in the direction of sharing more data about how iPlayer's content gets consumed. I'd expect more data to be trickling out soon if the thirst is there.

One day I might sit down and write an 'official' blog on the detail of the system and how it works.. if anyone's interested, that is.

Note: this is a personal blog and my views aren't necessarily those of the BBC, or anyone else for that matter. Ahem.

Twonk

I like Twitter. I find it more useful now than I imagined when I signed up for it three years or so ago. For me, its usefulness began when a few close friends of mine, and some of my family, started using it in earnest. Then it became a tool for just working out where people were and what they were up to, as much as it was for stalking celebrities, getting news feeds, discovering links and feeling plugged into the zeitgeist. As nonentities and companies started flooding the service, I stayed resolutely private and felt isolated from it all, because I had my 30 or so followers who I also followed and who largely all followed each other too. More broadcast-y than instant messaging and less arduous than blogging, Twitter's always had an IRC channel vibe for me, which I really like.

(Incidentally, I dumped Facebook three or four months ago. Best move I've ever made)

Recently, though, I started wondering if I've been missing out on the other ways of using it and consuming the nonsense it could potentially push to me, so I renamed by private account and created a public account as well, so I could use that to follow the few public figures and news bots I'm vaguely interested in, and possibly start talking about non-whingey work stuff in a more involved way. The private account remains for all necessary cathartism and gossiping.

So what do I make of the wider Twitter universe, outside of my imposed bubble? Well, first impressions aren't necessarily good. I see a lot of:

• Self-appointed 'social media expert' types, grinning out from their avatars in a state of perpetual self-satisfaction, waiting for the gravy train to pull in at the next stop
  
• Comedians continually whoring out their latest non-hilarious endeavours
  
• Spates of irritating hashtag trends - #movieswithunemployedpeople, etc
  
• Masses and masses of linkspam, which is what tumblr is bloody for
  
• General pushy careerist types being quite mean and rude about other people
  

And so on. The linkspam thing is the most interesting one, though. I've seen it happen every time a new fad comes along - users just leap on it and funnel everything they want to say, share or play with through it. Maybe Twitter is just too simple, too useful to people to stay within its original constraints, and an inevitable part of mass acceptance are the antipatterns above (and more).

Notes on making iTunes play nicely with an Airport Disk and Airtunes

OK, so six months later, along comes another post. This time it's practical, and a result of the experiences I had recently in trying to downsize my CD collection and shift all of my music onto a shared NAS drive. As it happens, our network at home is already based around an Airport Extreme sase station that supports the attachment and sharing of USB disks as network drives - so I purchased a cheap-arse Western Digital 1TB USB external drive, formatted it as HFS+ (non-journalled), shoved it into the back of the AEBS and away I went..

Setup: iTunes 8.2, Macbook 2nd Gen, OS X Tiger 10.4.11, Airport Extreme, Airport Express with Airtunes.

1. Telling iTunes what to do

Changing the location of the iTunes Library in the preferences and then using 'Consolidate Library..' is by far the easiest option for physically moving all your songs. See Apple's official guide here. All your metadata is preserved and it all Just Works. One pain - you have to copy over the network, so ideally, connect your Mac to the base station using a network cable rather than using wireless for this move as it'll be much, much quicker.

An alternative, do-it-yourself approach that gets you more control over where your music lives is manually editing the paths in the iTunes Music Library.xml file using a text editor. You still have to delete the actual library database and re-import the XML file into iTunes. This way you can connect the eventual NAS drive to your Mac's USB port, copy all the music across, connect it to the AEBS and hammer iTunes into doing the right thing by changing the paths in the XML file. However, for novices or people without much time on their hands, I wouldn't recommend this approach. YMMV.

2. Don't put everything on the NAS

Keeping your album art & iTunes databases on your Mac's local disk seems to be the smart option regardless of how you migrate your music (the Consolidate Library method works this way). You can't shove the whole of iTunes' working files on the NAS and expect that to be speedy - or, even worse, assume that you can share a single library between two instances of iTunes in this way. This is because iTunes continually updates its database files as it's working.

3. Check your wireless network

The performance of your wireless network is by far the biggest factor in making iTunes not suck donkey's balls with this setup. Connected via wired Ethernet, you should find day-to-day performance (including CD ripping) pretty much on par with having the library on local disk, with maybe a second or two of lag occasionally.

Over Airport, though, it's a different story. If your Airport network is not performing well - something you might not usually realise if all you ever do is access and download files from the Internet, as your Internet connection will always be much slower than the speed your wireless network ought to be capable of.

When iTunes attempts to play a file, it will buffer a certain amount of the file into memory before it starts playing it, so achieving low network latency is essential for this process to appear snappy enough to not be annoying - nothing's going to irritate you more than iTunes beachballing every time it moves from track to track.

Things to check:

1. Your wifi mode - 802.11b/g/n ?
   
2. Interference from nearby networks - use iStumbler or a similar wireless network tool to see if certain channels are crowded with networks, and move your network to a less busy channel.
   
3. Overall signal strength - is your AEBS positioned optimally? Are all wireless clients getting a good signal? Use the statistics function of Airport Utility (hidden away in the 'Advanced' tab of 'Manual Setup') to see how your Airport network is performing and be prepared to move the base station(s) around to achieve better speeds/less interference.
   

4. Airtunes bumpiness

Solving point 3 should also solve any AirTunes performance issues you might be having. AirTunes works by streaming a losslessly-encoded version of the music stream to the Airport Express base station, which in itself should not tax a modern 802.11g (or better) network but a weak signal can increase drop-outs and stuttering. By moving the AEBS a little higher and the Express a little nearer to get the best signal-to-noise ratio, I've had it working pretty seamlessly.

An edge case to be aware of is if you have any music encoded in any non-iTunes native codecs - that is, any other than MP3, AIFF or AAC - it cannot be sent over AirTunes. Even through iTunes can play Ogg Vorbis, etc via Quicktime - it can't send files encoded in these formats over AirTunes, which is annoying when files such as these show up in playlists. Even more annoying is that if it's streaming to a set of remote speakers and encounters such a track, iTunes doesn't just skip it - it'll play it out loud on the local computer, whilst the remote speakers go silent! This is annoying, and I can't help wondering if it's more of a 'feature' to get you to migrate all your music into Apple's own formats rather than free, open-source codecs like Ogg Vorbis. That said, if I was that much of a militant open-sourcer I wouldn't be using iTunes in the first place, so I can't get too righteous about it.

5. Some things are just going to take longer

Wireless just isn't as good as wired for some things. With the convenience of sending data around wirelessly comes certain trade-offs. iTunes may feel less snappy when navigating your library and jumping between songs, as it waits for the wireless disk to catch up. This will annoy you for a week or two, but then the great adaptive nature of the human brain kicks in and you'll just get used to it.

Anyway, I hope that's of use to some people struggling to get this sort of setup working. It's still a little way off being the perfect wireless-music-anywhere nirvana I was gunning for, but it does feel like I've freed up my listening a bit.

Start

Given how infrequently I bother posting here, I thought there'd at least be some amusement value at posting at the very beginning of the year. True to form, I'm already off by one day.

Anyway, happy 2009. Being thousands of miles away from home gives a strong feeling of dislocation, not just physically but psychologically. I lost track of the weekdays almost immediately when we arrived, irrelevant as they are when you've got nothing to do. And the fact it's a brand new year hasn't really registered yet. 2008 was difficult in many ways, so I'm not exactly sad to see the back of it and have the chance to use the division of a new year starting to attempt to try and sort out various things.

As if by chance, today marks the tenth anniversary of me starting my first professional web development job. In 1998/9, jobs that had anything to do with the internet were both rare and fuzzily defined at best - so when I landed a job as a 'Webmaster' at a forward-thinking advertising agency in Newcastle, 6 months after leaving university and kicking around in temp jobs, I was pretty excited. In retrospect, I was very lucky, in the right place at exactly the right time. To think I'd end up where I am now, doing what I'm doing, would have been certifiable insanity.

So, whilst I don't exactly yearn for the days of administering hundreds of sites being served from a tired Powermac running System 7.6 on the end of a dial-back ISDN line, and building them out of handcranked HTML, Lasso and Filemaker Pro (shudder), I'm grateful for the in-at-the-deep-end head start I got on things in my 18 months there.

We don't arrive back in the UK until Tuesday, so I've still got plenty of time to think about what happens this year. There are vague plans, and whole areas where I just haven't done the thinking yet. Meanwhile.. ten years. Blimey.