WARNING, this is a long post.

On 3 December I will have been at the M&G Online for a full year, so I figured it’s time to do a retrospective and reflect on some of the highs and lows.

Before I joined I had an idea in my head about what I would be doing and where my attentions would be focussed. Primarily I expected to be working on new projects and preparing for the redevelopment of the M&G Online’s main news site. My discussions with Matthew Buckland before joining were detailed and lengthy so no surprises there. In the back of my mind however, I was also expecting to do a lot of work converging the print and online units of the company and I was generally expecting to remain quite strictly within what one would traditionally define as “online newspaper” type of work.

The way we approached my work from the outset was to create a list of project ideas on a very abstract level. So, for instance, there were items that simply said “local version of Digg” and so on. There were 21 items on the list and about a third were actually related to newspaper business in any traditional sense. The plan was that I would bide my time while the company sorts out some of its internal issues by prototyping some new ways for a media company to interact with and leverage its existing audience. Once everything was in place we would start the big project, which is the main news site redevelopment. What this meant was that I would be the sole developer on these projects until I started hiring for the big one.

For those of you who don’t know me, I am a bit of a mercenary when it comes to programming. I like to prototype in days rather than weeks and I am not beholden to a designer because I do all that stuff myself. With this in mind, we had the freedom to think very laterally about what we wanted to do. There was no red tape other than agreement between myself, Matthew Buckland and Riaan Wolmarans, the editor. The three of us see eye-to-eye most of the time and when we don’t we negotiate, quickly.

News in Photos

The first project I developed was not on the list and I didn’t really tell anyone I was working on it until it was done, mainly because it was over the December period and Matthew was away. Riaan knew about it but he’d asked for a slide-show widget for photos on the M&G site, he didn’t think it was going to turn into something on its own. The News In Photos was my first project and, honestly, the one I find the least interesting despite it being very popular with our audience. It incorporates several of the practices that have turned into staple modules for all the work we do now – folksonomies, audience contributions in the form of content submission and commentary, extensive logging of reader behavior that is most notable in the swarm, RSS for search and tags, as well as a public widget.

The way we generate the tag cloud on this site is different from the way we have done it subsequently, and that’s worthy of a few lines here. The tag cloud at the bottom of the front page is populated by user interest not volume of matching content. I can’t remember why I decided to do it this way but it has highlighted certain problems over time. Firstly, audience interest and what the newsroom deems newsworthy do not always overlap completely. On one particular day we had a photo of a PETA representative wearing a bikini made of lettuce. Understandably this was a popular photo but it skewed the tag cloud for a week. Secondly, as a web site matures and is indexed by the major search engines, the traffic to the archive starts to reflect the popularity of search terms rather than the editorial choices of the newsroom. News events in the past, for instance the Rugby World Cup, continue to take a strong position in the tag cloud. Since then we have generated tag clouds based on volume of content rather than traffic.

When it snowed in Johannesburg we saw the first spike in user contributions. We took over a hundred submissions of snowfall and used a lot of them. There was a corresponding spike in our traffic at the same time and we learned a valuable lesson about citizen journalism: for lots of photographic citizen journalism to take place you need a relatively slow-moving event to take place that affects a large group of people. This gives them the time to realise that taking a photo and submitting it to a web site is something they can do relatively safely. Few ordinary people, taken by surprise and facing great personal risk, are going to spend the time to take a snapshot to send to a web site. The snows here and the fires in California have been a great example of what is needed: a big event that lasts several days or hours, giving citizen journalists time to get out of the way of danger first and then begin taking photos as they grasp the enormity of the situation.

M&G Mobile

The Mail & Guardian mobile site was a quick project to complete, mainly because of the forced simplicity of mobile HTML standards, with big results. The service has been growing its traffic by about 35% month on month and is one of our fastest growing products. It has a few interesting features – the ability to personalise the content on the home page and the ability to do an internal archive search and we’re in the process of adding a few new features to it. During the development period Matthew Buckland made the crucial decision to publicize access to the site not via the URL but via an SMS and WAP push system – to get a link to the site, SMS mg to 32368. The net result is that the complexity of different phone models, browsers and web setup was by-passed almost completely. When you send the SMS you get a bookmark sent back to you that the each phone handles in its own way but has the same outcome: the browser opens the page without the user having to understand how to do this themselves. This has been one of the key usability features that has made the site a success.

Amatomu

On our project list was a line item called “SA blog aggregator” and the history of this site is quit interesting. One Thursday or Friday I was thinking about the next big project we’d do and was interested in something in the social media space. At the top of my mind was a site like Muti or Laaik.it but I was undecided, mainly because I didn’t want the M&G to bully its way into a space that was so heavily invested in by local entrepreneurs. I had a meeting with Mike Stopforth on the same day and we discussed it in some depth and he helped me make the decision not to go that route. In fact his suggestion was to create a local version of Technorati and this, of course, clicked with the item on our project list. I told him we were going to go ahead with it because we both left the lunch meeting thinking, obviously, different things. Whereas I went back and started work on Amatomu, he met with Justin Hartman and started the development process that led to Afrigator.

I spent that weekend building Amatomu and we started closed-group testing on the Monday – Afrigator followed a few weeks later. Regardless of the possibilities for us to work together rather than, seemingly, against each other the sites turned out to be quite different and Afrigator’s recent beta launch has extended that difference significantly. Amatomu is a hybrid between an aggregator and an analytics service, and it appeals to the narcissism of the blogosphere by displaying comparative rankings of local blog traffic. What it has also achieved is the creation of a rallying point for blogger activism and has helped bloggers understand where they fit in with the rest of the local blogosphere.

The site was and is spoken of highly – we got coverage in Maverik, Financial Mail, Fin24, Intelligence and literally hundreds of blogs locally and abroad. The politics of it has not been easy though. It doesn’t quite fit in with the rest of the M&G’s content offering and we were wise to keep the M&G branding to a minimum and let it run as a community driven service. There have been many times when I have wanted to incorporate editorial filtering into the site to allay concerns about quality and, luckily, Matthew Buckland has stopped me every time. Again this goes back to the conflict between what is popular and what represents quality. Often some of the best bloggers get relatively little exposure because what they have to say simply isn’t popular, despite being important reading.

I have taken a lot of criticism personally for my occasional culling of blogs that offer affiliate advertising links almost exclusively, or blogs that have been registered as part of a nefarious circle of search engine optimization sites. The key achievement, however, is the fact that the site has sent close to 3 million click-throughs to local blogs and I think this has helped the blogosphere grow in size and, to some extent, stature.

Bullardgate

Possibly the lowlight of my year, I spoke out strongly in defense of the blogosphere when David Bullard, the popular Sunday Times columnist attacked bloggers, saying they are the kind of people that would gun down their college mates. This came close after the Virginia tech killings and it hit a nerve. The problem was exacerbated by the insult caused to some of my friends and ex-students who were heading up the multimedia division of The Times. I learned a valuable lesson here, and should have followed Matthew’s advice when he said it just isn’t cool to attack your competition so violently. Mea Culpa.

Nugget

There is no underestimating the impact that having a baby boy in April has had on my life. He goes by the code-name of Nugget but his real name is Michael Liam and he is both a bundle of joy and a productivity killer. I can’t remember the last time I had proper sleep and have felt zombie-like since he was born. It threw into stark relief my reliance on working at home in the evenings to get my job done and my slowdown in production has been the topic of several discussions about my psychological state of mind. Luckily I have a very supportive work environment and have been able to work around this problem.

The positives far out-weigh the negatives though, as any father will tell you. The enjoyment I get from watching the little guy grow and learn new things – he got his first tooth yesterday – are immeasurable.

Amagama

I can express, with some emotional distance now, that the migration of Blogmark, the M&G’s blogging site to a new Wordpress platform called Amagama was a painful process. Technically it was one of the most difficult things I have had to do and it was completely thankless because the core community became alienated during the process and, on more than one occasion I was the recipient of subtle death threats. In our defense, we suspected this might happen but needed to change the spirit of the platform from a forum to a full-blown blog-hosting platform and we did this successfully. I often use this migration as a case study on how not to do things but I think that might be slightly unfaithful to our stated intentions when we started the migration. David Robert Lewis still hates me for it.

Thought Leader

First off, this is not about patting anyone on the back, it’s a sober reflection on the first three months of the Mail & Guardian’s Thought Leader blogs. There is an overwhelming sense, shared by myself, Matthew Buckland and Riaan Wolmarans, that we have achieved something we can be proud of. On a pragmatic level the three of us are never satisfied – this has almost become a principle that drives us forward, but Thought Leader has surprised us. This is not to say the outcomes were unintentional, just that there is still an element of surprise when things worked out the way you wanted them to, or better.

The history of the site
In January this year, at the end of the my first month at the M&G Online, we settled on a list of 21 projects that we wanted or needed to tackle. On that list was a line item for a blogging platform for our journalists. In my mind at least, this was one of the more daunting projects because of the amount of convincing it was going to take to get our journalists, already pressed for time, to offer the company more of it for free. In March I collated a list of features from Matthew and Riaan and I did a reasonably comprehensive survey of what other major newspapers were doing with their blogs, both locally and internationally. The Times had already launched their blogging platform but we ended up spending a lot of time looking at commentisfree.co.uk, a blogging platform run by our shareholders, The Guardian.

As we began to think about how to structure the site commentisfree acted as a reference point and you can see clear similarities between the two, especially in terms of the Contributors A-Z and, to some extent, the design.

The design process

Our design process often feels like we’re shooting from the hip, but its actually just a reflection our focus on the user experience. We start by designing, in as much detail as possible, the primary templates for the site with minute detail. We then look at it from the perspective of a user and start hacking until we have something that looks like a real web site with every single link, button or tab. Once we’re satisfied then we start thinking about we’re going to achieve what we see on the screen. The front-end is the primary driver of the development and the technical implementation is really just joining the dots. I have included the final screen we settled on for the home page here so you can what the starting point was for the development work.

To put this process in perspective, I think a lot of sites start out with a less clear notion of the front-end and this causes a chain of technical adjustments that have to be made after the first iteration. This project and process generally means the changes are minor tweaks, often to the stylesheets and block positioning rather than anything else. In the case of Thought Leader the adjustments between first presentation and going live took less than a day of work. This is not to say that I followed the designs religiously and there were a few things that changed during the development process. With a collective 35 years of web experience between me, Matt and Riaan we allow each other the leeway to make these kind of adjustments as long as the core concept remains the same. This is only possible with a small, highly experienced team thinking in the same way, I suppose. It works for us.

Building the site

There were four possible routes to implementing the site as it was designed: use Wordpress Multiuser, use Lyceum, use Wordpress single user or build a custom blogging platform. I don’t like Drupal so that wasn’t an option. We eventually settled on a single-user implementation of Wordpress for a variety of reasons:

1. The clincher was that single user WP allows you to modify the URL pattern so we could have http://thoughtleader.co.za/bloggername – this was NB because we wanted each blogger to be able to promote a short URL.
2. In our past experience with WP Mu, aggregation of the content onto the home page and sections was going to be very difficult. Mu creates separate database tables for each blog, so maintaining synchronisation between what is aggregated and what is in each blog table becomes a nightmare. We also needed a moderation workflow process that would work much better in the single user version because we could simply use a plugin for this.
3. Much of the templating would have to be done on a site-wide level but WP single user has excellent author template functions that could be used to build the author pages, using the modified URL structure
4. Lyceum was an option but we found the WP single user solution quicker and it was something we were used to working with.
5. Building our own system seemed like reinventing the wheel, especially when it came to rewriting trackpacks and pingbacks and the under-the-hood stuff that makes blogging systems tick.

So we chose single user Wordpress, wrote a few plugins, hacked the backend admin to filter all content by user account unless you are an admin and built a new Wordpress theme that makes the site look different for ordinary Wordpress blogs. Ironically the new version of WP, which was released a month after we went live, would have saved us some time because a few of the features we built as plugins are built into the new version, like editorial approval on group blogs.

Editorial structure and impact

Riaan never lets us forget that Thought Leader has introduced a significant workload that didn’t exist before. He has had to get up at sparrows to start work on it every day so that he has time to continue his normal editorial work on the main news site. Another issue that is often forgotten is the work involved in managing the contributor invitation process. Contributors need to be invited, emailed, followed-up with and their accounts need to be created, photos need to be cropped, bios have to be subbed and so on. It has not been plain sailing but we now have over a hundred contributors on board and the discussion on the site is healthy.

Convergence

Thanks to the support from our print editor, Ferial Haffajee, Thought Leader was promoted to the M&G journalists as a extension of their work in the paper and the site was given two regular and substantial columns in the paper’s Comment & Analysis section. This has been the most substantial convergence we have seen between print and online so far and it has been hugely successful. We even have Thought Leader banners that we use at M&G functions to promote the platform and this highlights the massively collaborative process this site has become in the company. Somehow it has captured the imaginations of the print journalists and the editor and we, in the online team, have been reveling at the sudden excitement on the faces of people who are generally sceptical of what we do.

Coming Soon
For the past 2 months I have been working on a social networking site that is a few days away from public launch. I’ll talk more about this when it’s up.

M&G Web site redevelopment – the big one

The time has finally come for us to start rebuilding our main news site and this is the ultimate test. It’s one thing to mess around with sites that do half a million page impressions a month and something completely different when you’re deal with 5 to 6 million. We have decided to make the process very public, which means that I am blogging the progress as we go along, asking the community for advice, publishing the design proposals for comment and so on.

I have hired a team of two developers, who I believe to be excellent although unknown in the Web 2 space, who will do the bulk of the implementation work from now on. This frees me up to do my job description, which is coming up with new ideas.

So what has it been like working at a media company?

This is a tough question to answer because I don’t think my experience is a good indicator of what it would have been like at other companies. For one thing, the total lack of red-tape has given me the opportunity to do the things I wanted to do and within the proper time-frames. Working with Riaan and Matthew has been a pleasure because we share a vision and each has areas of expertise that the other doesn’t. There is around 36 collective years of experience between the three of us and it shows when it comes to decision-making. Big decisions are made within minutes rather than weeks and we instinctively know when we’re about to do something dumb, most of the time.

People constantly confuse the M&G for a big media company and there are some reasons for this – the paper is doing important work politically and the web site is the biggest single newspaper site in the country. We are, however, a small company and a lot of power lies with the individual. This last part is key to our ability to pursue a strategy of innovation.

During the time I have been at the company the newspaper has hit its all-time high in circulation and the web site its all-time high in local readership. Both of these achievements have been the result of good team work and a lot of commitment from everyone involved. My small contribution has been the creation of sites that have increased our local readership by about 12% over the year and the real work is only just beginning.

A word of thanks

Matthew Buckland is the best in the business, an endless tower of support and one of the toughest task-masters I have ever worked with. Riaan Wolmarans has and endless resource of patience and commitment to make the editorial side of my projects work. Daniella, my wife, has been a pillar of strength and has allowed me to sacrifice countless hours of our private time for the greater good. To all three of you, thanks for making my year at the M&G the biggest year of my life.