Perl on Rails – Why the BBC Fails at the Internet
By seb
Perl on Rails is a project by the smart chaps over in BBC Audio and Music Interactive that replicates the Ruby On Rails MVC framework in Perl. They’re obviously rather proud of themselves, and I understand that internally the project is making waves. Whilst I applaud the technical achievement of the individual developers, I deplore the situation that has forced them to do this.
The problem is that the BBC doesn’t control its own technical infrastructure. In an act of staggering short-sightedness it was outsourced to Siemens as part of a much wider divesting of the BBC Technology unit. In typical fashion for the BBC, they managed to select a technology supplier without internet operations experience. We can only assume that this must have seemed like an acceptable risk to the towering intellects running the BBC at the time. Certainly the staff at ground level knew what this meant, and resigned en masse.
Several years later this puts the BBC in the unenviable situation of having an incumbent technology supplier which takes a least-possible-effort approach to running the BBC’s internet services. In my time at the BBC, critical operational tasks were known to take days or even weeks despite a contractual service level promising four hour response times. Actual code changes for deploying new applications were known to take months. An upgrade to provide less than a dozen Linux boxes for additional server capacity – a project that was over a year old when I joined the BBC – was still being debated by Siemens when I left, eighteen months later.
The BBC’s infrastructure is shockingly outdated, having changed only by fractions over the past decade. Over-priced Sun Enterprise servers running Solaris and Apache provide the front-end layer. This is round-robin load balanced, there’s no management of session state, no load-based connection pool.
The front-end servers proxy to the application layer, which is a handful of Solaris machines running Perl 5.6 – a language that was superseded with the release of Perl 5.8 over five and a half years ago. Part of the reason for this is the bizarre insistence that any native modules or anything that can call code of any kind must be removed from the standard libraries and replaced with a neutered version of that library by a Siemens engineer.
Yes, that’s right, Siemens forks Perl to remove features that their engineers don’t like.
This means that developers working at the BBC might not be able to code against documented features or interfaces because Siemens can, at their sole discretion, remove or change code in the standard libraries of the sole programming language in use. It also means that patches to the language, and widely available modules from CPAN may be several major versions out of date – if they are available at all. The recent deployment of Template Toolkit to the BBC servers is one such example – Siemens took years and objected to this constantly, and when finally they assented to provide the single most popular template language for Perl, they removed all code execution functions from the language.
So talented, underpaid, and frustrated software engineers at the BBC are forced to make a decision. Either they can produce websites using static HTML, and make a few remote calls to limited Perl functions, decorating their page with SSIs, or they can fight against a reticent and incompetent technology supplier to make use of a crippled and outdated language on servers that more than likely are unable to meet the capacity requirements of a dynamic application being used by the BBC’s audience. Software engineers at the BBC must become masters of the sleight-of-hand, using every smoke and mirrors tactic they can to conjure the appearance of dynamic websites, not exactly what you would expect from one of the largest media corporations in the world. Oh, and if you’re an external agency working for the BBC and hoping to write a new application or build on technologies that the rest of the world has taken for granted for the best part of a decade, you might as well forget it. There’s only one externally available development server, and it’s not in synchronisation with the live environments.
It doesn’t have to be this way. If, instead of forcing its teams to waste valuable license fee payers’ money on duplicating existing free software, the BBC decided to take control of its technical infrastructure and provide a viable platform for complex, dynamic applications, then that creativity, effort, and time could be directed at making more of the kind of applications that make the BBC great.
Some work is already progressing in this direction. A large part of the BBC’s Creative Futures project is what the BBC calls “BBC 2.0″ (often mistakenly referred to by executives and television-types as “Web 2.0″
. The last I heard this was planning to deploy an architecture based around Java, Tomcat, Hibernate, Velocity, and MySQL. Whilst I disagree with the choice of technology for many reasons, this is at least an important step in the right direction for the BBC – as long as they exert control over the infrastructure from end to end.
It’s a ridiculous situation, and I know that many talented and respected technical staff have left the BBC in the past few years citing frustration at the insufficient technical infrastructure, and the inability of both Siemens and BBC management to keep up with the pace of technological change. Unfortunately, unless something dramatic changes with the upper levels of BBC management to recognise the nature of the problem, it’s a situation that will remain the status quo for a long time to come.
