Real-Time News Syndication

Real time, fast payments, and A.I. to boot

Omni Digital are delighted to work with SWNS to produce the world's first real-time news syndication platform - a single source for news outlets to find copy, images and video.

Since its inception in 1970, SWNS has become the UK's largest news syndicator. With nine offices stretching from London to New York. They produce, verify and distribute original and engaging copy, pictures and video around the clock.

The project is co-funded by SWNS and Google's DNI innovation fund - and beta launch is scheduled for early 2017.

The system aims at beta to solve three core problems within the news industry:

  • delivery of a single unit of publication-ready news, comprising of images, copy and video
  • real-time attribution and sourcing of content
  • complete transparency of provenance of all news

The challenge this project proposes is massive: scale of storage, scale of day-to-day processing, and the complexities of A.I. We are proud to work on it, and proud to be associated with forward-thinkers like SWNS. 

History

The projects roots go back a fair few years - as the SWNS team grew and became the UK's largest news syndicator, they realised that the industry was changing and moving towards the content-is-king model.

Independently, Omni had been working on ideas around content attribution on the web - a tricky problem given the scale of data involved.

A chance connection was made in 2015, and a relationship was born. Omni ran a series of requirements discovery workshops SWNS team, and physically shadowed key members of staff as they went about their daily routines. Together, we realised very quickly the scale of a system which was needed far exceeded original expectations..

We made a joint pitch to the Google DNI fund in late 2015, and we were delighted to receive the joint funding we needed to get going in early 2016. 

The Scope

The scale of meeting the requirements of the service - both in terms of storage and processing - are what make this a truly exciting project. At beta launch, the system is expected to receive a gigabyte of raw news stories a day, including text, image, video and audio. This data must be processed over to extract all relevant file-level and content-level metadata upon submission - and the resultant blob of information must be passed against the output of all know client media outlets in order to establish evidence for a the usage of a given story. The size of memory and processing involved is staggering.

The most fundamental part of the successful delivery on this project is the A.I. - and ours and SWNS' joint ability to train it in time for launch. We'll need to answer some fundamental questions about the news stories themselves - who are the protagonists; what are the themes, who is saying what, to whom, where - and with this data we must demonstrate a better understanding of what news really is, and ultimately where it is being used.

First Steps to Success

At the time of writing, the project is four sprints in, and the basic foundations of the system have been laid. The system can store all stories and their constituent media - text, images, video and audio - and all file-level metadata is extracted automatically.

The architecture of the system itself is service-based - meaning that the whole system is in fact the output of several smaller services. Each service is currently hosted with Digital Ocean, and is built largely in Python. Django and PostgreSQL have been used for the services which provide storage and handle serving data to the users.

The project is run with an Agile methodology, and the first four weeks were focussed solely on establishing a basic understanding of the system and SWNS' own domain knowledge. Omni did this by running a series of workshops and testing assumptions made. We prototyped quickly in Python and Django using test-driven-development, and using our early prototypes we gathered feedback from SWNS and further refined our ideas.

As the project is Agile, we were quickly able to present SWNS with working software - albeit light on functionality - and this has helped build a sense of quick progress and trust in a quality product. Every two weeks, we release new functionality to a pre-production environment, and slowly the system will take its place within SWNS workflow. 

A key benefit of this approach is that many of the SWNS staff members will effectively grow their confidence in the system as the system itself grows in functionality. Our experience shows that this approach means that theres less of a system shock on launch for admin users, and the burden of training is greatly reduced.

Precisely because we ran the project in this way, we were able to identify and adapt to change and deliver far greater value with far more appropriate software than could have been planned.

The Next Phase

As the storage mechanism and overall architecture is now in place, our workflow will divide somewhat. On one hand, we will begin load testing the architecture against the expected volumes at launch, as well as against certain growth scenarios. Our aim here is to test our implementation, improve and refine where possible, and assess the lifecycle of the system.

At the same time, we will be moving on to build the billing and editorial workflow systems, which will allow SWNS admins to trace the provenance of all stories and pay contributors in the most open, transparent and swift way the news industry has ever seen.

Following this, the focus will be on creating the bulk of the A.I. suite and writing the spider and the story matching function, which will fetch global news articles for the system to attempt to create weighted matches with stories and assets loaded by contributors.  

Once the system is feature-complete, we will move into a series of sprints to create the most fast and fluid experience for our users. We will be able to effectively skin a working system, giving us the freedom to focus on the truly creative front-end and design work necessary to give a good user experience. 

As with all of our projects, we will start with an art direction phase, and as our design and UX is Agile too, we'll test our new front-end against real life users, and refine our front-end work with their feedback.

Launch itself is scheduled for early March 2017, and will be to private beta users. The launch itself will be soft, with many of the features of this system having been in use for some time by that point. 

Looking ahead, a system of this scale will open up unlimited opportunities for SWNS and its contributors, and Omni is truly excited to be working with them as their digital partner. We're beginning to identify some of the features we'll be rolling out post-beta launch, and it's safe to say they are even more exiting than the current challenges!

"Working with Omni has been an exhilarating experience.   Across our company we have been impressed with how quickly the development team got to grips with our workflow, and designed innovative solutions to make it work better. 

From our initial concept Omni have created a bespoke tool which is beyond anything we could have imagined by ourselves.“

Jon Mills, Picture Editor - and project lead for SWNS