Durrants Ltd, founded in 1880, is the UK's leading media monitoring company with over 6000 clients from every sector of the economy. The company monitors 10,000 UK sources continuously (plus sources from over 100 other countries) and scans over 1.5 million news pages a month. With such a high volume of incoming content it is imperative that Durrants is at the cutting edge of news monitoring technology. With this in mind they asked text search technology experts Flax to completely replace their existing search engine with an open source solution.
The project had several main aims: to significantly reduce the hardware budget, to reduce the load on Durrants' existing relational database, to improve the quality of results delivered to clients and to provide a stable platform for the development of new features.
The new search engine had to match as closely as possible the features of the existing system, so that Durrants' valuable database of client keywords could be re-used. Durrants were also keen to add new features to improve further the quality of results sent to clients.
Stephen Wicks, CTO of Durrants, said "We considered several leading commercial vendors of search software, but eventually decided that Flax's open source solution was the most appropriate in terms of performance, flexibility and cost. The Flax team's experience in developing custom search applications was also an important factor, as we knew this project would present unique challenges in terms of scaling and accuracy."
The new system, built on the Xapian search engine, supports a detailed query syntax that supersedes and enhances the previous engine's features. Boolean operators, fuzzy matching, word weighting and support for punctuation are all available. The system even allows for errors in scanned content, with matching for known OCR errors. Working closely with the team at Durrants who build client profiles, the Flax team identified cases where certain requirements produced large numbers of false positives and have reduced this to a minimum: in some cases where the previous engine produced 95% false positives, results from the Flax solution are 95% correct with only 5% false positives.
Stephen Wicks of Durrants said "The new Flax system runs on virtual machines, and we're currently seeing a reduction in hardware budget to 15% of previous levels. As Durrants continues to grow as a business we now have a predictable and cost-effective way of scaling up the search engine – something that would not have been possible with a closed source solution. We can also easily develop new features to support our internal teams".
The Flax team continue to work with Durrants on enhancing and improving the search solution. Charlie Hull of Flax says "The Durrants project is a great example of how open source search technology can be used to solve problems outside traditional site or intranet search. We've used a search engine as the foundation for a complete media-monitoring solution, and open source software has allowed us to build a highly scalable, accurate and extensible system. We look forward to our continuing relationship with Durrants and to helping them maintain their market-leading position."
