Over the last few months you’ve probably seen a number of announcements about how OpenCalais has been chosen by one organization or another to support its business.

In a number of recent meetings I’ve been asked the (very fair) question, Why OpenCalais and not one of the other . . . → Read More: Why OpenCalais?]]> (Re-purposing a post of mine from www.opencalais.com)

Over the last few months you’ve probably seen a number of announcements about how OpenCalais has been chosen by one organization or another to support its business.

In a number of recent meetings I’ve been asked the (very fair) question, Why OpenCalais and not one of the other entity extraction services out there?

Given that the question seems to be coming up more often as the number of extraction services increases, I thought I’d get my best understanding of why many major players we’ve announced (and an equal number we haven’t) have chosen to go with OpenCalais. And – at the end – I’ll mention a few reasons why others haven’t chosen OpenCalais.

So, in no particular order, why do organizations choose Calais?

Thomson Reuters

OpenCalais is provided by Thomson Reuters – the largest professional information organization in the world.

If you’re interested in kicking around some semantic technologies in your spare time this doesn’t really matter. If you’re incorporating those technologies deep within your business – or, as is true with many users – actually building a new business on top of them, this becomes pretty important. Basically – you need to know that the service is going to be there for you.

Facts & Events

With the increase in structured content assets like Wikipedia / DBpedia, it’s become pretty easy to knock out a basic entity extraction tool. And – while we like entity extraction as much as anyone else – it’s really just the tiniest starting point in what you can and will need to do.

OpenCalais extracts a wide range of facts and events from unstructured content and lets you know what’s happening in your content – not just tags for things.

• Facts are things like “John Doe is CEO of XYZ Corporation.”
• Events are things like “XYZ Corporation today announced that it would acquire ACME Corporation.”

OpenCalais is the only service that does this in a production-strength manner.

Reliability

OpenCalais stays up. It’s hosted in mirrored data centers thousands of miles apart from each another. It’s monitored 7*24. It basically doesn’t go down – even during system upgrades and maintenance. We stopped adding 9s after we got beyond 99.99% uptime.

Accuracy

We’ve been building the tools underneath OpenCalais for over a decade. They’ve been used by hundreds of organizations and many many thousands of end users. One of the things we’ve learned is that accuracy matters. While no NLP system is perfect, we’re convinced ours is the best and we have some ideas in the pipeline to increase accuracy even more.

Integration

We basically focus on providing great semantic plumbing. But we know that not everyone wants to be a plumber. We’ve worked to integrate (or motivate others to integrate) OpenCalais with a wide range of tools including Drupal, WordPress, WordPress Multiuser, Oracle, Lucene, Coldfusion, Flash, Firefox, Prolog, Lisp, Django, Java, PHP, Python, Alfresco, Perl, .NET, Ruby, TopBraid and a few others.

From content management systems to language-specific libraries – there are lots of ways to get started quickly.

Linked Data

We’re serious about Linked Data. We’re also worried about the proliferation of incorrect links and the effects of link rot. So, rather than just pointing to Linked Data assets out on the cloud and risking that they’ll go stale, we host our own Linked Data cloud, which is kept up to date with both Thomson Reuters contributed content as well as regularly validated links to other sources such as DBpedia, Freebase and others.

SocialTags

Pure semantic extraction is great – but sometimes you need more. If you’re writing about Porsches and Ferraris you’d probably like to have categorization concepts like “sports cars” and “automobiles” returned to you with your semantic metadata. OpenCalais does this via our ever-improving SocialTags concept tagging capability. It’s good now, and it’s going to get a lot better soon.

Focus

OpenCalais is here to provide great semantic plumbing. We’re not trying to sell ads. We’re not trying to provide the prettiest decorations for blogs. We build the plumbing – you architect the solutions.

Now, in a spirit of transparency, here’s why some people don’t choose OpenCalais:

Languages

We’re great in English and okay in French and Spanish (we extract entities but neither facts nor events in these two languages). We intend to implement more languages in the future – but for the time being we’re concentrating our efforts on improved functionality and accuracy in English.

Complexity

OpenCalais isn’t a simple tagging tool. What it returns to the calling application is a reasonably complex RDF construct. It takes a little time to get up to speed on RDF and how to use it in your applications. We think it’s worth it because it’s the most flexible and powerful format we know of.

Performance in Knowledge Domain ‘x’

Where ‘x’ is fashion or square dancing or rugby. OpenCalais is optimized for performance in the general world of business – that’s where we excel.

We have extended OpenCalais to take steps in other areas (such as sports, media, etc.) – but if you need deep semantic extraction capabilities related to protein binding – there are better places to look.

The Gist: Release 4 of Calais will be a big deal. In that release we’ll go beyond the ability to extract semantic data from your content. We will link that extracted semantic data to datasets from dozens of other information sources, from Wikipedia to Freebase to the . . . → Read More: Life in the Linked Data Cloud: Calais Release 4]]> (Re purposed from the blog post on http://www.opencalais.com/node/9501)

The Gist: Release 4 of Calais will be a big deal. In that release we’ll go beyond the ability to extract semantic data from your content. We will link that extracted semantic data to datasets from dozens of other information sources, from Wikipedia to Freebase to the CIA World Fact Book. In short – instead of being limited to the contents of the document you’re processing, you’ll be able to develop solutions that leverage a large and rapidly growing information asset: the Linked Data Cloud.

The goal of this post is just to give our community a heads-up to start thinking and planning.

During the course of 2008 we’ve had three significant releases of Calais, with additional point releases nearly each month along the way. We’ve added new knowledge domains, improved performance, delivered integration with a range of tools and developed new user-facing applications. It’s been a year of amazing growth in our developer community and the capabilities of the Calais service.

While every previous release has accomplished something significant, Release 4 is going to introduce something that we think is game changing – and that’s life in the Linked Data cloud. It’s important enough that we want to give all the members of our community time to think about it, prepare for it and get your brains in gear on how you might use it.

Every release of Calais up to this point has focused on meeting the need to extract semantic information from text. Release 4 builds on this by creating the ability to harvest the Linked Data cloud using that semantic data.

For this all to make sense we need to introduce a few things. If you already know about de-referenceable URIs and the Linked Data cloud – skim ahead. If not – please take a moment to ingest the background you need.

When you send text to Calais it returns several things: entities, facts, events and categories. For purposes of today’s discussion we’re going to focus in on entities. Entities are just what they sound like – they are things. Some specific examples are people, companies, organizations, geographies, sports teams and music albums.

When Calais extracts an entity from your text it returns (at least) a few things. It tells you the name of the entity and it tells you what type of entity it is. Unlike other extraction services we don’t just return a list of things – Calais tells you it found a thing of type=Company and a value=IBM or type=Person and value=Jane Doe. But – there’s something else Calais returns that hasn’t meant very much up until now: it returns a Uniform Resource Identifier (URI) for that entity. There’s nothing magic about URIs – they are simply a unique identifier for every entity that Calais discovers. Here’s an example (it’s not pretty) of what the URI for the Company IBM looks like:

d.opencalais.com/comphash-1/7c375e93-de13-3f56-a42d-add43142d9d1

Well, that doesn’t look very useful does it? If you were to pull up that URI (when Release 4 is out) all you’d see is RDF with links to places called DBpedia and Freebase and Reuters. But keep those links in mind: they’re the key to a whole new world.

Linked Data is the name of a movement underway (not too surprisingly, initiated by Sir Tim Berners-Lee) that sets a standard and expected behavior for publishing and connecting data on the web. This isn’t about publishing web pages – this is about turning those web pages into data that’s accessible to programs to work with. We’ll give you a quick example to make it real: Wikipedia is one of the single largest sets of information across a broad range of topics in the world. It’s really great if I’m a person who’s casually looking for information on a particular topic – but it’s not so great if I’m a computer program that wants to use that data. Why? Because it’s formatted and organized for people – not computers – to read.

But Wikipedia has a twin – in fact a Linked Data twin – called DBpedia. DBpedia has the same structured information as Wikipedia – but translated into a machine-readable format called RDF and accessible via the Linked Data standards. And, Wikipedia is not alone. A growing cloud of information sets from DBpedia to the CIA World Fact Book to U.S. Census data to Musicbrainz – and many others – is becoming available. What’s important is that this cloud is 1) growing, and 2) interoperable. There are “pointers” from entries in DBpedia to entries in Musicbrainz and back to entries in Geonames – it’s another big Web – but this time it’s a Web of Data.

So – lots of words and arcane concepts. Let’s try to bring it all together into something that makes sense. We’ll put one sentence out there – and then we’ll give a few examples.

Beginning with Calais Release 4 you and the programs you develop will be able to go from many of the entities Calais extracts directly to the Linked Data Cloud.

A simple example:

I want to process today’s business news. For each article I want to extract all of the companies mentioned – but only if the article also mentions a merger or acquisition. I am only interested in companies whose headquarters (or those of their subsidiaries) are located in New York State. Do all of that and give me a widget for my news site titled “Merger Activity for NY Consulting Companies”. And oh, by the way, this isn’t a research project – I want you to do it real time for the 10,000 pieces of news I process every day.

How would you do that? Option 1 is to hire a bunch of researchers, give them a fast internet connection and teach them to type very very fast.  Option 2 is to write some code that looks like this:

For each Article

Submit to Calais, get response

If MergerAcquisition exists then

For each Company

Retrieve Calais Company URI, extract DBpedia link

Send Linked Data inquiry to DBpedia, get response

If CompanyIndustry contains “Consulting”

If CompanyHeadquarters = “New York”

Put them on the list

For each subsidiary

Send Linked Data query to Dbpedia, get result

If CompanyHeadquarters = “New York”

Put them on the list

(lots of endif’s)

Print the list

That really is a pretty straightforward example. How about companies in the news with at least one subsidiary doing business in an area that the CIA Factbook considers dangerous? Or books released by authors who attended Harvard who live in Ohio? Or … . We think you get the idea.

So. The summary. The combination of semantic data extraction (generic extraction, tags, keywords won’t do the trick) + de-referenceable URIs (entity identifiers you and your programs can retrieve) + the Linked Data Cloud = amazing stuff.

We’d like you to start thinking about it.

Calais is an initiative by Thomson Reuters to provide one of the core building blogs of the Semantic Web: semantic metadata generation. At the core of Calais is a web service that ingests text content, analyzes using natural language processing, . . . → Read More: Calais Ecosystem: Calais for Drupal]]> Time to start talking about great tools that have been built on top of Calais.

Calais is an initiative by Thomson Reuters to provide one of the core building blogs of the Semantic Web: semantic metadata generation. At the core of Calais is a web service that ingests text content, analyzes using natural language processing, machine learning, lexicons and statistical analysis to extract semantic data from the text and return it as structured information – primarily as RDF. Enough about Calais – I’ll write a big long post about it in the near future.

One of our biggest goals with Calais is to develop – or help others develop – tools that translate this from geekdom to real world usability. One of the areas of focus for that is to integrate Calais within a variety of content presentation and management platforms. There’s a wide range of those platforms – but Drupal stands out as being one of the fastest growing ones in the mid-tier publishing space.

Shortly after Calais was released two members of the Phase2Technology team – Frank Febbraro and Irakli Nadareishvili just stepped up and made it happen by building the Calais Modules for Drupal.

These modules provide a strong building block for construction semantically-enabled Calais applications. The modules provide seamless integration between a range of Drupal node types and the Calais service.

From their description…

The Calais module lets you configure which Content Types you want to request Calais metadata on update. The entities returned can then be automatically assigned to vocabularies related to the Content Types, or it can only suggest terms based on the Calais metadata and allow the user to select the terms you want to associate (think of del.icio.us recommending tags). A flexible set of hooks allows 3rd party modules to make modifications before or after Calais terms have been applied. There are many level of configuration and integration and this is just the beginning.

The Calais Tag Modifier module allows for basic blacklisting of tags, so that you never get terms suggested that you don’t care about. The term substitution mechanism also allows you to modify returned metadata before it gets assigned or suggested.

Beyond what Phase2 has developed to date, the Calais Initiative and Phase2 have agreed to work together over the coming six months to release a series of significant enhancements built on the Calais modules. These enhancements will be oriented toward even tighter integration of Calais with Drupal and providing a comprehensive Calais-powered set of capabilities such as topic hubs and other publisher-oriented features.

So – that’s the description: here’s what’s cool. One of the hottest publishing platforms in the world is integrated with Calais. Users can get access to Calais’ capabilities with essentially zero effort. And – all of this was built buy two highly motivated guys that saw a need and just moved in and got it done.