Category Archives: Text Mining

Andreas Blumauer

Why SKOS thesauri matter – the next generation of semantic technologies

by , Category: Search Engines, Semantic Web Applications, Text Mining, Tools & Software

As a matter of fact still a lot of “semantic technologies” are around which do nothing else than pure statistical analysis of text. Sure, this is better than simple full text search but there are still quite a lot of opportunities to improve search, especially when it comes to more sophisticated applications like “similarity search”, the search for similar documents to enable cross-reading or recommendation systems.

Providers of first generation semantic technologies calculate rather basic “semantic networks” by co-occurency analysis which results sometimes in  disappointing results. Bearing in mind that Google just bought a company (“Google buys Metaweb“) which has been working on one of the largest knowledge bases in the world, we could assume that some of the last miles towards a semantic search engine can be achieved by applying thesauri or other structured knowledge bases.

A demo application was recently developed by PoolParty team where one can find out how thesauri will improve search results on top of second generation semantic technologies. With PoolParty SKOS based controlled vocabularies can be managed and also can be enriched with linked data. PoolParty Tag & Content Recommender analyzes virtually any text or website to recommend corresponding tags, concepts from (in this case) STW (Standard Thesaurus für Wirtschaft), DBpedia and respective articles from Wikipedia.

STW which was developed by the German National Library of Economics (ZBW) provides vocabulary on any economic subject: about 6,000 standardized subject headings and about 18,000 entry terms to support individual keywords.

This background knowledge is used in this demo app to improve the search for similar documents dramatically:

Similarity between two documents can be calculated not only on a key-phrase basis but also on a rather conceptual basis. Even if two documents do not have one single word or phrase in common they can be identified as “similar documents”.

This can be achieved because thousands of important relations between economic subjects are represented in the domain specific thesaurus. Thus, in this special case best results are achieved with documents from economics (for instance from Econstor) but of course for other recommender systems thesauri from other domains can be used instead of STW.

Nevertheless, also this approach can be improved and this development is underway: SKOS thesauri enriched with Linked Data do an even better job. This kind of third generation semantic technologies are currently developed by LASSO project and LOD2 project, two innovative projects in the area of linked data and the semantic web.

Andreas Blumauer

Calais, Zemanta or textwise?

by , Category: Mashups & Web services, Text Mining

Beside W3C´s Linked Data Initiative, it were semantic services like Calais, Zemanta or textwise which have made the advantages of the Semantic Web visible for a broader community in the last few months.

Each of those services follow a slightly different approach, but in a nutshell: They all offer an API to provide “similarity search” around social media or also to enhance enterprise information management.

Like a magic bullet those services offer a relief from information overflow and seem to become kind of a “semantic web killer application“.

If you´re familiar with one or many of those services, drop a comment and let us know, what you´ve been experienced so far, or also if you can think of any applications or further developments you would like to see around these kind of services.

If you are not familiar with this stuff, for a quick demo go to

The widget uses text from this blog to calculate similar stuff from the web.


Related articles by Zemanta
Reblog this post [with Zemanta]
Thomas Thurner

Keep the Semantic Web trusty

by , Category: Corporate Semantic Web, Mashups & Web services, Politics, Privacy & Information Ethics, Text Mining
Tim Berners-Lee at a Podcast Interview
Image via Wikipedia

In recent days – here at Semantic Web Company – we have had a lot of discussions on how the future of the Semantic Web (name it Web3.0 if you like) will develop. Several stakeholders on the future of the Semantic Web see already, that also a potential danger will come along with the technical realisation of the web3.0: This is the present possibility to create applications and mashups with semantic technologies that are a real drain on privacy and information ethics. Without an underpinning discussion about the ethical framework within technolgies like linked data, text-mining, biometric-systems and geo-systems in combination with the web of data, the whole domain is in danger to be doomed like genetic engineering some years ago.

It’s crucial for the public opinion on the Semantic Web, to adress the immanent risks regarding privacy and ethics. In this context I’ll see also Tim Berners-Lee‘s statement yesterday: “W3C wants to help make sure data use is appropriate,” he said. Berners-Lee, who is director of W3C, said in an interview on Wednesday that the teams working on the Semantic Web project are making sure that privacy principles are included in its architecture: “The Semantic Web project is developing systems which will answer where data came from and where it’s going to — the system will be architectured for a set of appropriate uses.”

Maybe it’s an important step in keeping the further development of Semantic Web trusty in the eyes of public opinion, that the W3C has privacy and information ethics on their agenda and persons like Berners-Lee stand with their reputation for it. But it is also crucial to build this awareness on the corporate side. Only if everyone within the domain follows a common ethic understanding we have a public opinion, which is on the future potential of the Semantic Web, and not in fear of the same.

Reblog this post [with Zemanta]
Thomas Schandl

Tom Tague on Open Calais 4

by , Category: Companies & Institutions, Linked Data & Open Data, Mashups & Web services, Text Mining, Tools & Software

The recent release of Open Calais v4 offers excting new possibilities by making a great contribution to Linked Data efforts.

Previous releases of Thomson Reuter’s Open Calais web service already produced promising results by extracting named entities, facts and events from user submitted contet – especially news articles. Now these extracted concepts come with an URI and are linked into the LOD cloud – specifically to DBpedia, Freebase, Musicbrainz, CIA world fact book and others. Tom Tague

On this occasion Tom Tague, vice president of the Calais creators ClearForest, answered questions the Semantic Web Company had about the goals of Open Calais. 

The latest release of Open Calais produces metadata conforming to linked data principles. You provide this great service free to everyone via your web service.
What led to that decision, which benefits are there for Thomson Reuters?

Thomson Reuters has the largest trusted content sources in the world – but we don’t have all the content in the world. We believe that the world is going to want to integrate highly managed and trustworthy content assets such as those provided by Thomson Reuters with the low latency, highly diverse content exploding on the web. Fundamentally what we’re trying to achieve is nearly effortless interoperability of content between any two partners – Calais enables this by extracting the semantic metadata buried in your content but then takes it a step further. By linking those semantic elements to the Linked Data cloud we are setting the stage for the dramatic enhancement of any content source – and we hope that many will choose Thomson Reuters as one of the methods for enhancing that content.

It seems with Open Calais you use a hybrid business model, which integrates end users in a form of enterprise collaboration into value creation.
Do you think such a business model is viable during the long run and what are your experience so far?

As of right now Calais isn’t truly a “Business”.  It’s a strategic initiative that’s setting at least a piece of the stage for the Linked Content Economy. Our goal is to understand how this new content economy is going to involve and to make certain that we have a leadership position as it moves from a concept to reality.

Apart from the thousands of users submitting content to Open Calais, there is also a community of developers making their own applications around your core app. How important are the social dynamics of the Open Source community for the success of Open Calais?

Extraordinarily important. Calais is a web service – which means it’s relevant to about 0.0001% of the population. We are absolutely reliant on the creativity, energy and domain expertise of our developer community to translate Calais from a technology to an end-user relevant capability. And – as a user-driven project we also rely on our developers and users to give us feedback on what they like, what they don’t and where they think we should head.
What are your plans regarding to offering your service in German?

We hope to get there in 2009. We’ve released basic French and are gearing up for additional languages in the coming year.

Thank you, Tom, for your answers! We look forward to more applications like Semantic Proxy and Linked Facts that demonstrate the great protential of the Calais engine.