Downloads & Free Reading Options - Results

Imagine an automatic news filtering system that tracks company news. Given the news item 'FDA approves ciprofloxacin for victims of anthrax inhalation', how can the system know that the drug mentioned is an antibiotic produced by Bayer? Or consider an information professional searching for data on RFID technology - how can a computer understand that the item 'Wal-Mart supply chain goes real time' is relevant for the search? Algorithms we present can do just that. When humans approach text processing tasks, such as text categorization, they interpret documents in the context of their background knowledge and experience. On the other hand, conventional information retrieval systems represent documents as bags of words, and are restricted to learning from individual word occurrences in the (necessarily limited) training set. We propose to enrich document representation through automatic use of vast repositories of human knowledge. To this end, we use knowledge concepts derived from the Open Directory Project and Wikipedia, the largest Web directory and encyclopedia, respectively. In the preprocessing phase, a feature generator analyzes the input documents and maps them onto relevant concepts. The latter give rise to a set of generated features that augment the standard bag of words. Feature generation is accomplished through contextual analysis of document text, thus implicitly performing word sense disambiguation. Coupled with the ability to generalize from words to concepts, this approach addresses the two main problems of natural language processing synonymy and polysemy. Categorizing documents with the aid of knowledge-based features leverages information that cannot be deduced from the training documents alone. Empirical results confirm that this knowledge-intensive representation brings text categorization to a qualitatively new level of performance across a diverse collection of datasets. We also propose a new, knowledge-based approach for computing the degree of semantic relatedness of texts. ©2006 Microsoft Corporation. All rights reserved.