Building Relevance Judgments without Pooling

A novel method is represented for building query relevance judgments without system pooling using subtopic clustering in conjunction with relevance feedback. The new method is referred to as Relevance Feedback Clustering (RFC). RFC builds on a previously developed method Sanderson and Hedio (2004) that uses relevance feedback to replace manual interactive query reformulation. RFC method is shown to be robust in building relevance judgments even in the absence of proper text processing tools, as demonstrated for Arabic with minimal processing, and to be consistently better than the one suggested by Sanderson and Hedio. Moreover, RFC was applied to the TREC 2002 CLIR test collection, which contains Arabic newswire articles from ¿Agence Française de Presse¿ (AFP). In addition, the work reports the conditions under which the produced relevance judgments and official TREC relevance judgments rank different systems in ways that highly correlate.