Expressiveness and Performance of Full-Text Search Languages

Abstract

We study the expressiveness and performance of full-text search languages. Our main motivation is to provide a formal basis for comparing such languages and to develop a model for full-text search that can be tightly integrated with structured search. We develop a formal model for full-text search based on the positions of tokens (words) in the input text, and develop a full-text calculus (FTC) and a full-text algebra (FTA) with equivalent expressive power. This suggests a notion of completeness for full-text search languages and can be used as a basis for a study of their expressiveness. We show that existing full-text languages are incomplete and develop {\tt COMP}, a complete full-text search language. We also identify practical subsets of {\tt COMP} that are more powerful than existing languages, develop efficient query evaluation algorithms for these subsets, and study experimentally their performance.