Indexer

An indexer processes pages collected by the crawler. First it decides which pages to index (it might discard duplicate documents). Most search engines then build some variant of an inverted index data structure for words (text index) and links (structured index).