.A Token is an occurence of a term from the text of a field. It consists of a term s text, the start and end offset of the term in the text of the field, and a type string. The start and end offsets permit applications to re-associate a token with it

A Token is an occurence of a term from the text of a field. It consists of a term s text, the start and end offset of the term in the text of the field, and a type string for Linux.