rdf_tokenize_literal/2

Documentation
- Reference manual
- Packages
  - SWI-Prolog Semantic Web Library 3.0
    - Plugin modules for rdf_db
      - library(semweb/rdf_litindex): Indexing words in literals
        
        rdf_find_literals/2
        
        rdf_token_expansions/2
        
        rdf_tokenize_literal/2
        
        Literal maps: Creating additional indices on literals

Availability::- use_module(library(semweb/rdf_litindex)).

rdf_tokenize_literal(+Literal, -Tokens)

Tokenize a literal, returning a list of atoms and integers in the range -1073741824 ... 1073741823. As tokenization is in general domain and task-dependent this predicate first calls the hook rdf_litindex:tokenization(Literal, -Tokens). On failure it calls tokenize_atom/2 from the NLP package and deletes the following: atoms of length 1, floats, integers that are out of range and the english words and, an, or, of, on, in, this and the. Deletion first calls the hook

rdf_litindex:exclude_from_index(token, 
X)

. This hook is called as follows:

no_index_token(X) :-
        exclude_from_index(token, X), !.
no_index_token(X) :-
        ...