• tokenize
• tokenized
• tokenizer
• tokenizes
• tokenizers

tokenize (suomeksi)

Määritelmät:

• (lb, en, transitive, computing) To reduce to a set of tokens by parse, parsing.
• (lb, en, transitive) To treat as a token minority.