Open topic with navigation

# Character Tokenization

You can tokenize characters into N-grams of a specified size. Set the `NGram` configuration parameter in your language configuration section to the number of characters to use in each N-gram group.

Note: You must not use `NGram` with the `SentenceBreaking` configuration parameter.

For example, if you set `NGram` to `2`, IDOL Server tokenizes the word Hello as:

    he el ll lo

To tokenize only multibyte strings, set `NGramMultiByteOnly` to `True`.

    [Japanese]
    NGram=2
    NGramMultiByteOnly=True

For this configuration, if you have a document that contains both English and Asian (multibyte) text, IDOL Server tokenizes the Asian text according to the `NGram` parameter. It does not tokenize the English text.

To tokenize only multiple-byte strings in Chinese, Japanese, and Korean characters (and ignore multiple-byte strings in other languages), set `NGramOrientalOnly` to `True`.

    [Japanese]
    NGram=2
    NGramOrientalOnly=True

For this configuration, if you have a document that contains multibyte text in both Japanese and Greek, IDOL Server tokenizes the Japanese text according to the `NGram` parameter. It does not tokenize the Greek multibyte text.

© 2016 Hewlett Packard Enterprise Development LP | Send documentation feedback to HPE Close We welcome your comments! To open the configured email client on this computer, open an email window. Otherwise, copy the information below to a web mail client, and send this email to AutonomyTPFeedback@hpe.com. Product: Topic Title: Help Topic: Feedback:

_HP_HTML5_bannerTitle.htm