File Content Extraction 25.3
Filter SDK Java Programming Guide

# Character Encoding

To ensure that all filtered text is output in the same character encoding, File Content Extraction performs character encoding conversion. In most cases, File Content Extraction can determine the character encoding used in a source file, and automatically converts the filtered text to the encoding you choose. OpenText recommends that you specify your preferred target encoding. In the rare cases where File Content Extraction cannot detect the character encoding used in a source file, you can also specify the source encoding.

## Character Set Detection

Many formats specify their encoding as part of the format. However some formats do not, especially text-based formats. File Content Extraction can analyze the text and recognize several common character encodings, and it can use this information to convert to the target character encoding.

To avoid performance costs, File Content Extraction limits the amount of input data that it uses to perform this analysis. This limit might affect the accuracy of analysis for large files with very few non-ASCII characters.

When your license enables it, File Content Extraction can also perform advanced character set detection. Many encodings have significant overlap in the bytes that are used to encode characters, with the same bytes used to express different characters in different encodings. Advanced character set detection performs frequency analysis on the characters of the text to determine the most likely language and encoding.

Advanced character set detection works best on large portions of natural language. It might not be able to determine the correct source encoding where the text:

- is very short

- contains a random sequence of characters

- has some blocks of characters repeated many times

- contains characters from multiple languages

## Specify a Target Character Encoding

OpenText recommends that you specify a target character encoding when you initialize File Content Extraction, and recommends using UTF-8 or UTF-16 because these are widely supported and can encode a diverse range of characters.

DEPRECATED: Target character sets other than UTF-8 and UTF-16 are deprecated in File Content Extraction version 24.4 and later. OpenText recommends that you only use the widely supported UTF-8 and UTF-16 formats. Other character sets are still available, but might be removed in future. For a full list of these target encodings, see Coded Character Sets.

To specify a target character encoding

- In the Java API, specify the target encoding when you instantiate the Filter object. For example:

      Filter myFilter = new Filter("YOUR_LICENSE", Filter.CHARSET_UTF8, 0);

  After filtering, you can verify the output encoding by calling `getTargetCharSet()` on the Filter object. If the result is `KVCS_UNKNOWN`, File Content Extraction was unable to determine the source character encoding and therefore no conversion occurred. If you know the character encoding used in the source file you can specify it through the API - see Specify a Source Character Encoding.

## Performance Considerations

When a file format does not specify a character encoding, File Content Extraction attempts to detect the encoding automatically. Some character encodings, including UTF-8 and UTF-16, can be detected by core File Content Extraction functionality but others can be detected only if your license includes advanced character set detection. Advanced character set detection is enabled by default (if it is included in your license), but can increase the time required to filter some documents.

You can disable advanced character set detection on a file-by-file basis. Before doing this, be aware that File Content Extraction cannot output filtered text in your chosen encoding unless it detects the encoding of the source file, or you specify the source encoding yourself though the API.

To disable advanced character set detection

- In the Java API, call the method `setCharacterSetDetection()` on the Filter object, and pass in the value `false`.

## Specify a Source Character Encoding

In most cases, File Content Extraction can automatically detect the character encoding of an input file and specifying a source encoding is not necessary. You might need to specify the source character encoding if you have disabled advanced character set detection.

To specify the source character encoding

- In the Java API, call the `setSourceCharSet()` method of the Filter object. For example:

      myFilter.setSourceCharSet(Filter.CHARSET_8859_1);

## Disable Character Encoding Conversion

You can completely disable character encoding conversion, and retain the original character encoding of the document.

To disable character encoding conversion

- In the Java API, set the `FILTERFLAG_NODEFAULTCHARSETCONVERT` flag when you instantiate the Filter object. For example:

      Filter myFilter = new Filter("YOUR_LICENSE", Filter.CHARSET_UTF8, Filter.FILTERFLAG_NODEFAULTCHARSETCONVERT);

Copyright 1997-2025 Open Text
Version 25.3 | Last updated July 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: File Content Extraction 25.3

Topic Title: Character Encoding