File Content Extraction 25.4
Filter SDK Python Programming Guide

# Character Encoding

To ensure that all filtered text is output in the same character encoding, File Content Extraction performs character encoding conversion. In most cases, File Content Extraction can determine the character encoding used in a source file, and automatically converts the filtered text to the encoding you choose. OpenText recommends that you specify your preferred target encoding. In the rare cases where File Content Extraction cannot detect the character encoding used in a source file, you can also specify the source encoding.

## Character Set Detection

Many formats specify their encoding as part of the format. However some formats do not, especially text-based formats. File Content Extraction can analyze the text and recognize several common character encodings, and it can use this information to convert to the target character encoding.

To avoid performance costs, File Content Extraction limits the amount of input data that it uses to perform this analysis. This limit might affect the accuracy of analysis for large files with very few non-ASCII characters.

When your license enables it, File Content Extraction can also perform advanced character set detection. Many encodings have significant overlap in the bytes that are used to encode characters, with the same bytes used to express different characters in different encodings. Advanced character set detection performs frequency analysis on the characters of the text to determine the most likely language and encoding.

Advanced character set detection works best on large portions of natural language. It might not be able to determine the correct source encoding where the text:

- is very short

- contains a random sequence of characters

- has some blocks of characters repeated many times

- contains characters from multiple languages

## Specify a Target Character Encoding

## Performance Considerations

When a file format does not specify a character encoding, File Content Extraction attempts to detect the encoding automatically. Some character encodings, including UTF-8 and UTF-16, can be detected by core File Content Extraction functionality but others can be detected only if your license includes advanced character set detection. Advanced character set detection is enabled by default (if it is included in your license), but can increase the time required to filter some documents.

You can disable advanced character set detection on a file-by-file basis. Before doing this, be aware that File Content Extraction cannot output filtered text in your chosen encoding unless it detects the encoding of the source file, or you specify the source encoding yourself though the API.

To disable advanced character set detection

## Specify a Source Character Encoding

In most cases, File Content Extraction can automatically detect the character encoding of an input file and specifying a source encoding is not necessary. You might need to specify the source character encoding if you have disabled advanced character set detection.

To specify the source character encoding

Copyright 1997-2025 Open Text
Version 25.4 | Last updated October 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: File Content Extraction 25.4

Topic Title: Character Encoding