KeyView 24.1

# Example Development of fffFillBuffer()

The following is an example of how the `fpFillBuffer()` function in `foliosr` could be developed. The example demonstrates how the code changes as limitations of the implementation are identified. With each implementation, code revisions are shown in bold.

## Implementation 1—fpFillBuffer() Function

    /*****************************************************************
    *Function: fffFillBuffer()
    *Summary: Read fff input from stream and parse into kvtoken.h codes
    *****************************************************************/
    int pascal _export fffFillBuffer(
        void    *pCFContext,
        BYTE    *pcBuf,
        UINT    *pnBufOut,
        int    *pnPercentDone,
        UINT    cbBufOutMax )
    {
        BOOL bRetVal;
        TPfffGlobals *pContext = (TPfffGlobals *)pCFContext;
        pContext->pcBufOut = pcBuf;
        fffReadSourceFile(pContext);
        bRetVal = fffProcessBuffer(pContext, pcBuf);
        *pnPercentDone = (int)(pContext->unTotalBytesProcessed *
        (UINT)100 / pContext->unFileSize);
        *pnBufOut = (UINT)(pContext->pcBufOut - pcBuf);
        return (bRetVal ? KVERR_Success : KVERR_General);
    }

The parameters in `fffFillBuffer()` are as follows:

Parameter | In/Out | Description
--- | --- | ---
`pCFContext` | In | A pointer to the context structure of the custom reader.
`pcBuf` | In/Out | A pointer to the token output buffer.
`pnBufOut` | Out | A pointer to the number of bytes written to the output buffer.
`pnPercentDone` | Out | A pointer to the percentage complete.
`cbBufOutMax` | In | The maximum number of bytes that the token output buffer can hold.

## Structure of Implementation 1

1. The local variable `pContext` is set to the address of the `pCFContext` void pointer, cast to a pointer to the global context structure for the reader. This provides access to all members of this structure.
2. After setting the `pContext` variable, a call is made to read the source file.
3. Next, a call is made to `fffProcessBuffer()`. The second parameter in the call is a pointer to the token output buffer. If this call fails, usually because of memory allocation errors, it returns `FALSE`.
4. The percentage complete is calculated.
5. The number of `BYTES` written to the token output buffer is calculated. This is based on the value of `pContext->pcBufOut`, which is increased each time a token is written to the buffer.
6. The function returns to the structured access layer.
7. Subsequent calls to `fffFillBuffer()` are made by the structured access layer until the percentage complete is 100.

## Problems with Implementation 1

- There is a limit to the size of the token output buffer, typically 4 KB. If `fffProcessBuffer()` generates a token stream larger than this, there is a memory overflow. If `fffProcessBuffer()` generates a small token stream and the entire file has not been read, the output token buffer is underutilized.
- It might not be possible to process the entire input buffer from the source file because of boundary conditions. An example of a "boundary condition" is when the input buffer terminates part way through a control sequence in the original document. Another file read operation is required before the complete control sequence can be parsed.
- This function might be interrupted by other calls from the structured access layer to process headers, footers, footnotes, and endnotes, or to retrieve the document summary information. This can cause values of variables in the global context to change, and the source file to be repositioned.

## Implementation 2—Processing a Large Token Stream

Implementation 2 addresses the problem of processing a token stream that is larger than the output buffer size limit.

    /*****************************************************************
    * Function:   fffFillBuffer()
    * Summary:    Read fff input from stream and parse into kvtoken.h codes
    *****************************************************************/
    int pascal _export fffFillBuffer(
        void    *pCFContext,
        BYTE    *pcBuf,
        UINT    *pnBufOut,
        int    *pnPercentDone,
        UINT    cbBufOutMax )
    {
        BOOL bRetVal  = TRUE;
        TPfffGlobals *pContext = (TPfffGlobals *)pCFContext;
        pContext->pcBufOut      = pcBuf;
        pContext->cbBufOutMax   = 9 * cbBufOutMax / 10;  /* Process the portion of the fff file that is in the input buffer but do   * not return from the fffFillBuffer() function unless the output buffer is   * at least 90% full. If any of the memory allocations fail during the   * execution of fffProcessBuffer(), bRetVal will be set to FALSE, resulting   * in this conversion failing "gracefully".                                */
        do
        {
            if( pContext->bBufOutFull )
            {
                pContext->bBufOutFull = FALSE;
            }
            else
            {
                fffReadSourceFile(pContext);
            }
            bRetVal = fffProcessBuffer(pContext, pcBuf);
            *pnPercentDone = (int)(pContext->unTotalBytesProcessed *
            (UINT)100 / pContext->unFileSize);
        }while( bRetVal && !pContext->bBufOutFull && *pnPercentDone < 100 );
        *pnBufOut = (UINT)(pContext->pcBufOut - pcBuf);
        return (bRetVal ? KVERR_Success : KVERR_General);
    }

## Structure of Implementation 2

1. `cbBufOutMax` is used to set `pContext->cbBufOutMax`. This is used in `fffProcessBuffer()` to monitor how full the token output buffer becomes as the source file is processed.
2. When the source file input buffer has been processed, `fffProcessBuffer()` returns, and the percentage complete is calculated.
3. If the token output buffer is not filled to a value greater than `pContext->cbBufOutMax`, `pContext->bBufOutFull` remains set to `FALSE`, and if the percentage complete is less than 100, the `do-while` loop is re-entered without returning from this function to the structured access layer. There is another call to `fffReadSourceFile()`, followed by `fffProcessBuffer()`.
4. When the token output buffer is filled to a value greater than `pContext->cbBufOutMax`, `pContext->bBufOutFull` is set to `TRUE`. In this case, the `do-while` loop ends, the number of bytes written to the token output buffer is calculated, and control returns to the structured access layer.
5. The structured access layer continues to make calls to `fffFillBuffer()` until the entire source file is processed.
6. Each time the structured access layer calls `fffFillBuffer()`, another empty token output buffer is provided for the custom reader to use.
7. If the previous call to `fffFillBuffer()` exited because the previous token output buffer exceeded allowable capacity, `pContext->bBufOutFull` is reset to `FALSE` and no call is made to read the next buffer from the input source file.

## Problems with Implementation 2

- It might not be possible to process the entire input buffer from the source file because of boundary conditions.
- This function might be interrupted by other calls from the structured access layer to process headers, footers, footnotes, or endnotes, or to retrieve the document summary information. This can cause values of variables in the global context to change, and the source file to be repositioned.

## Boundary Conditions

A boundary condition can result from many situations arising from input file processing. For example, the input buffer might end with an incomplete command. In Folio flat files, this could be an incomplete element. In other word processing documents, a boundary condition might result from an incomplete control sequence, a split double-byte character, or a partial UTF-7 or UTF-8 sequence. These can be handled jointly by `fffProcessBuffer()`, which must detect the boundary condition, and `fffReadSourceFile()`.

The following example shows partial code used in `fffReadSourceFile()`:

    /****************************************************************
     *
     * Function:    fffReadSourceFile()
     *
     ***************************************************************/
    int pascal fffReadSourceFile(TPfffGlobals *pContext)
    {
        int nBytes;
        /* Transfer remaining data to beginning of buffer prior to next read */
        if( pContext->nResidualBytes )
        {
            memcpy(pContext->cInputBuf, pContext->pcBufIn, pContext->nResidualBytes);
        }
        /* Read from file, without over-writing any text from the previous buffer */
        nBytes = (*pContext->pIO->kwReadFunc)(pContext->pIO,
                pContext->cInputBuf + pContext->nResidualBytes,
                BUFFERSIZE - pContext->nResidualBytes);
        /* Update input buffer control parameters */
        pContext->unTotalBytesRead += (UINT)nBytes;
        pContext->pcBufIn = pContext->cInputBuf;
        pContext->pcBufInMax = pContext->pcBufIn + pContext->nResidualBytes + nBytes;
        pContext->nResidualBytes = 0;
        return nBytes;
    }

If `fffProcessBuffer()` is unable to process the entire input source file buffer, it sets the value for `pContext->nResidualBytes`. When the next call to `fffReadSourceFile()` is made, any residual bytes are copied to the beginning of the input source file buffer, and the number of bytes to be read is reduced to make sure that this buffer does not overflow.

A good way to test the code for boundary conditions is to vary the size of `BUFFERSIZE` and make sure that the results remain consistent.

NOTE: With `ReadSourceFile()`, the source file can be read by calls to retrieve header or footer information. If this occurs, the value for `pContext->unTotalBytesRead` is incorrect.

## Implementation 3—Interrupting Structured Access Layer Calls

Implementation 3 addresses the problem of boundary conditions and interrupting calls from the structured access layer.

    /****************************************************************************
    * Function:   fffFillBuffer()
    * Summary:    Read fff input from stream and parse into kvtoken.h codes
    ****************************************************************************/
    int pascal _export fffFillBuffer(
        void    *pCFContext,
        BYTE    *pcBuf,
        UINT    *pnBufOut,
        int    *pnPercentDone,
        UINT    cbBufOutMax )
    {
        double dTotalBytesProcessed, dFileSize;
        BOOL bRetVal = TRUE;
        TPfffGlobals *pContext = (TPfffGlobals *)pCFContext;
        pContext->pcBufOut = pcBuf;
        pContext->cbBufOutMax = 9 * cbBufOutMax / 10;
    /* Process the portion of the fff file that is in the input buffer but do
    * not return from the fffFillBuffer() function unless the output buffer is
    * at least 90% full. If any of the memory allocations fail during the
    * execution of fffProcessBuffer(), bRetVal will be set to FALSE, resulting
    * in this conversion failing "gracefully". */
        do
        {
             if( pContext->bBufOutFull )
            {
                pContext->bBufOutFull = FALSE;
            }
             else
            {
                fffReadSourceFile(pContext);
            }
             bRetVal = fffProcessBuffer(pContext, pcBuf);
             if( pContext->bHeaderCompleted )
            {
                *pnPercentDone = 100;
                pContext->bHeaderCompleted = FALSE;
            }
             else if( pContext->bFooterCompleted )
            {
                *pnPercentDone = 100;
                pContext->bFooterCompleted = FALSE;
            }
             else
            {
                if( pContext->unTotalBytesProcessed >= pContext->unFileSize )
                {
                    *pnPercentDone = 100;
                }
                else if( pContext->unFileSize < FFF_MAX_ULONG )
                {
                    *pnPercentDone = (int)(pContext->unTotalBytesProcessed *
        (UINT)100 / pContext->unFileSize);
                }
                 else
                {
                    dTotalBytesProcessed = pContext->unTotalBytesProcessed;                dFileSize = pContext->unFileSize;
                    *pnPercentDone = (int)(dTotalBytesProcessed * 100 / dFileSize);
                }
        }
        }while( bRetVal && !pContext->bBufOutFull && *pnPercentDone < 100 );
        *pnBufOut = (UINT)(pContext->pcBufOut - pcBuf);
        return (bRetVal ? KVERR_Success : KVERR_General);
    }

## Structure of Implementation 3

- The most significant change in Implementation 3 is the addition of the code that checks whether the processing of the header or footer is complete. The variables for `pContext->bHeaderCompleted` and `pContext->bFooterCompleted` are set to `TRUE` in `fffProcessBuffer()` when a header or footer is processed and the end of that portion of the document is reached.
- The other piece of code added in Implementation 3 is unique to `foliosr`. Folio files can be 50 MB or larger. Therefore, an unsigned integer is too small to accurately calculate the percentage complete. If the file size exceeds `FFF_MAX_ULONG`, which is defined as `(UINT)(0xFFFFFFFF / 0x64)`, the doubles are used for that calculation.
- Prior to returning, the token output buffer is as full as possible and never overflows. The minimum number of calls is made.

Copyright 2016-2023 Open Text
Version 24.1 | Last updated December 2023

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: IDOL KeyView 24.1

Topic Title: Example Development of fffFillBuffer()