Decoding TIFF LZW codes not yet in the dictionary

Question

I made a decoder of LZW-compressed TIFF images, and all the parts work, it can decode large images at various bit depths with or without horizontal prediction, except in one case. While it decodes files written by most programs (like Photoshop and Krita with various encoding options) fine, there's something very strange about the files created by ImageMagick's convert, it produces LZW codes that aren't yet in the dictionary, and I don't know how to handle it.

Most of the time the 9 to 12-bit code in the LZW stream that isn't yet in the dictionary is the next one that my decoding algorithm will try to put in the dictionary (which I'm not sure should be a problem although my algorithm fails on an image that contains such cases), but at times it can even be hundreds of codes into the future. In one case the first code after the clear code (256) is 364, which seems quite impossible given that the clear code clears my dictionary of all codes 258 and above, in another case the code is 501 when my dictionary only goes up to 317!

I have no idea how to deal with it, but it seems that I'm the only one with this problem, the decoders in other programs load such images fine. So how do they do it?

Here's the core of my decoding algorithm, obviously due to how much code is involved I can't provide complete compilable code in a compact manner, but since this is a matter of algorithmic logic this should be enough. It follows closely the algorithm described in the official TIFF specification (page 61), in fact most of the spec's pseudo code is in the comments.

void tiff_lzw_decode(uint8_t *coded, buffer_t *dec)
{
    buffer_t word={0}, outstring={0};
    size_t coded_pos;   // position in bits
    int i, new_index, code, maxcode, bpc;

    buffer_t *dict={0};
    size_t dict_as=0;

    bpc = 9;            // starts with 9 bits per code, increases later
    tiff_lzw_calc_maxcode(bpc, &maxcode);
    new_index = 258;        // index at which new dict entries begin
    coded_pos = 0;          // bit position

    lzw_dict_init(&dict, &dict_as);

    while ((code = get_bits_in_stream(coded, coded_pos, bpc)) != 257)   // while ((Code = GetNextCode()) != EoiCode) 
    {
        coded_pos += bpc;

        if (code >= new_index)
            printf("Out of range code %d (new_index %d)\n", code, new_index);

        if (code == 256)                        // if (Code == ClearCode)
        {
            lzw_dict_init(&dict, &dict_as);             // InitializeTable();
            bpc = 9;
            tiff_lzw_calc_maxcode(bpc, &maxcode);
            new_index = 258;

            code = get_bits_in_stream(coded, coded_pos, bpc);   // Code = GetNextCode();
            coded_pos += bpc;

            if (code == 257)                    // if (Code == EoiCode)
                break;

            append_buf(dec, &dict[code]);               // WriteString(StringFromCode(Code));

            clear_buf(&word);
            append_buf(&word, &dict[code]);             // OldCode = Code;
        }
        else if (code < 4096)
        {
            if (dict[code].len)                 // if (IsInTable(Code))
            {
                append_buf(dec, &dict[code]);           // WriteString(StringFromCode(Code));

                lzw_add_to_dict(&dict, &dict_as, new_index, 0, word.buf, word.len, &bpc);
                lzw_add_to_dict(&dict, &dict_as, new_index, 1, dict[code].buf, 1, &bpc);    // AddStringToTable
                new_index++;
                tiff_lzw_calc_bpc(new_index, &bpc, &maxcode);

                clear_buf(&word);
                append_buf(&word, &dict[code]);         // OldCode = Code;
            }
            else
            {
                clear_buf(&outstring);
                append_buf(&outstring, &word);
                bufwrite(&outstring, word.buf, 1);      // OutString = StringFromCode(OldCode) + FirstChar(StringFromCode(OldCode));

                append_buf(dec, &outstring);            // WriteString(OutString);

                lzw_add_to_dict(&dict, &dict_as, new_index, 0, outstring.buf, outstring.len, &bpc); // AddStringToTable
                new_index++;
                tiff_lzw_calc_bpc(new_index, &bpc, &maxcode);

                clear_buf(&word);
                append_buf(&word, &dict[code]);         // OldCode = Code;
            }
        }

    }

    free_buf(&word);
    free_buf(&outstring);
    for (i=0; i < dict_as; i++)
        free_buf(&dict[i]);
    free(dict);
}

As for the results that my code produces in such situations it's quite clear from how it looks that it's only those few codes that are badly decoded, everything before and after is properly decoded, but obviously in most cases the subsequent image after one of these mystery future codes is ruined by virtue of shifting the rest of the decoded bytes by a few places. That means that my reading of the 9 to 12-bit code stream is correct, so this really means that I see a 364 code right after a 256 dictionary-clearing code.

Edit: Here's an example file that contains such weird codes. I've also found a small TIFF LZW loading library that suffers from the same problem, it crashes where my loader finds the first weird code in this image (code 3073 when the dictionary only goes up to 2051). The good thing is that since it's a small library you can test it with the following code:

#include "loadtiff.h"
#include "loadtiff.c"
void loadtiff_test(char *path)
{
    int width, height, format;
    floadtiff(fopen(path, "rb"), &width, &height, &format);
}

And if anyone insists on diving into my code (which should be unnecessary, and it's a big library) here's where to start.

Answer 1

The bogus codes come from trying to decode more than we're supposed to. The problem is that a LZW strip may sometimes not end with an End-of-Information 257 code, so the decoding loop has to stop when a certain number of decoded bytes have been output. That number of bytes per strip is determined by the TIFF tags ROWSPERSTRIP * IMAGEWIDTH * BITSPERSAMPLE / 8, and if PLANARCONFIG is 1 (which means interleaved channels as opposed to planar), by multiplying it all by SAMPLESPERPIXEL. So on top of stopping the decoding loop when a code 257 is encountered the loop must also be stopped after that count of decoded bytes has been reached.