Adapting Boyer-Moore Implementation

Question

I'm trying to adapt the Boyer-Moore c(++) Wikipedia implementation to get all of the matches of a pattern in a string. As it is, the Wikipedia implementation returns the first match. The main code looks like:

char* boyer_moore (uint8_t *string, uint32_t stringlen, uint8_t *pat, uint32_t patlen) {
    int i;
    int delta1[ALPHABET_LEN];
    int *delta2 = malloc(patlen * sizeof(int));
    make_delta1(delta1, pat, patlen);
    make_delta2(delta2, pat, patlen);

    i = patlen-1;
    while (i < stringlen) {
        int j = patlen-1;
        while (j >= 0 && (string[i] == pat[j])) {
            --i;
            --j;
        }
        if (j < 0) {
            free(delta2);
            return (string + i+1);
        }

        i += max(delta1[string[i]], delta2[j]);
    }
    free(delta2);
    return NULL;
}

I have tried to modify the block after if (j < 0) to add the index to an array/vector and letting the outer loop continue, but it doesn't appear to be working. In testing the modified code I still only get a single match. Perhaps this implementation wasn't designed to return all matches, and it needs more than a few quick changes to do so? I don't understand the algorithm itself very well, so I'm not sure how to make this work. If anyone can point me in the right direction I would be grateful.

Note: The functions make_delta1 and make_delta2 are defined earlier in the source (check Wikipedia page), and the max() function call is actually a macro also defined earlier in the source.

Answer 1

Boyer-Moore's algorithm exploits the fact that when searching for, say, "HELLO WORLD" within a longer string, the letter you find in a given position restricts what can be found around that position if a match is to be found at all, sort of a Naval Battle game: if you find open sea at four cells from the border, you needn't test the four remaining cells in case there's a 5-cell carrier hiding there; there can't be.

If you found for example a 'D' in eleventh position, it might be the last letter of HELLO WORLD; but if you found a 'Q', 'Q' not being anywhere inside HELLO WORLD, this means that the searched-for string can't be anywhere in the first eleven characters, and you can avoid searching there altogether. A 'L' on the other hand might mean that HELLO WORLD is there, starting at position 11-3 (third letter of HELLO WORLD is a L), 11-4, or 11-10.

When searching, you keep track of these possibilities using the two delta arrays.

So when you find a pattern, you ought to do,

if (j < 0)
{
    // Found a pattern from position i+1 to i+1+patlen
    // Add vector or whatever is needed; check we don't overflow it.
    if (index_size+1 >= index_counter)
    {
        index[index_counter] = 0;
        return index_size;
    }
    index[index_counter++] = i+1;

    // Reinitialize j to restart search
    j = patlen-1;

    // Reinitialize i to start at i+1+patlen
    i += patlen +1; // (not completely sure of that +1)

    // Do not free delta2
    // free(delta2);

    // Continue loop without altering i again
    continue;
}
i += max(delta1[string[i]], delta2[j]);
}
free(delta2);
index[index_counter] = 0;
return index_counter;

This should return a zero-terminated list of indexes, provided you pass something like a size_t *indexes to the function.

The function will then return 0 (not found), index_size (too many matches) or the number of matches between 1 and index_size-1.

This allows for example to add additional matches without having to repeat the whole search for the already found (index_size-1) substrings; you increase num_indexes by new_num, realloc the indexes array, then pass to the function the new array at offset old_index_size-1, new_num as the new size, and the haystack string starting from the offset of match at index old_index_size-1 plus one (not, as I wrote in a previous revision, plus the length of the needle string; see comment).

This approach will report also overlapping matches, for example searching ana in banana will find b*ana*na and ban*ana*.

UPDATE

I tested the above and it appears to work. I modified the Wikipedia code by adding these two includes to keep gcc from grumbling

#include <stdio.h>
#include <string.h>

then I modified the if (j < 0) to simply output what it had found

    if (j < 0) {
            printf("Found %s at offset %d: %s\n", pat, i+1, string+i+1);
            //free(delta2);
            // return (string + i+1);
            i += patlen + 1;
            j = patlen - 1;
            continue;
    }

and finally I tested with this

int main(void)
{
    char *s = "This is a string in which I am going to look for a string I will string along";
    char *p = "string";
    boyer_moore(s, strlen(s), p, strlen(p));
    return 0;
}

and got, as expected:

Found string at offset 10: string in which I am going to look for a string I will string along
Found string at offset 51: string I will string along
Found string at offset 65: string along

If the string contains two overlapping sequences, BOTH are found:

char *s = "This is an andean andeandean andean trouble";
char *p = "andean";

Found andean at offset 11: andean andeandean andean trouble
Found andean at offset 18: andeandean andean trouble
Found andean at offset 22: andean andean trouble
Found andean at offset 29: andean trouble

To avoid overlapping matches, the quickest way is to not store the overlaps. It could be done in the function but it would mean to reinitialize the first delta vector and update the string pointer; we also would need to store a second i index as i2 to keep saved indexes from going nonmonotonic. It isn't worth it. Better:

    if (j < 0) {
        // We have found a patlen match at i+1
        // Is it an overlap?
        if (index && (indexes[index] + patlen < i+1))
        {
            // Yes, it is. So we don't store it.


            // We could store the last of several overlaps
            // It's not exactly trivial, though:
            // searching 'anana' in 'Bananananana'
            // finds FOUR matches, and the fourth is NOT overlapped
            // with the first. So in case of overlap, if we want to keep
            // the LAST of the bunch, we must save info somewhere else,
            // say last_conflicting_overlap, and check twice.
            // Then again, the third match (which is the last to overlap
            // with the first) would overlap with the fourth.

            // So the "return as many non overlapping matches as possible"
            // is actually accomplished by doing NOTHING in this branch of the IF.
        }
        else
        {
            // Not an overlap, so store it.
            indexes[++index] = i+1;
            if (index == max_indexes) // Too many matches already found?
                break; // Stop searching and return found so far
        }
        // Adapt i and j to keep searching
        i += patlen + 1;
        j = patlen - 1;
        continue;
    }