Do I need to free the strtok resulting string?

Question

Or rather, how does strtok produce the string to which it's return value points? Does it allocate memory dynamically? I am asking because I am not sure if I need to free the token in the following code:

The STANDARD_INPUT variables is for exit procedure in case I run out of memory for allocation and the string is the tested subject.

int ValidTotal(STANDARD_INPUT, char *str)
{
    char *cutout = NULL, *temp, delim = '#';
    int i = 0; //Checks the number of ladders in a string, 3 is the required number
    temp = (char*)calloc(strlen(str),sizeof(char));
    if(NULL == temp)
        Pexit(STANDARD_C); //Exit function, frees the memory given in STANDARD_INPUT(STANDARD_C is defined as the names given in STANDARD_INPUT)
    strcpy(temp,str);//Do not want to touch the actual string, so copying it
    cutout = strtok(temp,&delim);//Here is the lynchpin - 
    while(NULL != cutout)
    {
        if(cutout[strlen(cutout) - 1] == '_')
            cutout[strlen(cutout) - 1] = '\0'; \\cutout the _ at the end of a token
        if(Valid(cutout,i++) == INVALID) //Checks validity for substring, INVALID is -1
            return INVALID;
        cutout = strtok(NULL,&delim);
        strcpy(cutout,cutout + 1); //cutout the _ at the beginning of a token
    }
    free(temp);
return VALID; // VALID is 1
}

Answer 1

As others mentioned, strtok uses its first parameter, your input string, as the memory buffer. It doesn't allocate anything. It's stateful and non-thread safe; if strtok's first argument is null, it reuses the previously-provided buffer. During a call, strtok destroys the string, adding nulls into it and returning pointers to the tokens.

Here's an example:

#include <stdio.h>
#include <string.h>

int main() {
    char s[] = "foo;bar;baz";
    char *foo = strtok(s, ";");
    char *bar = strtok(NULL, ";");
    char *baz = strtok(NULL, ";");

    printf("%s %s %s\n", foo, bar, baz); // => foo bar baz
    printf("original: %s\n", s); // => original: foo
    printf("%ssame memory loc\n", s == foo ? "" : "not "); // => same memory loc

    return 0;
}

s started out as foo;bar;baz\0. Three calls to strtok turned it into foo\0bar\0baz\0. s is basically the same as the first chunk, foo.

Valgrind:

==89== HEAP SUMMARY:
==89==     in use at exit: 0 bytes in 0 blocks
==89==   total heap usage: 1 allocs, 1 frees, 1,024 bytes allocated
==89==
==89== All heap blocks were freed -- no leaks are possible

While the code below doesn't fix all of the problems with strtok, it might help get you moving in a pinch, preserving the original string with strdup:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main() {
    const char s[] = "foo;bar_baz";
    const char delims[] = ";_";
    char *cpy = strdup(s);
    char *foo = strtok(cpy, delims);
    char *bar = strtok(NULL, delims);
    char *baz = strtok(NULL, delims);

    printf("%s %s %s\n", foo, bar, baz); // => foo bar baz
    printf("original: %s\n", s); // => original: foo;bar_baz
    printf("%ssame memory loc\n", s == foo ? "" : "not "); // => not same memory loc

    free(cpy);
    return 0;
}

Or a more full-fledged example (still not thread safe):

#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

void must(
    bool predicate,
    const char *msg,
    const char *file,
    const unsigned int line
) {
    if (!predicate) {
        fprintf(stderr, "%s:%d: %s\n", file, line, msg);
        exit(1);
    }
}

size_t split(
    char ***tokens,
    const size_t len,
    const char *s,
    const char *delims
) {
    char temp[len+1];
    temp[0] = '\0';
    strcpy(temp, s);
    *tokens = malloc(sizeof(**tokens) * 1);
    must(*tokens, "malloc failed", __FILE__, __LINE__);
    size_t chunks = 0;

    for (;;) {
        char *p = strtok(chunks == 0 ? temp : NULL, delims);

        if (!p) {
            break;
        }

        size_t sz = sizeof(**tokens) * (chunks + 1);
        *tokens = realloc(*tokens, sz);
        must(*tokens, "realloc failed", __FILE__, __LINE__);
        (*tokens)[chunks++] = strdup(p);
    }

    return chunks;
}

int main() {
    const char s[] = "foo;;bar_baz";
    char **tokens;
    size_t len = split(&tokens, strlen(s), s, ";_");

    for (size_t i = 0; i < len; i++) {
        printf("%s ", tokens[i]);
        free(tokens[i]);
    }

    puts("");
    free(tokens);
    return 0;
}

Answer 2

strtok manipulates the string you pass in and returns a pointer to it, so no memory is allocated.

Please consider using strsep or at least strtok_r to save you some headaches later.

Answer 3

According to the docs:

Return Value

A pointer to the last token found in string.

Since the return pointer just points to one of the bytes in your input string where the token starts, whether you need to free depends on whether you allocated the input string or not.

Answer 4

The first parameter to the strtok(...) function is YOUR string:

str
C string to truncate. Notice that this string is modified by being broken into smaller strings (tokens). Alternativelly, a null pointer may be specified, in which case the function continues scanning where a previous successful call to the function ended.

It puts '\0' characters into YOUR string and returns them as terminated strings. Yes, it mangles your original string. If you need it later, make a copy.

Further, it should not be a constant string (e.g. char* myStr = "constant string";). See here.

It could be allocated locally or by malloc/calloc.

If you allocated it locally on the stack (e.g. char myStr[100];), you don't have to free it.

If you allocated it by malloc (e.g. char* myStr = malloc(100*sizeof(char));), you need to free it.

Some example code:

#include <string.h>
#include <stdio.h>
int main()
{
   const char str[80] = "This is an example string.";
   const char s[2] = " ";
   char *token;

   /* get the first token */
   token = strtok(str, s);

   /* walk through other tokens */
   while( token != NULL ) 
   {
      printf( " %s\n", token );

      token = strtok(NULL, s);
   }

   return(0);
}

NOTE: This example shows how you iterate through the string...since your original string was mangled, strtok(...) remembers where you were last time and keeps working through the string.