strtok does not appear to modify the input string

Question

I've always understood that strtok should be regarded with caution due to the fact that it modifies its input string by inserting NULLs at the last token locations. This is also validated by cppreference.

However, on trying to validate an example from cplusplus.com, I found that in VS2010 in 32 bit Windows 7, strtok is in fact NOT inserting NULLs in the original string. I was able to determine this by modifying the NULL in the argument from the example to str, and the program loops repeatedly, passing "This" as the token, which (as I interpreted this), is not the same behavior as passing the previous pointer, as cppreference claims.

In addition, I thought maybe the const-ness of string literals may have been at fault, so I copied the string

char str2[] ="- This, a sample string.";
char str[50];
strcpy(str,str2);

and ran it again, but the loop repeated. Debugger shows that the input string is not modified.

Can someone explain where I am going wrong here? edit: I think it's my interpretation of "The behavior is the same as if the previously stored pointer is passed as str."

Thank you.

EDIT: exact code:

/* strtok example */
#include <iostream>
#include <stdio.h>
#include <string.h>

int main ()
{
  char str2[] ="- This, a sample string.";
  char str[50];
  strcpy(str,str2);
  char * pch;
  printf ("Splitting string \"%s\" into tokens:\n",str);
  pch = strtok (str," ,.-");
  while (pch != NULL)
  {
    printf ("%s\n",pch);
    pch = strtok (str, " ,.-");
    printf("%s\n", str);
  }
  std::cin.ignore();
  return 0;
}

Output from code:

Splitting string "- This, a sample string." into tokens:
This
- This
This
- This
This
- This
This
- This
This
- This
This
- This
This
- This

EDIT: RESOLVED Should I delete this garbage or let it stay? haha I don't want to take points away from the poor people who had to deal with this

Answer 1

Firstly, it is not NULL that is inserted into the original string, it is a zero-character that is inserted. While I understand that this is probably what you wanted to say, it is still not a good idea to involve a well-known and completely unrelated macro NULL.

Secondly, if strtok fails inserting zero characters into the original string, it simply won't work as intended. For this reason I strongly believe that you somehow misinterpreted the results of your experiments. strtok does modify the input string even in VS2010 under 32 bit Windows 7.

---

The output from your code you posted clearly shows that the string was modified. The original value of str was "- This, a sample string.". The value of str printed from inside the loop is just "- This". The string got truncated specifically because strtok inserted a zero-character right after "- This" (more precisely, the , character got replaced with \0 character).

Answer 2

I was able to determine this by modifying the NULL in the argument from the example to str, and the program loops repeatedly

It does this because when you pass in str (or a non-NULL pointer), strtok() starts over, so it will just tokenize what's now a single token (returning it over and over). That's the documented behavior when a non-NULL pointer is passed in to strtok().

When you pass in NULL as the first argument to strtok() that tells it to pick up where it left off last time (it keeps track of that state in a static variable somewhere, which is one of the problems with strtok()).