strtok() returns NULL on subsequent calls even though there are more tokens to be read

Question

I'm having an issue in my program where calling strtok(NULL, "\r\n"); returns a NULL after I made a function call even though there are still tokens in the stream. I've looked over this for a while and can't figure out what it is about this function call that changes the behaviour of subsequent strtok() calls.

I'd be really grateful to anyone who can help. Cheers.

Main function:

int main() 
{ 
    char raw[] = "0 4 96 30\r\n3 4 64 60\r\n3 5 64 20\r\n3 2 32 40\r\n5 1 100 20\r\n20 3 4 30\r\n"; 

    char* line = strtok(raw, "\r\n");            //line == "0 4 96 30" OK
    line = strtok(NULL, "\r\n");                 //line == "3 4 64 60" OK
    line = strtok(NULL, "\r\n");                 //line == "3 5 64 20" OK
    struct Process current = parseProcess(line); //Now strtok calls after this will return NULL...
    line = strtok(NULL, "\r\n");                 //line == NULL (supposed to be "3 2 32 40")
    line = strtok(NULL, "\r\n");                 //line == NULL

    return 0; 
}   

The struct and function used:

#include <stdio.h> 
#include <stdlib.h>
#include <string.h> 

struct Process {
    long timeArrived;
    long processId;
    long memorySizeReq;
    long jobTime;
    long remainingTime;
};

//Rips values from the input and puts it into a struct
struct Process parseProcess(char* input){
    struct Process output;

    //Makes a back up as to not mutate input
    char* temp = malloc(strlen(input) * sizeof(char));
    strcpy(temp, input);
    char* token = strtok(temp, " ");

    //Fills out the fields of the parsed output
    for(int i=0; i<4; i++){
        switch(i){
            case 0:
                output.timeArrived = atoi(token);
                token = strtok(NULL, " ");
                break;
            case 1:
                output.processId = atoi(token);
                token = strtok(NULL, " ");
                break;
            case 2:
                output.memorySizeReq = atoi(token);
                token = strtok(NULL, " ");
                break;
            case 3:
                output.jobTime = atoi(token);
                output.remainingTime = atoi(token);
                token = strtok(NULL, " ");
                break;
        }
    }
    return output;
}

Answer 1

strtok is not reentrant and is used inside parseProcess so the next calls in main use a different context. Furthermore strtok(raw, "\r\n"); considers both '\r' and '\n' as separators as well as any sequence of these characters, the consequence of this is strtok() won't return empty tokens for empty lines in the source string.

Regarding POSIX strtok_r, following the comments, it does not work in windows, you do, however, have strtok_s. There are other ways to do this, but given that you are using strtok() here is an example of this could be implemented:

Live demo

struct Process parseProcess(char* input){
    struct Process output; 
    char* temp = malloc(strlen(input) + 1); //notes 1, 2, 3
    strcpy(temp, input);   
    char* strmax; //for **stmax parameter
    char* token = strtok_r(temp, " ", &strmax);
    for(int i = 0; i < 4; i++){
        switch(i){
            case 0:
                output.timeArrived = atoi(token);
                token = strtok_r(NULL, " ", &strmax);
                break;
            case 1:
                output.processId = atoi(token);
                token = strtok_r(NULL, " ", &strmax);
                break;
            case 2:
                output.memorySizeReq = atoi(token);
                token = strtok_r(NULL, " ", &strmax);
                break;
            case 3:
                output.jobTime = atoi(token);
                output.remainingTime = atoi(token);
                token = strtok_r(NULL, " ", &strmax);
                break;
        }
    }
    free(temp);   //free the allocated memory
    return output;    
}

Notes:

1. Reserve space for null terminator when you allocate with strlen.

2. Memory allocation is expensive, if you can, avoid it, in this case you could use char temp[strlen(input) + 1];.

   MSVC quibles with this because it's a variable lenght arrays and says it's an error, it's not, VLAs are valid in C, you can use gcc or clang compilers, if you want to use them.

3. sizeof(char) is not needed, char is 1 byte across platforms.

//...
#ifdef _MSC_VER  //improved portability
    #define strtok_r strtok_s
#endif 

int main() 
{   
    char raw[] = "0 4 96 30\r\n3 4 64 60\r\n3 5 64 20\r\n3 2 32 40\r\n5 1 100 20\r\n20 3 4 30\r\n"; 
    char* strmax;
    char* line = strtok_r(raw, "\r\n", &strmax);            //line == "0 4 96 30" OK                                 
    line = strtok_r(NULL, "\r\n", &strmax);                 //line == "3 4 64 60" OK
    line = strtok_r(NULL, "\r\n", &strmax);                 //line == "3 5 64 20" OK                                   
    struct Process current = parseProcess(line); 
    line = strtok_r(NULL, "\r\n", &strmax);                 //line == "3 2 32 40" OK                                  
    line = strtok_r(NULL, "\r\n", &strmax);                 ///line == "5 1 100 20" OK;

    return 0; 
}