C language reading columnated text file

Question

First of all let me ask for your forgiveness if this is too trivial, I am not a C developer, usually I program in Fortran.

I am in need to read some columnated text files. The problem I have is that some columns can have blank space (non filled value) or not fully filed field.

Let me use a short example of the problem. Lets say I have a generator program like:

#include <stdio.h>
#include <stdlib.h>

int main(){

   printf("xxxx%4d%4.2f\n",99,3.14);

}

When I execute this program I get:

$ ./t1
xxxx  993.14

If I get it into a text file and try to read using (e.g.) sscanf with the code:

#include <stdio.h>
#include <stdlib.h>

int main() {

   char *fmt = "%*4c%4d%4f";
   char *line = "xxxx  993.14";
   int  ival;
   float fval;

   sscanf(line,fmt,&ival,&fval);

   printf(">>>>%d|%f\n",ival,fval);
}

The result is:

$ ./t2
>>>>993|0.140000

What is the problem here? The sscanf seems to think that all space is meaningless and should be discarded. So the "%4c" does what it is meant to be, it counts 4 characters without discarding any blank space and discards everything due to "". Next the %4d start skipping all blank spaces and start count the 4 characters of the field upon finding the first valid character for the conversion. So the value, meant to be 99 becomes 993, and the 3.14 becomes 0.14.

In Fortran the reading code would be:

program t3

   implicit none

   integer :: ival
   real    :: fval
   character(len=30) :: fmt="(4x,i4,f4.0)"
   character(len=30) :: line="xxxx  993.14"

   read(line,fmt) ival, fval
   write(*,"('>>>>',i4,'|',f4.2)") ival,fval

end program t3

and the result would be:

$ ./t3
>>>>  99|3.14

That is, the format specification states the field width and nothing is discarding in conversion, except if instructed to by the "nX" specification.

Some final remarks to help the helpers:

1. The format to be read is an international standard and there is no way to change it.
2. The number of existing files is to big to think of intervention or format change.
3. It is not a CSV or similar format.

4. The code has to be in C for integration in a free software package.

   Sorry to be too long, trying to state the problem as completely as possible.

   The question is: Is there a way to tell sscanf to not skip the blank spaces? If not, is there a simple way to do it in C or it will be necessary write an specialized parser for each record type?

   Thank you in advance.

Answer 1

*scanf() is not designed to handle fixed column width with non-intervening white-space.

With sscanf(), to not skip spaces, code must use "%c", "%n", "%[]" as all other specifiers skip leading white-space and those skipped characters do not contribute to a width limit.

---

To scan the printed line, which in now in buffer, take advantage that the only use of '\n' is at the end of the line.

char str_int[5];
char str_float[5];
int n = 0;

sscanf(buffer, "%*4c%4[^\n]%4[^\n]%n", str_int, str_float, &n);
if (n != 12 || buffer[n] != '\n') Fail();
// Now convert str_int, str_float as needed.

Another way to use sscanf() would be to parse buffer as

int  ival;
float fval;
if (strlen(buffer) != 13) Fail();
if (sscanf(&buffer[8], "%f", &fval) != 1) Fail();
buffer[8] = '\0';
if (sscanf(&buffer[4], "%d", &ival) != 1) Fail();

---

Note: The 4s in the below do not specified the output width as 4 characters. 4 is the minimum width to print.

printf("xxxx%4d%4.2f\n",ival, fval);

Code could use the following to detect problems.

if (13 != printf("xxxx%4d%4.2f\n",ival, fval)) Fail();

Watch out for

printf("xxxx%4d%4.2f\n",123, 9.995000001f); // "xxxx 12310.00\n"

Answer 2

I am posting what I think is a final conclusion from the answers I have got so far and from other sources.

What is a very trivial task in Fortran is not a so trivial task in other languages. I guess — not sure — that the same task could be as easy as in Fortran in other languages. I think that Cobol, Pascal, PL/I and others from the time of punched card probably could be trivial.

I think that most languages nowadays are more comfortable with different data structure and inherited its I/O structure from C. I think that Java, Python, Perl(?) and others could serve as examples.

From what I saw in this thread there are two main problems to read / convert fixed column length text data with C.

The first problem is that, as Philip said in his answer: “The tool’s trying to be smart of helpful and it’s biting you in the ass.” Quite right! The point is that it seems that C text I/O thinks that “white space” is something like a NULL character and should be thrown away, completely disregarding any information of the start of field. The only exception to that seems to be the %nc that get exactly n chars, even blanks.

The second problem is that the conversion “tag” (how is that called?) %nf will keep converting while it finds a valid character, even if you say stop at the 4th character.

If we join those two problems with a field completely filled with white space, depending on the conversion tool used, it throws an error or keeps going madly looking for something meaningful.

At the end of the day, it seems that the only way is to extract the field length to another memory area, dynamically allocated or not (we can have an area for each column length), and try to parse this separate area, taking into account the possibility of a full white space area to cache the error.

Answer 3

When reading fixed-length fields with sscanf, it is best to parse the values as character strings (which you could do a number of ways), and then perform independent conversion of each of the fields. This allows you to handle conversion/error detection on a per-field basis. For example, you could use a format string of:

char *fmt = "%*4s%2[^0-9]%s";

which would read/discard the 4 leading characters, then read 2-chars as your integer, followed by the remainder of line (or up until the next whitespace) as a string containing your float value.

To handle the storage and parsing of line as fixed length fields, you could use temporary character arrays to hold each of the strings and then use sscanf to fill them much as you have attempted to do with the integer and float directly. e.g.:

char istr[8] = {0};
char fstr[16] = {0};
...
sscanf (line,fmt,istr,fstr);

(note: you could use minimum storage of istr[3] and fstr[7] in this given case, adjust the storage length as required, but providing space for the nul-terminating character)

You can then use strtol and strtof to provide conversion with error checking on each value. For example:

errno = 0;
if ((ival = (int)strtol (istr, NULL, 10)) == 0 && errno)
    fprintf (stderr, "error: integer conversion failed.\n");
    /* underflow/overflow checks omitted */

and

errno = 0;
if ((fval = strtof (fstr, NULL)) == 0 && errno)
    fprintf (stderr, "error: integer conversion failed.\n");
    /* nan and inf checks omitted */

Putting all the pieces together in you example, you could use something like:

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

int main() {

    char *fmt = "%*4s%2[^0-9]%s";
    char *line = "xxxx  993.14";

    char istr[8] = {0};
    char fstr[16] = {0};
    int ival;
    float fval;

    sscanf (line,fmt,istr,fstr);

    errno = 0;
    if ((ival = (int)strtol (istr, NULL, 10)) == 0 && errno)
        fprintf (stderr, "error: integer conversion failed.\n");
        /* underflow/overflow checks omitted */

    errno = 0;
    if ((fval = strtof (fstr, NULL)) == 0 && errno)
        fprintf (stderr, "error: integer conversion failed.\n");
        /* nan and inf checks omitted */

    printf(">>>>%d|%6.2f\n",ival,fval);

    return 0;
}

Example/Output

$ >>>>0|993.14

Answer 4

If each element is of fixed width you don't really need scanf(), try this

char copy[5];
const char *line = "xxxx  993.14";
int ival;
float fval;

copy[0] = line[4];
copy[1] = line[5];
copy[2] = line[6];
copy[3] = line[7];
copy[4] = '\0'; // nul terminate for `atoi' to work

ival = atoi(copy);
fval = atof(&line[8]);

fprintf(stdout, "%d -- %f\n", ival, fval);

If you want (probably should) you can use strtol() instead of atoi() and strtof() instead of atof() to check for malformed data.

Both these functions take a parameter to store the unconverted/invalid characters, you can check the passed pointer in order to verify that there was a problem with conversion.

Or if you really want scanf() do the same, capture the integer + whitespaces to a char array and then convert it to int later, like this

char integer[5];
const char *line = "xxxx  993.14";
int ival;
float fval;

if (sscanf(line, "%*4c%4[0-9 ]%f", integer, &fval) != 2)
    return -1;
ival = atoi(integer);

fprintf(stdout, "%d -- %f\n", ival, fval);

The format "%*4c%4[0-9 ]%f" will

1. Skip the first four characters including white spaces.
2. Scan the next four characters if they consist only of digits or white spaces.
3. Scan the rest of the input string searching for a matching float value.

Answer 5

First off, I dunno. There might be some way to wrangle sscanf to recognize the whitespace towards your integer count. But I just don't think scanf was made for this sort of format in mind. The tool's trying to be smart of helpful and it's biting you in the ass.

But if it's columnated data and you know the position of the various fields, there's a really easy work around. Just extract the field you want.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char** argv)
{
  char line[] = "xxxx  893.14";
  char tmp[100];
  int thatDamnNumber;
  float myfloatykins;

  //Get that field
  memcpy(tmp, line+4, 4);
  sscanf(tmp, "%d", &thatDamnNumber);

  //Kill that field so it doesn't goober-up the float
  memset(line+4, ' ', 4);
  sscanf(line, "%*4c%f", &myfloatykins);

  printf("%d %f\n", thatDamnNumber, myfloatykins);

  return 0;

}

If there is a lot of this, you could make some generalized functions: integerExtract(int positionStart, int sizeInCharacters), floatExtract(), etc.