CODEWITHSUNDEEP
ctextnon-english
mohxinn
mohxinn

Reputation: 19

Reading text file with non-english character in C

Is it possible to read a text file hat has non-english text?

Example of text in file:

E 37

SVAR:

Fettembolisyndrom. (1 poäng)

Example of what is present in buffer which stores "fread" output using "puts" :

E 37 SVAR:

Fettembolisyndrom. (1 poäng)

Under Linux my program was working fine but in Windows I am seeing this problem with non-english letters. Any advise how this can be fixed?

Program:

#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <string.h>

int debug = 0;

int main(int argc, char* argv[])
{
    if (argc < 2)
    {
        puts("ERROR! Please enter a filename\n");
        exit(1);
    }
    else if (argc > 2)
    {
        debug = atoi(argv[2]);
        puts("Debugging mode ENABLED!\n");
    }

    FILE *fp = fopen(argv[1], "rb");
    fseek(fp, 0, SEEK_END);
    long fileSz = ftell(fp);
    fseek(fp, 0, SEEK_SET);

    char* buffer;
    buffer = (char*) malloc (sizeof(char)*fileSz);
    size_t readSz = fread(buffer, 1, fileSz, fp);
    rewind(fp);

    if (readSz == fileSz)
    {
        char tmpBuff[100];
        fgets(tmpBuff, 100, fp);

        if (!ferror(fp))
        {
            printf("100 characters from text file: %s\n", tmpBuff);
        }
        else
        {
            printf("Error encounter");
        }
    }

    if (strstr("FRÅGA",buffer) == NULL)
    {
        printf("String not found!");
    }

    return 0;
}

Sample output

Text file

Upvotes: 0

Views: 1942

Answers (1)

AlexP
AlexP

Reputation: 4430

Summary: If you read text from a file encoded in UTF-8 and display it on the console you must either set the console to UTF-8 or transcode the text from UTF-8 to the encoding used by the console (in English-speaking countries, usually MS-DOS code page 437 or 850).

Longer explanation

Bytes are not characters and characters are not bytes. The char data type in C holds a byte, not a character. In particular, the character Å (Unicode <U+00C5>) mentioned in the comments can be represented in many ways, called encodings:

  • In UTF-8 it is two bytes, '\xC3' '\x85';
  • In UTF-16 it is two bytes, either '\xC5' '\x00' (little-endian UTF-16), or '\x00' '\xC5' (big-endian UTF-16);
  • In Latin-1 and Windows-1252, it is one byte, '\xC5';
  • In MS-DOS code page 437 and code page 850, it is one byte, '\x8F'.

It is the responsibility of the programmer to translate between the internal encoding used by the program (usually but not always Unicode), the encoding used in input or output files, and the encoding expected by the display device.

Note: Sometimes, if the program does not do much with the characters it reads and outputs, one can get by just by making sure that the input files, the output files, and the display device all use the same encoding. In Linux, this encoding is almost always UTF-8. Unfortunately, on Windows the existence of multiple encodings is a fact of life. System calls expect either UTF-16 or Windows-1252. By default, the console displays Code Page 437 or 850. Text files are quite often in UTF-8. Windows is old and complicated.

Upvotes: 1

Related Questions