CODEWITHSUNDEEP
c++regexboostutf-8
user695652
user695652

Reputation: 4275

Running Ascii regex over non-ASCII characters with UTF-8

#include <boost/regex.hpp>

#include <string>
#include <vector>
#include <iostream>

int main(int argc, char* argv[]) {

    std::string text = argv[1];
    std::string patterns = argv[2];

    boost::regex regex = boost::regex(patterns);

    boost::smatch match;

    std::cout << boost::regex_search(text, match, regex) << std::endl;    
}

If I run the program over the input hello¿ ¿ (containing a non-ASCII character with UTF-8 encoding) it returns 0 i.e. not found, but if I run it over the input hel√ √ (again containing non-ascii) it returns 1, i.e. found.

My question: What is the expected behavior of boost::regex (i.e. the ascii version) when run over utf characters?

Edit: Thanks for all the comments, I am still interested as to why exactly 1 is output, since both the text and the regex contain non-ascii characters. My guess would be that the bytes are interpreted as ascii and thus they match.

Upvotes: 12

Views: 2375

Answers (2)

Roland
Roland

Reputation: 336

It is a bug and not a feature: I tried your example on a better system (g++ 4.9.2 on Windows MinGW) and all happens well:

#include <iostream>
#include <string>
#include <regex>
int main()
{ std::string text ="hello¿"; // or "hello√"
  std::string patterns ="¿";  // or "√"
  std::regex regex = std::regex(patterns);
  std::smatch match;
  std::cout << std::regex_search(text, match, regex) << std::endl;
}

with output:

1

Upvotes: 0

Martin Lemburg
Martin Lemburg

Reputation: 507

  1. Using regular expressions on ASCII strings, is about using "bytes" to find a pattern in.
    Using regular expressions on UTF-8 strings, is about using regular expressions on "multi-byte" sequences, where a sequence represents a Unicode code point.

    Thus the regular expression gets applied to a Unicode string with an encoding with variable byte-count per character.

    UTF-8 strings contain multi-byte sequences with 1 to 4 bytes, which representing a Unicode "character". In UTF-8 only ASCII 7 bit characters are 1 byte "wide".

    So - using an ASCII regular expression engine on an UTF-8 encoded string, ignores the multi-byte sequences in the UTF-8 encoded string and causes a pattern matching byte by byte. The results of this ASCII regular expression engine usage on an UTF-8 encoded string is invalid.

    Please take a look at http://utfcpp.sourceforge.net.

    To get the regular expressions working on UTF-8 encoded strings, you have to …

    • have UTF-8 string iterators usable with the regular expressions, or
    • use std::codecvt_utf8 in combination of setting temporarily the global locale to get the regular expression working, or
    • have to convert the UTF-8 encoded string into a UTF-16 encoded string to be used with an Unicode regular expression engine - based on std::wstring.

  2. The regex_search function returns a boolean and true on a match.
    In your case the ASCII regular expression pattern matches a part of the UTF-8 encoded string, which is parsed invalidly as ASCII string - as you assumed!
    If you have English text in an UTF-8 encoded string, then an ASCII regular expression engine can be used safely. Leaving the ASCII 7 bit range, makes the result of the ASCII regular expression engine unreliable.

Upvotes: 6

Related Questions