CODEWITHSUNDEEP
pythonpandascorpuscsv
user1700890
user1700890

Reputation: 7732

Reading MSR paraphrase corpus into Pandas

I downloaded MSR paraphrase corpus from MSR and tried loading it into dataframe, but got the following error:

import pandas as pd
df = pd.read_csv(r'C:\MSRParaphraseCorpus\msr_paraphrase_test.txt', sep = '\t' )

ERROR: 

CParserError                              Traceback (most recent call last)
<ipython-input-10-35d992467320> in <module>()
----> 1 df = pd.read_csv(r'C:\MSRParaphraseCorpus\msr_paraphrase_test.txt', sep = '\t' )

c:\python34\lib\site-packages\pandas\io\parsers.py in parser_f(filepath_or_buffer, sep, dialect, compression, doublequote, escapechar, quotechar, quoting, skipinitialspace, lineterminator, header, index_col, names, prefix, skiprows, skipfooter, skip_footer, na_values, na_fvalues, true_values, false_values, delimiter, converters, dtype, usecols, engine, delim_whitespace, as_recarray, na_filter, compact_ints, use_unsigned, low_memory, buffer_lines, warn_bad_lines, error_bad_lines, keep_default_na, thousands, comment, decimal, parse_dates, keep_date_col, dayfirst, date_parser, memory_map, float_precision, nrows, iterator, chunksize, verbose, encoding, squeeze, mangle_dupe_cols, tupleize_cols, infer_datetime_format, skip_blank_lines)
    472                     skip_blank_lines=skip_blank_lines)
    473 
--> 474         return _read(filepath_or_buffer, kwds)
    475 
    476     parser_f.__name__ = name

c:\python34\lib\site-packages\pandas\io\parsers.py in _read(filepath_or_buffer, kwds)
    258         return parser
    259 
--> 260     return parser.read()
    261 
    262 _parser_defaults = {

c:\python34\lib\site-packages\pandas\io\parsers.py in read(self, nrows)
    719                 raise ValueError('skip_footer not supported for iteration')
    720 
--> 721         ret = self._engine.read(nrows)
    722 
    723         if self.options.get('as_recarray'):

c:\python34\lib\site-packages\pandas\io\parsers.py in read(self, nrows)
   1168 
   1169         try:
-> 1170             data = self._reader.read(nrows)
   1171         except StopIteration:
   1172             if nrows is None:

pandas\parser.pyx in pandas.parser.TextReader.read (pandas\parser.c:7566)()

pandas\parser.pyx in pandas.parser.TextReader._read_low_memory (pandas\parser.c:7806)()

pandas\parser.pyx in pandas.parser.TextReader._read_rows (pandas\parser.c:8423)()

pandas\parser.pyx in pandas.parser.TextReader._tokenize_rows (pandas\parser.c:8297)()

pandas\parser.pyx in pandas.parser.raise_parser_error (pandas\parser.c:20715)()

CParserError: Error tokenizing data. C error: Expected 5 fields in line 34, saw 6

So I looked at line 34, and it looks perfectly fine. 

  fname = r'C:\MSRParaphraseCorpus\msr_paraphrase_test.txt'
    with open(fname, encoding="utf8") as f:
        content = f.readlines()

    content[34]

Output:

'0\t1268500\t1268733\tAgainst the Japanese currency, the euro was at 135.92/6.04 yen against the late New York level of 136.03/14.\tThe dollar was at 117.85 yen against the Japanese currency, up 0.1 percent.\n'

Upvotes: 0

Views: 713

Answers (1)

clockwatcher
clockwatcher

Reputation: 3363

The problem is with the open quotes on line 34 (as mentioned in my comment). Disable the csv reader's quoting by passing in csv.QUOTE_NONE. Try:

import csv
import pandas as pd

df = pd.read_csv(r'C:\MSRParaphraseCorpus\msr_paraphrase_test.txt', sep = '\t', quoting=csv.QUOTE_NONE)

Upvotes: 3

Related Questions