Visual FoxPro - Memo field operations lead to "String is too long to fit" error

Question

Fox: Visual FoxPro 6 / SP5 & Visual FoxPro 9 / SP2 OS: Windows 7 Professional & Windows 10 Professional

I have a legacy application that imports data from a text file of wage information. Each data line is SSN, Name, Wage. As you can expect, these wage files can get quite large, a few hundred MB sometimes.

The data from the wage file is imported into a memo field of a FoxPro table.

append memo xmemo.xmemo from (m.drive_file)

This works as it should, even with files nearing 400MB.

Then there's some code that verifies and checks the new data in the memo field for things that may need to be stripped out. (linefeeds and carriage returns). The code that does this uses the OCCURS() function.

m.nchr10 = occurs(chr(10), xmemo.xmemo) && count for line feeds

m.nchr13 = occurs(chr(13), xmemo.xmemo) && count for carriage returns

Within the last week, the above two lines have failed with the error "String is too long to fit". Smaller wage files (about 60MB or so), will work fine. But anything larger than that will throw the error. I can even reproduce the error from the command line with

?occurs("x", xmemo.xmemo) && show a count of 'x' characters.

Background:

1. This application has been in production for over 20 years and has taken large files without issue all of that time.

2. I can reproduce the error in Fox 6 and Fox 9.

At this point, I'm stumped. Anyone have any idea what the issue is? Thanks in advance to anyone that can offer some help.

Answer 1

For each memo, I would copy it to a file (COPY MEMO MemoFieldName TO FileName), Fopen() it and use fgets() to process the file line by line

Answer 2

First....thanks to everyone that's replied so far. I'm sorry if this seems confusing. It's confusing to me as well.

Second, I was wrong about the working file size. The memo field can accept data up to 2GB. But if you use a string operation against any memo field holding more than about 16MB worth of data, you will get the 'String is too long to fit' error. This is the part that's new and confusing. It's CLEARLY in the documentation as such, but like I said...this has worked without error for a very long time with very large files.

As for why it's not working:

1. There HAS BEEN very recent updates to my Win 7 and Win 10 PC's that I use to develop. The timing does coincide with when this stopped working correctly.

2. The error occurs regardless of the contents of the memo field (thank you, Tamar).

3. No changes have been made to the import code. I'm sure of this, as I'm the only FoxPro Dev here. I'm not the original Dev for the app, but it's been under my umbrella and mine alone for about 6 years and I've never needed to work with the 'import' module.

4. The data isn't funky. It's just alpha and numeric text. We deal with wages and they come to us in files that are formatted like below. The Fox app takes the entire file of data, brings it into the memo field, checks it for anything that needs to be stripped out, then processes the file line by line.

    E2021000000000        9999999999       99      12212021119999
    S999999999LASTNAME   FIRSTNAME   X47   00000000000000  12212021119999
    S999999999LASTNAME   FIRSTNAME   X47   00000000000000  12212021119999
    S999999999LASTNAME   FIRSTNAME   X47   00000000000000  12212021119999
    S999999999LASTNAME   FIRSTNAME   X47   00000000000000  12212021119999
    S999999999LASTNAME   FIRSTNAME   X47   00000000000000  12212021119999
    S999999999LASTNAME   FIRSTNAME   X47   00000000000000  12212021119999
    S999999999LASTNAME   FIRSTNAME   X47   00000000000000  12212021119999
    S999999999LASTNAME   FIRSTNAME   X47   00000000000000  12212021119999
    S999999999LASTNAME   FIRSTNAME   X47   00000000000000  12212021119999
    F99999999
   

At this point, I've spent way too much time trying to figure it out. I’m going to chalk it up the Windows updates or just…magic. My solution, at least until the app can be truly replaced, is to take any large files (15MB+) from my Users, split them up into 15-16MB chunks, and then let them get processed individually. As you can imagine, this takes a chunk of my time, but this is wage data and it's time sensitive.

Thanks again, to anyone that spent any brain power on this.