Reading text Files - single line vs. multiple lines

Question

I am working on a particular scenario, where I have to read from a Text File, parse it, extract meaningful information from it, perform SQL queries with the information and then produce a reponse, output file.

I have about 3000 lines of code. Everything is working as expected. However I have been thinking of a connendrum that could possibly dissrupt my project.

The text file being read (lets call it Text.txt) may consist of a single line or multiple lines.

In my case, a 'line' is identified by its segment name - say ISA, BHT, HB, NM1, etc... each segment ending is identified by a special character '~'.

Now if the file consists of multiple lines (such that each line corresponds to a single segment); say:-

ISA....... ~

NM1....... ~

DMG....... ~

SE........ ~

and so on.... then my code essentially reads each 'line' (i.e. each segment), one at a time and stores it into a temp buffer using the following command :-

         ReadLn(myFile,buffer);

and then performs evaluations based on each line. Produces the desired output. No problems.

---

However the issue is... what if the file consists of only a single line (consisting of multiple segments), represented as:-

ISA....... ~NM1....... ~DMG....... ~SE........ ~

then with my ReadLine command I read the entire line instead of each segment, one at a time. This doesn't work for my code.

I was thinking about creating an if, else statement pair...which is based on how many lines my Txt.txt file consists of..such as:-

if line = 1:- then extract each segment at a time...seperated by the special character '~' perform necessary tasks (3000 lines of code) else if line > 1:- then extract each line at a time (corresponding to each segment) perform necessary tasks (3000 lines of code).

now the 3000 lines of code is repeated twice and I don't find it elegant to copy and paste all of that code twice.

I would appreciate if I could get some feedback on how to possibly solve this issue, such that, regardless of a one-line file or multiple-line file...when i proceed to evaluate, i only use one segment at a time.

Answer 1

There are many possible ways of doing this. Which is best for you might depend on how long these files are and how important performance is.

A simple solution is to just read characters one at a time until you hit your tilde delimiter. The routine ReadOneItem below shows how this can be done.

procedure TForm1.Button1Click(Sender: TObject);
const
  FileName = 'c:\kuiper\test2.txt';
var
  MyFile : textfile;
  Buffer : string;

  // Read one item from text file MyFile.
  // Load characters one at a time.
  // Ignore CR and LF characters
  // Stop reading at end-of-file, or when a '~' is read

  function ReadOneItem : string;
  var
    C : char;
  begin
    Result := '';

    // loop continues until break
    while true do
      begin

        // are we at the end-of-file? If so we're done
        if eof(MyFile) then
          break;

        // read in the next character
        read ( MyFile, C );

        // ignore CR and LF
        if ( C = #13 ) or ( C = #10 ) then
          {do nothing}
        else
          begin

            // add the character to the end
            Result := Result + C;

            // if this is the delimiter then stop reading
            if C = '~' then
              break;
          end;
      end;
  end;


begin
  assignfile ( MyFile, FileName );
  reset ( MyFile );
  try

    while not EOF(MyFile) do
      begin
        Buffer := ReadOneItem;
        Memo1 . Lines . Add ( Buffer );
      end;

  finally
    closefile ( MyFile );
  end;
end;

Answer 2

I would use a file mapping via the Win32 API CreateFileMapping() and MapViewOfFile() functions, and then just parse the raw data as-is, scanning for ~ characters and ignoring any line breaks you might encounter in between each segment. For example:

var
  hFile: THandle;
  hMapping: THandle;
  pView: Pointer;
  FileSize, I: DWORD;
  pSegmentStart, pSegmentEnd: PAnsiChar;
  sSegment: AnsiString;
begin
  hFile := CreateFile('Path\To\Text.txt', GENERIC_READ, FILE_SHARE_READ, nil, OPEN_EXISTING, 0, 0);
  if hFile = INVALID_HANDLE_VALUE then RaiseLastOSError;
  try
    FileSize := GetFileSize(hFile, nil);
    if FileSize = INVALID_FILE_SIZE then RaiseLastOSError;
    if FileSize > 0 then
    begin
      hMapping := CreateFileMapping(hFile, nil, PAGE_READONLY, 0, FileSize, nil);
      if hMapping = 0 then RaiseLastOSError;
      try
        pView := MapViewOfFile(hMapping, FILE_MAP_READ, 0, 0, FileSize);
        if pView = nil then RaiseLastOSError;
        try
          pSegmentStart := PAnsiChar(pView);
          pSegmentEnd := pSegmentStart;
          I := 0;
          while I < FileSize do
          begin
            if pSegmentEnd^ = '~' then
            begin
              SetString(sSegment, pSegmentStart, Integer(pSegmentEnd-pSegmentStart));
              // use sSegment as needed...
              pSegmentStart := pSegmentEnd + 1;
              Inc(I);
              while (I < FileSize) and (pSegmentStart^ in [#13, #10]) do
              begin
                Inc(pSegmentStart);
                Inc(I);
              end;
              pSegmentEnd := pSegmentStart;
            end else
            begin
              Inc(pSegmentEnd);
              Inc(I);
            end;
          end;
          if pSegmentEnd > pSegmentStart then
          begin
            SetString(sSegment, pSegmentStart, Integer(pSegmentEnd-pSegmentStart));
            // use sSegment as needed...
          end;
        finally
          UnmapViewOfFile(pView);
        end;
      finally
        CloseHandle(hMapping);
      end;
    end;
  finally
    CloseHandle(hFile);
  end;