CODEWITHSUNDEEP
windowsuser-interfacedllstreamstdout
Joel Brandt
Joel Brandt

Reputation: 176

Using stdout in a Win32 GUI application: crashes if I don't have a redirect to file in arguments

I'm building a Win32 GUI app. Inside that app, I'm using a DLL that was intended to be used in a command line app.

Suppose Foo.exe is my GUI app, and bar() is a function in the DLL that prints "hello" to stdout. Foo.exe calls bar().

If I run Foo.exe from the command line with a redirect (>) (i.e. Foo.exe > out.txt) it writes "hello" to out.txt and exits normally (as expected).

However, if I run Foo.exe without a redirect (either from cmd.exe or by double-clicking in Windows Explorer), it crashes when bar() is called.

If I run Foo.exe inside the debugger with the redirect in the command line (set through VS's properties for the project) and call "GetStdHandle(STD_OUTPUT_HANDLE)", I get a reasonable address for a handle. If I call it without the redirect in the command line, I get 0.

Do I need something to "initialize" standard out? Is there a way that I can set up this redirect in the application startup? (Redirecting to a file would be ideal. But just throwing out the data printed by the DLL would be okay, too.)

Finally, I suspect that the DLL is writing to stdout through the CRT POSIX-like API, because it is a cross-platform DLL. I don't know if this matters.

I've tried creating a file with CreateFile and calling SetStdHandle, but that doesn't seem to work. I may be creating the file incorrectly, however. See code below.

HANDLE hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
// hStdOut is zero  

HANDLE hFile;
hFile = CreateFile(TEXT("something.txt"),        // name of the write
                   GENERIC_WRITE,               // open for writing
                   0,                           // do not share
                   NULL,                        // default security
                   CREATE_NEW,             // create new file only
                   FILE_ATTRIBUTE_NORMAL,  // normal file
                   NULL);                  // no attr. template
BOOL r = SetStdHandle(STD_OUTPUT_HANDLE, hFile) ;   
hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
// hStdOut is now equal to hFile, and r is 1

bar();
// crashes if there isn't a redirect in the program arguments

UPDATE: I just found this article: http://support.microsoft.com/kb/105305. It states "Note that this code does not correct problems with handles 0, 1, and 2. In fact, due to other complications, it is not possible to correct this, and therefore it is necessary to use stream I/O instead of low-level I/O."

My DLL definitely uses file handles 0,1 and 2. So, there may be no good solution to this problem.

I'm working on a solution that checks for this case, and re-launches the exe appropriately using CreateProcess. I'll post here when I'm done.

Upvotes: 3

Views: 3562

Answers (3)

PeterSvP
PeterSvP

Reputation: 2046

Can you tell me which library do you use? This problem have good solution. Write small stub launcher EXE (in GUI mode but with NO windows!) that have your icon and that all shortcuts launch. Make this stub EXE "CreateProcess" the real EXE with redirected output to "NUL" or "CON", or, CreateProcess() it suspended, take its STDOUT, doing nothing with it. This way, your original EXE should work without visible console, but will actually have where to write - in the handles 0,1 and 2 that are taken by the parent invisible stub EXE. Note that killing the parent EXE may make the child lost its handles - and - crash.

You may end up with two processes in Task Manager. So you can try making these 2 processes a job like Google Chrome does.

On your question Do I need something to "initialize" standard out? - only your parent / launcher can pre-initialize your STDOUT for handles 0,1 and 2 "properly".

Upvotes: 0

Francesco
Francesco

Reputation: 125

The solution that I've found is the following:

  • obtain a valid File HANDLE in some way to direct the standard output. Lets call the file handle "fh". (please note that on Windows a File HANDLE is not the same thing of a file descriptor)
  • associate a file descriptor to the file handle with _open_osfhandle (see http://msdn.microsoft.com/en-us/library/kdfaxaay.aspx for details) Lets call "fd" the new file descriptor, an int value.
  • call dup2 to associate the STDOUT_FILENO to the given file descriptor: dup2(fd, STDOUT_FILENO)
  • create a file strem associated to the stdout file descriptor FILE* f = _fdopen(STDOUT_FILENO, "w");
  • memset stdout to the content of f: *stdout = *f
  • call SetStdHandle on the given file handle: SetStdHandle(STD_OUTPUT_HANDLE, ofh);

Please note that I've not tested exactly this sequence but something slightly different.

I don't know if some steps are redundant.

In any case the following article explain very well the concept of file handle, descriptor et fiel stream:

http://dslweb.nwnexus.com/~ast/dload/guicon.htm

Upvotes: 2

ixe013
ixe013

Reputation: 10171

You must build foo.exe as a console application with the /SUBSYSTEM switch. Windows will allocate a console (stdout) for your application automatically, which can be :

  • The current console
  • A redirection to a file
  • A pipe to another program's STDIN

If you build foo.exe as a GUI application, the console is not allocated by default, wich explains the crash

If you must use the GUI subsystem, it can still be done with AllocConsole. This old WDJ article has sample code to help you.

Upvotes: 0

Related Questions