How to send results from TestComplete directly to TeamCity

Question

A previous post on this subject has been helpful (see How to export TestExecute/TestComplete results to teamcity?)

Right now, our TestComplete solution creates a text file that MSBuild consumes with a Microsoft.Build.Utilities.Task helper that sends the messages back up to TeamCity. However, I'm wondering if we can do better.

The TeamCity help topic (http://confluence.jetbrains.com/display/TCD65/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ServiceMessages) says, messages "should be printed into standard output stream of the build". What I'm trying to figure out is, can I directly access that stream from the TestComplete script?

I was hoping that it would be something as simple as: dotNET.System.Console.WriteLine_11("##teamCity[testSuiteStarted name='Foo']");

But that is obviously not giving me the correct output stream. Thoughts? Can it be done?

Answer 1

If you are using TestExecute this has become "Relatively" simple recently

Execute TestExecute with the following parameters:

/exportlog:"log\Index.html" /SilentMode /DoNotShowLog

For a full rundown of TestExecute parameters check: https://support.smartbear.com/testexecute/docs/running/automating/command-line/command-line.html

Setup your TestComplete project with a build level report:

Please be aware it is case sensitive

On your build configuration, in the general settings tab, export the generated log folder as an artifact.

Once the build has run, you will be able to see the test log embedded within TeamCity:

This is using TestComplete V14 and TeamCity 2019.1

Answer 2

The issue you are experiencing is that TestComplete does not support writing to the standard output stream, even if you invoke the CLR Console.WriteLine, you are writing to the process named tcHostingProcess.exe in which all CLR objects exist.

In order to work around this, you need a console application running that can accept messages from your TestComplete project. There are a number of ways to do this, but here is a proposed solution using TCP/IP for the necessary IPC.

In TeamCity

In the TeamCity build steps, you want the TestComplete or TestExecute process to launch without holding up the build script and then to launch a custom console application which will receive the messages from TestComplete.

start TestComplete.exe [arg1] [arg2] ...
TCConsoleHost.exe

Console Host Application

The console host program will start a TcpListener, and once a client connects, it will read messages from the resulting NetworkStream object and print them to the console. This program will continue until there is an error in reading from the stream (i.e. TestComplete has exited).

class Program
{
    static void Main(string[] args)
    {
        TcpListener listener = new TcpListener(IPAddress.Parse("127.0.0.1"), 9800);
        listener.Start();
        TcpClient client = listener.AcceptTcpClient();
        NetworkStream stream = client.GetStream();
        BinaryReader reader = new BinaryReader(stream);

        while (true)
        {
            try
            {
                string message = reader.ReadString();
                Console.WriteLine(message);
            }
            catch
            {
                break;
            }
        }
    }
}

Message Client Class

Likewise, we can create a TcpClient which can connect to our listening process and relay messages. The real trick here is to wrap this in a static class with a static constructor so that once TestComplete loads the CLR bridge, it will automatically connect and be ready to send the messages. This example has a TeamCity service message function SendMessage which automatically formats the message (including escaping single quotes).

public static class TCServiceMessageClient
{
    static BinaryWriter writer;
    static NetworkStream stream;

    static TCServiceMessageClient()
    {
        TcpClient client = new TcpClient();
        client.Connect("127.0.0.1", 9800);

        stream = client.GetStream();
        writer = new BinaryWriter(stream);
    }

    public static void SendMessage(string message)
    {
        writer.Write(string.Format("##teamcity[message text='{0}'", message.Replace("'","|'")));
    }
}

In TestComplete

Now, since the client will automatically connect when the CLR bridge is loaded, the only code necessary in TestComplete is:

dotNET["TCServiceMessageClient"]["TCServiceMessageClient"]["SendMessage"]("Hello TeamCity!");

Additional Notes

There are a few caveats with the above code, namely timing, reconnect, etc. A deployed solution should have better error handling of the network conditions. Additionally, it may be more desirable that the TCMessageHost application actually launches TestComplete directly with a System.Process object so that it can more reliably wait for the application to exit.

Answer 3

I'm working with TeamCity and TestComplete as well. I ran into the exact same problem, and you can't write to the console no matter what you try to do, at least not that I've found. I even tried writing a C# app that I used the CLR bridge in TestComplete to talk to it, and still nothing.

What we ended up doing was to create batch file that handles the TestExecute call. We then call the batch file from TeamCity. To call TestExecute use this (with your pjs file and /p specific to your tests):

start /wait TestExecute "{TestCaseID}.pjs" /r /e /p:"{TestCaseID}" /silentmode /forceconversion

Then we check the ERRORLEVEL property returned by TestExecute. A level of 2 means that the test execution failed. In that case, we print the following with an echo statement:

##teamcity[testFailed name='{TestCaseID} - {TestName}' message='TestExecute reported a failure' details='See artifacts']

After TestExecute closes, it will attempt to open the log file. We have our build servers set so that mht files are opened by the calculator since they take a while to open (set it as the default in the Windows Right-Click -> Open With menu). So, after TestExecute completes, it opens the calculator, then our batch file just kills the calculator using taskkill.

We then copy the result file from TestExecute into a specified location (this is a separate executable that we wrote). The we finish the test execution and publish our artifacts to TestExecute:

echo ##teamcity[testFinished name='{TestCaseID} - {TestName}' duration='%milliSecondsElapsed%']
echo ##teamcity[publishArtifacts 'C:\BuildResults\{GroupName}\{TestCaseID}\*.mht']

One thing to note: We were never able to get TestExecute to reliably run a single test at a time, so we actually use a list of our test cases to generate separate pjs and mds files for each of our test cases. Then we use those generated pjs and mds files in TestExecute. That's where the {GroupName}, {TestCaseID}, and {TestName} variables from above come from. Those are specific to our implementation, but you may need different indicators there.

Hope this helps!