Testing the Chatbots using botium and testmybot packages

Question

How to test the chatbots using botium or testmybot node packages?

I am not able to find any end to end sample to understand this.

Answer 1

There are several samples included in the Github repositories for Testmybot and Botium.

The Botium Wiki contains some useful information and a Walkthrough.

The basic steps for running a Botium script are as follows (from one of the samples)

Install requirements

• Install Node.js
• Install docker
• Install docker-compose

There are other samples available which don't require docker.

Initialize Botium directory

Open a command line window, create a directory, initialize NPM and download Botium package.

mkdir botium
cd botium
npm init
npm install --save botium-core

Load botium library

First, load the botium library and required classes.

const BotDriver = require('botium-core').BotDriver
const Capabilities = require('botium-core').Capabilities
const Source = require('botium-core').Source

Configure capabilities

Tell Botium what kind of chatbot is under test and how to connect to it. In this sample, the chatbot should be loaded into a docker container and Botium has to hook into the Microsoft Bot Framework.

const driver = new BotDriver()
    .setCapability(Capabilities.PROJECTNAME, 'core-CreateNewConversation')
    .setCapability(Capabilities.CONTAINERMODE , 'docker')
    .setCapability(Capabilities.BOTFRAMEWORK_API, true)
    .setCapability(Capabilities.BOTFRAMEWORK_APP_ID, 'my microsoft app id')
    .setCapability(Capabilities.BOTFRAMEWORK_CHANNEL_ID, 'facebook')

Configure chatbot repository

Botium retrieves the chatbot code directly from the source Github repository. As an alternative, the repository could be cloned first and loaded from a local directory. In a CI environment, loading from Git usually makes more sense. Additionally, the command to initialize the Git repository ("npm install"), the command to start the chatbot service ("npm start") and some environment variables are required to run the sample chatbot.

driver.setSource(Source.GITURL, 'https://github.com/Microsoft/BotBuilder-Samples.git')
    .setSource(Source.GITDIR, 'Node/core-CreateNewConversation')
    .setSource(Source.GITPREPARECMD, 'npm install')
    .setCapability(Capabilities.STARTCMD, 'npm start')
    .setEnv('MICROSOFT_APP_ID', 'my microsoft app id')
    .setEnv('MICROSOFT_APP_PASSWORD', 'my microsoft app password')
    .setEnv('NODE_DEBUG', 'botbuilder')
    .setEnv('DEBUG', '*')

Running a conversation and evaluate response

Botium provides a "fluent interface".

First, the Botium driver is initialized (work directory created, repository downloaded, docker network constructed, ...) and started.

driver.BuildFluent()
    .Start()
    ...

Then, a conversation is started by sending input to the chatbot ("UserSaysText") or by waiting for a reaction from the chatbot ("WaitBotSaysText"). The conversation is tailored to the chatbot in use. In case chatbot doesn't react or shows unexpected reaction, the conversation is ended immediately.

...
.UserSaysText('hi bot')
.WaitBotSaysText((text) => assert('You\'ve been invited to a survey! It will start in a few seconds...', text))
.WaitBotSaysText(null, 10000, (text) => assert('Hello... What\'s your name?', text))
.UserSaysText('John')
.WaitBotSaysText((text) => assert('Hi John, How many years have you been coding?', text))
.UserSaysText('5')
.WaitBotSaysText((text) => assert('What language do you code Node using?', text))
.UserSaysText('CoffeeScript')
.WaitBotSaysText((text) => assert('Got it... John you\'ve been programming for 5 years and use CoffeeScript.', text))
...

In the end, Botium is stopped and some cleanup tasks are performed. Don't forget the "Exec" call, otherwise nothing will be executed at all!

...
.Stop()
.Clean()
.Exec()
...

Run the program and watch output

Now run the program as usual in a command line window.

[ec2-user@ip-172-30-0-104 botframework]$ node botiumFluent.js
SUCCESS: Got Expected <You've been invited to a survey! It will start in a few seconds...>
SUCCESS: Got Expected <Hello... What's your name?>
SUCCESS: Got Expected <Hi John, How many years have you been coding?>
SUCCESS: Got Expected <What language do you code Node using?>
SUCCESS: Got Expected <Got it... John you've been programming for 5 years and use CoffeeScript.>
READY
[ec2-user@ip-172-30-0-104 botframework]$

TestMyBot

Botium is comparable to what Selenium/Appium are doing (unified API and "Page Object Model"). TestMyBot is a layer above Botium to integrate Botium with CI/CD pipelines and test runners like Mocha and Jasmine. The conversations don't have to be coded as above, but are "scripted" in text files, excel files or yml files, for example:

urvey

#me
hi

#bot
You've been invited to a survey! It will start in a few seconds...

#bot
Hello... What's your name?

#me
John

#bot
Hi John, How many years have you been coding?

#me
10

#bot
What language do you code Node using? 

#me
C#

#bot
I didn't understand. Please choose an option from the list.

#me
JavaScript

#bot
Got it... John you've been programming for 10 years and use JavaScript.

All of these files have to be placed in the directory spec/convo, and the test cases for Jasmine or Mocha (or any other test runner) are created on-the-fly with just a short scriptlet (placed into spec/testmybot.spec.js):

const bot = require('testmybot');

bot.helper.jasmine().setupJasmineTestSuite(60000);

It really helps to have knowledge of Jasmine or Mocha. When correctly set up, the only command to run is:

npm run test