Exception thrown for large number of Vertx connecting to Redis

Question

Trying to simulate scenario for heavy load with Redis (default config only). To keep it simple, when multi is issued immediately excute then close the connection.

  import io.vertx.core.*;
  import io.vertx.core.json.Json;
  import io.vertx.redis.RedisClient;
  import io.vertx.redis.RedisOptions;
  import io.vertx.redis.RedisTransaction;


  class MyVerticle extends AbstractVerticle {
      private int index;

      public MyVerticle(int index) {
          this.index = index;
      }

      private void run2() {
          RedisClient client = RedisClient.create(vertx, new RedisOptions().setHost("127.0.0.1"));
          RedisTransaction tr = client.transaction();

          tr.multi(ev2 -> {
              if (ev2.succeeded()) {
                tr.exec(ev3 -> {
                    if (ev3.succeeded()) {
                        tr.close(i -> {
                            if (i.failed()) {
                                System.out.println("FAIL TR CLOSE");
                                client.close(j -> {
                                    if (j.failed()) {
                                        System.out.println("FAIL CLOSE");
                                    }
                                });
                            }
                        });
                    }
                    else {
                        System.out.println("FAIL EXEC");
                        tr.close(i -> {
                            if (i.failed()) {
                                System.out.println("FAIL TR CLOSE");
                                client.close(j -> {
                                    if (j.failed()) {
                                        System.out.println("FAIL CLOSE");
                                    }
                                });
                            }
                        });
                    }
                });
              }
              else {
                  System.out.println("FAIL MULTI");
                  tr.close(i -> {
                      if (i.failed()) {
                          client.close(j -> {
                              if (j.failed()) {
                                  System.out.println("FAIL CLOSE");
                              }
                          });
                      }
                  });
              }
          });
      }

      @Override
      public void start(Future<Void> startFuture) {
          long timerID = vertx.setPeriodic(1, new Handler<Long>() {
              public void handle(Long aLong) {
                  run2();
              }
          });
      }

      @Override
      public void stop(Future stopFuture) throws Exception {
          System.out.println("MyVerticle stopped!");
      }

  }

  public class Periodic {
      public static void main(String[] args) {
          Vertx vertx = Vertx.vertx();

          for (int i = 0; i < 8000; i++) {
              vertx.deployVerticle(new MyVerticle(i));
          }
      }
  }

Although connections are closed properly I still get warning errors. All of them are thrown even before I put more logic within multi.

2017-06-20 16:29:49 WARNING io.netty.util.concurrent.DefaultPromise notifyListener0 An exception was thrown by io.vertx.core.net.impl.ChannelProvider$$Lambda$61/1899599620.operationComplete()
    java.lang.IllegalStateException: Uh oh! Event loop context executing with wrong thread! Expected null got Thread[globalEventExecutor-1-2,5,main]    
    at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:316)
    at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:193)
    at io.vertx.core.net.impl.NetClientImpl.failed(NetClientImpl.java:258)
    at io.vertx.core.net.impl.NetClientImpl.lambda$connect$5(NetClientImpl.java:233)
    at io.vertx.core.net.impl.ChannelProvider.lambda$connect$0(ChannelProvider.java:42)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
  at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
    at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
    at io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:233)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
    at java.lang.Thread.run(Thread.java:745)

Is there a reason for this error ?

Answer 1

You'll continue to get errors, because you test the wrong things.

First of all, vertices are not fat coroutines. They are thin actors. Meaning creating 500 of them won't speed things up, but probably will slow everything down, because event loop still needs to switch between them.

Second, if you want to prepare for 2K concurrent requests, put your Vertx application on one machine, and run wrk or similar tool over the network.

Third, your Redis is also on the same machine. I hope that won't be the case in your production, since Redis will compete with Vertx over CPU.

Once everything is setup correctly, I believe that you'll be able to handle 10K requests quite easily. I've seen Vertx handle 8K requests on modest machines with PostgreSQL.