CODEWITHSUNDEEP
databaseyugabytedb
user4955663
user4955663

Reputation: 1071

YugaByte CREATE DATABASE failure

I tried to run the yugabyte 2.0.8.0 quick start demo according to these instructions:

https://docs.yugabyte.com/latest/quick-start/install/

https://docs.yugabyte.com/latest/quick-start/create-local-cluster/

with 3 local nodes

./bin/yb-ctl --rf 3 create

And then explore ysql:

https://docs.yugabyte.com/latest/quick-start/explore-ysql/

but the db creation fails:

yugabyte=# CREATE DATABASE yb_demo;
CREATE DATABASE yb_demo;
ERROR:  Timed out: Write(tablet: 00000000000000000000000000000000, num_ops: 1, num_attempts: 9, txn: 00000000-0000-0000-0000-000000000000) passed its deadline 3393.992s (passed: 65.317s): Remote error (yb/rpc/outbound_call.cc:440): Service unavailable (yb/tserver/tablet_service.cc:291): Soft memory limit exceeded (at 93.81% of capacity), score: 0.00

yugabyte=# CREATE DATABASE yb_demo;
CREATE DATABASE yb_demo;
ERROR:  Already present: Keyspace 'yb_demo' already exists

yugabyte=# DROP DATABASE yb_demo;
DROP DATABASE yb_demo;
ERROR:  database "yb_demo" does not exist

yugabyte=# \c yb_demo;
FATAL:  database "yb_demo" does not exist
Previous connection kept

So the db creation failed. I cannot recreate it but it seems that I also cannot remove it nor connect to it. The error message hints that there was not enough memory.

free output is: 

              total        used        free      shared  buff/cache   available
Mem:        4045992     1400276     1419504       64104     1226212     2319448
Swap:       4194300           0     4194300

Questions:

How much memory is required for running YugaByte? In this case the runtime environment is 4GB Ubuntu 16.4 VirtualBox VM with desktop. Your HW requirements say min 2GB https://docs.yugabyte.com/latest/deploy/checklist/.

How can I recover this error? Either drop the failed db or finalize the db creation?

Thank you for your support.

Upvotes: 1

Views: 661

Answers (1)

Mihnea Iancu
Mihnea Iancu

Reputation: 46

How much memory is required for running YugaByte? In this case the runtime environment is 4GB Ubuntu 16.4 VirtualBox VM with desktop. Your HW requirements say min 2GB https://docs.yugabyte.com/latest/deploy/checklist/.

Indeed, the 2GB minimum is intended to be per node. Since you are running a 3-node cluster on one machine, it should help to increase it to 6GB or more.

Alternatively, if increasing the memory is not an option, you can tune the memory limits for the processes. By default Yugabyte allocates 10% of total memory to the yb-master process (metadata server) and 85% to the yb-tserver process (tablet-server).

In your case, the memory issue is most likely with the yb-master process (CREATE DATABASE command will initialize the metadata for the new database). You can inspect the current memory settings and usage using the Web UI to confirm.

  • yb-master: <node-ip>/7000/mem-trackers
  • yb-tserver: <node-ip>:9000/memz

Note: for a 3-node yb-ctl cluster the node IPs are typically 127.0.0.1, 127.0.0.2, and 127.0.0.3.

To update the yb-master memory limit you can use the memory_limit_hard_bytes flag (same flag can be used for the yb-tserver). For instance, to increase it to 600MB:

./bin/yb-ctl stop && ./bin/yb-ctl start --master_flags="memory_limit_hard_bytes=629145600"

Note: You can also use the same options directly when creating a cluster:

./bin/yb-ctl destroy && ./bin/yb-ctl create --master_flags="memory_limit_hard_bytes=629145600"

How can I recover this error? Either drop the failed db or finalize the db creation?

Regarding the memory issue causing the new database to be left in that state, that can happen in some corner cases in 2.0.8 if the DDL command (CREATE DATABASE fails with a system error in the middle).

For now, the simplest fix, if this is a test cluster, is to destroy the cluster and then recreate it (either on a machine with more memory or with updated flags as shown above). Otherwise you can use the yb-admin tool to clean up the partial metadata.

./bin/yb-admin delete_namespace ysql.yb_demo

Note: Some commits to fix such cases have landed recently [1], [2] and should be in the next release (though I can't confirm if they will fix this particular issue).

Upvotes: 1

Related Questions