CouchDB suddenly crashes with "Could not open file" errors

Question

Experts,

I just had a surprising problem with my couchDB installation: From one moment to the other no requests were possible, and the couchDB was no longer running.

A looking into the error log revealed the following stack trace (these are the first ones, very many similar follow). The mentioned db-files do exist in the specified directories and have the correct access rights (couchdb:couchdb, 640).

System is FreeBSD 10.1-RELEASE-p16, couchdb 1.6.1

[Thu, 20 Aug 2015 13:37:29 GMT] [error] [<0.28818.72>] Could not open file /var/db/couchdb/db1/abc.couch: no such file or directory
[Thu, 20 Aug 2015 13:37:29 GMT] [error] [<0.25245.72>] Could not open file /var/db/couchdb/db1/abc.couch: no such file or directory
[Thu, 20 Aug 2015 13:50:45 GMT] [error] [<0.13142.73>] Could not open file /var/db/couchdb/db1/def.couch: no such file or directory
[Thu, 20 Aug 2015 13:52:29 GMT] [error] [<0.15770.73>] Could not open file /var/db/couchdb/db1/abc.couch: no such file or directory
[Thu, 20 Aug 2015 13:52:29 GMT] [error] [<0.17891.73>] Could not open file /var/db/couchdb/db1/abc.couch: no such file or directory
[Thu, 20 Aug 2015 13:59:47 GMT] [error] [<0.23972.73>] {error_report,<0.31.0>,
                        {<0.23972.73>,crash_report,
                         [[{initial_call,{couch_file,init,['Argument__1']}},
                           {pid,<0.23972.73>},
                           {registered_name,[]},
                           {error_info,
                            {exit,
                             {{badmatch,{error,eacces}},
                              [{couch_file,init,1,
                                [{file,"couch_file.erl"},{line,314}]},
                               {gen_server,init_it,6,
                                [{file,"gen_server.erl"},{line,328}]},
                               {proc_lib,init_p_do_apply,3,
                                [{file,"proc_lib.erl"},{line,239}]}]},
                             [{gen_server,init_it,6,
                               [{file,"gen_server.erl"},{line,352}]},
                              {proc_lib,init_p_do_apply,3,
                               [{file,"proc_lib.erl"},{line,239}]}]}},
                           {ancestors,[<0.25603.73>]},
                           {messages,[]},
                           {links,[#Port<0.1119512>,<0.25603.73>]},
                           {dictionary,[]},
                           {trap_exit,true},
                           {status,running},
                           {heap_size,376},
                           {stack_size,27},
                           {reductions,564}],
                          [{neighbour,
                            [{pid,<0.25603.73>},
                             {registered_name,[]},
                             {initial_call,{erlang,apply,2}},
                             {current_function,{gen,do_call,4}},
                             {ancestors,[]},
                             {messages,[]},
                             {links,[<0.85.0>,<0.23972.73>]},
                             {dictionary,[]},
                             {trap_exit,false},
                             {status,waiting},
                             {heap_size,610},
                             {stack_size,16},
                             {reductions,61}]}]]}}
[Thu, 20 Aug 2015 13:59:47 GMT] [error] [<0.23088.73>] httpd 500 error response:
 {"error":"error","reason":"{{badmatch,{error,eacces}},\n [{couch_file,init,1,[{file,\"couch_file.erl\"},{line,314}]},\n  {gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,328}]},\n  {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,239}]}]}"}

[Thu, 20 Aug 2015 13:59:47 GMT] [error] [<0.26398.73>] {error_report,<0.31.0>,
                        {<0.26398.73>,crash_report,
                         [[{initial_call,{couch_file,init,['Argument__1']}},
                           {pid,<0.26398.73>},
                           {registered_name,[]},
                           {error_info,
                            {exit,
                             {{badmatch,{error,eacces}},
                              [{couch_file,init,1,
                                [{file,"couch_file.erl"},{line,314}]},
                               {gen_server,init_it,6,
                                [{file,"gen_server.erl"},{line,328}]},
                               {proc_lib,init_p_do_apply,3,
                                [{file,"proc_lib.erl"},{line,239}]}]},
                             [{gen_server,init_it,6,
                               [{file,"gen_server.erl"},{line,352}]},
                              {proc_lib,init_p_do_apply,3,
                               [{file,"proc_lib.erl"},{line,239}]}]}},
                           {ancestors,[<0.15629.73>]},
                           {messages,[]},
                           {links,[#Port<0.1119284>,<0.15629.73>]},
                           {dictionary,[]},
                           {trap_exit,true},
                           {status,running},
                           {heap_size,610},
                           {stack_size,27},
                           {reductions,573}],
                          [{neighbour,
                            [{pid,<0.15629.73>},
                             {registered_name,[]},
                             {initial_call,{erlang,apply,2}},
                             {current_function,{proc_lib,sync_wait,2}},
                             {ancestors,[]},
                             {messages,
                              [{ack,<0.26398.73>,
                                {error,
                                 {{badmatch,{error,eacces}},
                                  [{couch_file,init,1,
                                    [{file,"couch_file.erl"},{line,314}]},
                                   {gen_server,init_it,6,
                                    [{file,"gen_server.erl"},{line,328}]},
                                   {proc_lib,init_p_do_apply,3,
                                    [{file,"proc_lib.erl"},{line,239}]}]}}}]},
                             {links,[<0.85.0>,<0.26398.73>]},
                             {dictionary,[]},
                             {trap_exit,false},
                             {status,runnable},
                             {heap_size,233},
                             {stack_size,20},
                             {reductions,28}]}]]}}
[Thu, 20 Aug 2015 13:59:47 GMT] [error] [<0.85.0>] ** Generic server couch_server terminating 
** Last message in was {'EXIT',<0.25603.73>,
                           {{badmatch,{error,eacces}},
                            [{couch_file,init,1,
                                 [{file,"couch_file.erl"},{line,314}]},
                             {gen_server,init_it,6,
                                 [{file,"gen_server.erl"},{line,328}]},
                             {proc_lib,init_p_do_apply,3,
                                 [{file,"proc_lib.erl"},{line,239}]}]}}
** When Server state == {server,"/var/db/couchdb",
                            {re_pattern,0,0,0,
                                <<69,82,67,80,140,0,0,0,16,0,0,0,1,0,0,0,255,
                                  255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,
                                  0,0,64,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
                                  0,0,0,0,0,0,0,0,0,0,0,125,0,72,25,106,0,0,0,
                                  0,0,0,0,0,0,0,0,0,254,255,255,7,0,0,0,0,0,0,
                                  0,0,0,0,0,0,0,0,0,0,106,0,0,0,0,16,171,255,
                                  3,0,0,0,128,254,255,255,7,0,0,0,0,0,0,0,0,0,
                                  0,0,0,0,0,0,0,98,27,114,0,72,0>>},
                            100,99,"Wed, 19 Aug 2015 12:57:47 GMT"}
** Reason for termination == 
** {{case_clause,[]},
    [{couch_server,handle_info,2,[{file,"couch_server.erl"},{line,435}]},
     {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},
     {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

As I don't have a clue about erlang I do not understand a word - what could these errors mean and what could have produced them?

Thank you very much, Sebastian

Answer 1

The error tells you that you don't have access to the file. It might be either that the file doesn't exist or that you don't have access to some subdirectories with the user id that couchdb uses.

You could try to switch to the couchdb user with su and then touch the databases files to check if you really have access to the file to this point.

Answer 2

The only relevant error is {badmatch,{error,eacces}, which states that a POSIX EACCES error has occured. The rest of the output is the erlang way of showing the "state" around the crash, a needed thing in a parallel runtime.

According to the erlang docs (http://www.erlang.org/doc/man/file.html), the error eacces can also mean: "Missing permission for reading the file, or for searching one of the parent directories."

Did you also check for the correct parent directory permissions? Is something changing the filesystem? Is /var/db a NFS mount or something?