"Undefined symbols for architecture x86_64:" for library using cgo on macOS Sierra

Question

I am trying to make use of a library, https://github.com/go-steem/rpc, that makes use of some C code, which references a library.

The C library can be found here, https://github.com/bitcoin-core/secp256k1

I followed the steps to get that installed

$ ./autogen.sh
$ ./configure
$ make
$ ./tests
$ sudo make install  # optional

And have this output;

$ sudo make install
Password:
  CC       src/libsecp256k1_la-secp256k1.lo
  CCLD     libsecp256k1.la
  CC       src/tests-tests.o
  CCLD     tests
  CC       src/exhaustive_tests-tests_exhaustive.o
  CCLD     exhaustive_tests
 build-aux/install-sh -c -d '/usr/local/lib'
 /bin/sh ./libtool   --mode=install /usr/bin/install -c   libsecp256k1.la '/usr/local/lib'
libtool: install: /usr/bin/install -c .libs/libsecp256k1.0.dylib /usr/local/lib/libsecp256k1.0.dylib
libtool: install: (cd /usr/local/lib && { ln -s -f libsecp256k1.0.dylib libsecp256k1.dylib || { rm -f libsecp256k1.dylib && ln -s libsecp256k1.0.dylib libsecp256k1.dylib; }; })
libtool: install: /usr/bin/install -c .libs/libsecp256k1.lai /usr/local/lib/libsecp256k1.la
libtool: install: /usr/bin/install -c .libs/libsecp256k1.a /usr/local/lib/libsecp256k1.a
libtool: install: chmod 644 /usr/local/lib/libsecp256k1.a
libtool: install: /usr/bin/ranlib /usr/local/lib/libsecp256k1.a
 build-aux/install-sh -c -d '/usr/local/include'
 /usr/bin/install -c -m 644 include/secp256k1.h '/usr/local/include'
 build-aux/install-sh -c -d '/usr/local/lib/pkgconfig'
 /usr/bin/install -c -m 644 libsecp256k1.pc '/usr/local/lib/pkgconfig'

I try to run the upvote example from that Go library, go-steem/rpc/examples/upvote/ and get the following output;

$ go run main.go 
# github.com/go-steem/rpc/transactions
../../transactions/signing.c:5:10: fatal error: 'secp256k1.h' file not found

Already it feels as though the wheels are falling off...

Please bear with me as I do not develop in C, so I get a bit hack-y.

After much reading, and googling I decide to copy the files from the 'include' directory where I compiled libsecp256k1 into the same directory as the error is originating from.

You can see the files are not there;

$ ls -la ../../transactions/
total 48
drwxr-xr-x   8 shaunmorrow  staff   272 May  8 18:09 .
drwxr-xr-x  15 shaunmorrow  staff   510 May  8 18:09 ..
-rw-r--r--   1 shaunmorrow  staff   256 Apr 27 17:53 chains.go
-rw-r--r--   1 shaunmorrow  staff  3731 May  8 18:09 signed_transaction.go
-rw-r--r--   1 shaunmorrow  staff  1849 May  8 18:09 signed_transaction_test.go
-rw-r--r--   1 shaunmorrow  staff  3075 Apr 27 17:53 signing.c
-rw-r--r--   1 shaunmorrow  staff   408 Apr 27 17:53 signing.h
-rw-r--r--   1 shaunmorrow  staff  1049 May  8 18:09 transactions.go

and after the copy;

$ ls -la ../../transactions/
total 128
drwxr-xr-x  11 shaunmorrow  staff    374 Jul 18 19:08 .
drwxr-xr-x  15 shaunmorrow  staff    510 May  8 18:09 ..
-rw-r--r--   1 shaunmorrow  staff    256 Apr 27 17:53 chains.go
-rw-r--r--   1 shaunmorrow  staff  27071 Jul 18 19:08 secp256k1.h
-rw-r--r--   1 shaunmorrow  staff   1014 Jul 18 19:08 secp256k1_ecdh.h
-rw-r--r--   1 shaunmorrow  staff   4700 Jul 18 19:08 secp256k1_recovery.h
-rw-r--r--   1 shaunmorrow  staff   3731 Jul 18 19:05 signed_transaction.go
-rw-r--r--   1 shaunmorrow  staff   1849 May  8 18:09 signed_transaction_test.go
-rw-r--r--   1 shaunmorrow  staff   3075 Apr 27 17:53 signing.c
-rw-r--r--   1 shaunmorrow  staff    408 Apr 27 17:53 signing.h
-rw-r--r--   1 shaunmorrow  staff   1049 May  8 18:09 transactions.go

Now I get a new error;

$ go run main.go 
# github.com/go-steem/rpc/transactions
ld: library not found for -lsecp256k1
clang: error: linker command failed with exit code 1 (use -v to see invocation)

This has me reading and googling some more,

Finally I get even more hack-y and change transactions.go;

// #cgo LDFLAGS: -lsecp256k1
// #include <stdlib.h>
// #include "signing.h"
import "C"

becomes

// #cgo LDFLAGS: -L/usr/local/lib
// #include <stdlib.h>
// #include "signing.h"
import "C"

which fails, output on that later

I also try;

// #cgo LDFLAGS: -L/usr/local/lib -I/usr/local/include
// #include <stdlib.h>
// #include "signing.h"
import "C"

and copy the .h files into the /usr/local/include directory.

None of this works and now I am stuck with an error like this

$ go run main.go 
# github.com/go-steem/rpc/transactions
ld: library not found for -lsecp256k1
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ShaunsSePc-2:upvote shaunmorrow$ go run main.go 
# github.com/go-steem/rpc/transactions
Undefined symbols for architecture x86_64:
  "_secp256k1_context_create", referenced from:
      _sign_transaction in signing.o
      _verify_recoverable_signature in signing.o
  "_secp256k1_context_destroy", referenced from:
      _sign_transaction in signing.o
      _verify_recoverable_signature in signing.o
  "_secp256k1_ec_pubkey_serialize", referenced from:
      _verify_recoverable_signature in signing.o
  "_secp256k1_ecdsa_recover", referenced from:
      _verify_recoverable_signature in signing.o
  "_secp256k1_ecdsa_recoverable_signature_convert", referenced from:
      _verify_recoverable_signature in signing.o
  "_secp256k1_ecdsa_recoverable_signature_parse_compact", referenced from:
      _verify_recoverable_signature in signing.o
  "_secp256k1_ecdsa_recoverable_signature_serialize_compact", referenced from:
      _sign_transaction in signing.o
  "_secp256k1_ecdsa_sign_recoverable", referenced from:
      _sign_transaction in signing.o
  "_secp256k1_ecdsa_verify", referenced from:
      _verify_recoverable_signature in signing.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

At this point I really have no idea how to continue.

As you can see I am not experienced in C at all and have no idea how to test if the library libsecp256k1 is even installed properly!

This is where I ended up, but it's highly likely I took a wrong turn early in my journey, I would appreciate any help given as I have been struggling with this for a few nights already now :(

Not sure whats needed so here some env variables

$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/shaunmorrow/Work/go/"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/86/hlqptn5101z5bcydjz05qy8m0000gn/T/go-build689438019=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

Version is go version go1.8.3 darwin/amd64 with no old versions lying around.

Thanks!

Answer 1

Your problem is twofold:

1. Improperly configured libsecp256k1 package
2. Your C compiler not searching /usr/local directories for installed headers/libs

Fixing the improperly libsecp256k1 configured package

Your "undefined symbols" issue when it comes to linking C libraries sometimes points to an improperly configured package (in the sense of an Autotools package or a CMake package, not a Go package). Running ./configure --help, I see there is an option named --enable-module-recovery. Judging by names like _secp256k1_ecdsa_sign_recoverable and _secp256k1_ecdsa_recover, you need to add that option when configuring, meaning instead of executing the simpler ./configure, you should execute this:

./configure --enable-module-recovery

Is the C compiler broken?

Since the secp256k1.h header file isn't found in /usr/local/include, despite the fact that the header file most definitely exists after sudo make install is finished, it means your compiler doesn't search /usr/local.

Barring any fixes in the linked question, you can work around the issue by altering the source of the Go package as needed to add/modify the CFLAGS and LDFLAGS used when dealing with import "C" statements like this:

// #cgo CFLAGS: -I/usr/local/include
// #cgo LDFLAGS: -L/usr/local/lib -lsecp256k1
// #include <secp256k1.h>
import "C"

If you have pkg-config installed, you can use that instead of setting CFLAGS and LDFLAGS manually:

1. Export the PKG_CONFIG_PATH environment variable with a custom set of paths. Because no prefix (i.e. directory serving as the install root) was specified, /usr/local is assumed, meaning /usr/local/include will contain headers and /usr/local/lib will contain libraries. This means you need to export PKG_CONFIG_PATH on your command line as in export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/local/share/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig. The default setting of PKG_CONFIG_PATH when unset includes /usr/lib/pkgconfig and /usr/share/pkgconfig in that order on Linux and are specified as a fallback for other packages that might be used, though it may not be necessary in this case. The default set of paths may differ on OS X, so consult your pkg-config man page for reference.
2. Use // #cgo pkg-config: libsecp256k1, and CFLAGS and LDFLAGS will be set as necessary without you needing to do anything to them. Since not all systems have pkg-config installed, relying on this is perhaps a bad idea if you want a package to remain portable. Then again, I think it'd be preferable to the mess you dealt with since pkg-config simply wouldn't be found.
3. Change to the upvote directory and type make to build a working upvote binary.

Additional customization

Custom prefix

If you use something other than /usr/local as your prefix (e.g. ./configure --enable-module-recovery --prefix=/opt/libsecp256k1), then you'll need to adjust some things accordingly:

// #cgo CFLAGS: -I/opt/libsecp256k1/include
// #cgo LDFLAGS: -L/opt/libsecp256k1/lib -lsecp256k1
// #include "secp256k1.h"
import "C"

// or just use pkg-config and export the PKG_CONFIG_PATH environment
// variable containing the following paths:
//   /opt/libsecp256k1/lib/pkgconfig
//   /opt/libsecp256k1/share/pkgconfig
//   /usr/lib/pkgconfig
//   /usr/share/pkgconfig

You'll also need to modify the provided Makefile in the upvote directory to set the runtime path of the binary that gets built, else libsecp256k1.0.dylib will not be found:

# If you copy and paste this, replace the spaces in front of `go build`
# with a single horizontal tab character, else `make` will fail.
#
# Note that the "ldflags" specified are for the Go linker (go tool link),
# not the system's linker (ld).
build:
    go build -ldflags="-r /opt/libsecp256k1/lib"

For more information about working with cgo, check out the following resources:

• "C? Go? Cgo!" introduction to cgo on The Go Blog
• Cgo documentation