Hyperledger: Orderer exits when running setup with existing volumes due to bad certs

Question

I have one org which has single peer, orderer and CA running inside containers in same network and same VM with version 2.2.1.

I have persistent volumes which has production data. When I spin up the orderer containers with these volumes, I get TLS handshake failed with error remote error: tls: bad certificate server:Orderer error.

I am able to get these containers up and running: ca_orderer, ca.org1.myOrg.com, peer0.org1.myOrg.com, couchdb0.

Below are the scripts I used:

base.yaml

version: "2"
services:
  peer-base:
    image: hyperledger/fabric-peer:${FABRIC_VERSION}
    environment:
      - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
      - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=artifacts_default
      - CORE_LOGGING_LEVEL=INFO
      - CORE_PEER_GOSSIP_USELEADERELECTION=true
      - CORE_PEER_GOSSIP_ORGLEADER=false
      - CORE_PEER_GOSSIP_SKIPHANDSHAKE=true
      - CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/crypto/peer/msp
      - CORE_PEER_TLS_ENABLED=true
      - CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/crypto/peer/tls/server.key
      - CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/crypto/peer/tls/server.crt
      - CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/crypto/peer/tls/ca.crt
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric/peer
    command: peer node start
    volumes:
      - /var/run/:/host/var/run/

create-artifacts.sh

#!/bin/bash
source ../../.env

# Delete existing artifacts
rm ../build/genesis.block ../build/srm-channel.tx


# System channel
SYS_CHANNEL=${SYS_CHANNEL}

# channel name defaults to "mychannel"
CHANNEL_NAME=${CHANNEL_NAME}

echo $CHANNEL_NAME

# Generate System Genesis block
configtxgen -profile OrdererGenesis -configPath ../config/ -channelID $SYS_CHANNEL  -outputBlock ../build/genesis.block


# Generate channel configuration block
configtxgen -profile SRMChannel -configPath ../config/ -outputCreateChannelTx ../build/$CHANNEL_NAME.tx -channelID $CHANNEL_NAME

echo "#######    Generating anchor peer update for Org1MSP  ##########"
configtxgen -profile SRMChannel -configPath ../config/ -outputAnchorPeersUpdate ../build/Org1MSPanchors.tx -channelID $CHANNEL_NAME -asOrg Org1MSP

./createChannel.sh:

#!/bin/bash
source ../../.env

export CORE_PEER_TLS_ENABLED=true
export ORDERER_CA=${PWD}/../build/crypto-config/ordererOrganizations/infinichains.com/orderers/orderer.infinichains.com/msp/tlscacerts/tlsca.infinichains.com-cert.pem
export PEER0_ORG1_CA=${PWD}/../build/crypto-config/peerOrganizations/org1.infinichains.com/peers/peer0.org1.infinichains.com/tls/ca.crt
export FABRIC_CFG_PATH=${PWD}/../config/

export CHANNEL_NAME=${CHANNEL_NAME}

export DISCOVERY_AS_LOCALHOST=true

setGlobalsForPeer0Org1(){
    export CORE_PEER_LOCALMSPID="Org1MSP"
    export CORE_PEER_TLS_ROOTCERT_FILE=$PEER0_ORG1_CA
    export CORE_PEER_MSPCONFIGPATH=${PWD}/../build/crypto-config/peerOrganizations/org1.infinichains.com/users/Admin@org1.infinichains.com/msp
    export CORE_PEER_ADDRESS=localhost:7051
}

createChannel(){
    rm -rf ../build/channel-artifacts/*
    setGlobalsForPeer0Org1
    
    peer channel create -o localhost:7050 -c $CHANNEL_NAME \
    --ordererTLSHostnameOverride orderer.infinichains.com \
    -f ../build/${CHANNEL_NAME}.tx --outputBlock ../build/${CHANNEL_NAME}.block \
    --tls $CORE_PEER_TLS_ENABLED --cafile $ORDERER_CA
}

joinChannel(){
    setGlobalsForPeer0Org1
    peer channel join -b ../build/$CHANNEL_NAME.block
}

updateAnchorPeers(){
    setGlobalsForPeer0Org1
    peer channel update -o localhost:7050 --ordererTLSHostnameOverride orderer.infinichains.com -c $CHANNEL_NAME -f ../build/${CORE_PEER_LOCALMSPID}anchors.tx --tls $CORE_PEER_TLS_ENABLED --cafile $ORDERER_CA
}

createChannel
joinChannel
updateAnchorPeers

After this, I can see the the peer is up. But the orderer gets killed with below logs:

2025-01-02 11:49:37.880 UTC [orderer.common.server] initializeServerConfig -> INFO 004 Starting orderer with TLS enabled
2025-01-02 11:49:38.103 UTC [orderer.common.server] Main -> INFO 005 Not bootstrapping the system channel because of existing channels
2025-01-02 11:49:38.203 UTC [orderer.common.server] selectClusterBootBlock -> INFO 006 Cluster boot block is bootstrap (genesis) block; Blocks Header.Number system-channel=0, bootstrap=0
2025-01-02 11:49:38.207 UTC [orderer.common.server] Main -> INFO 007 Starting with system channel: sys-channel, consensus type: etcdraft
2025-01-02 11:49:38.207 UTC [orderer.common.server] Main -> INFO 008 Setting up cluster
2025-01-02 11:49:38.210 UTC [orderer.common.server] reuseListener -> INFO 009 Cluster listener is not configured, defaulting to use the general listener on port 7050
2025-01-02 11:49:38.210 UTC [orderer.common.server] reuseListener -> INFO 00a Cluster listener is not configured, defaulting to use the general listener on port 7050
2025-01-02 11:49:38.214 UTC [orderer.common.cluster] loadVerifier -> INFO 00b Loaded verifier for channel srm-channel from config block at index 36044
2025-01-02 11:49:38.232 UTC [orderer.common.cluster] loadVerifier -> INFO 00c Loaded verifier for channel sys-channel from config block at index 1
2025-01-02 11:49:38.233 UTC [certmonitor] trackCertExpiration -> INFO 00d The enrollment certificate will expire on 2026-01-02 11:48:00 +0000 UTC
2025-01-02 11:49:38.233 UTC [certmonitor] trackCertExpiration -> INFO 00e The server TLS certificate will expire on 2026-01-02 11:48:00 +0000 UTC
2025-01-02 11:49:38.233 UTC [certmonitor] trackCertExpiration -> INFO 00f The client TLS certificate will expire on 2026-01-02 11:48:00 +0000 UTC
2025-01-02 11:49:38.243 UTC [orderer.consensus.etcdraft] detectSelfID -> WARN 010 Could not find -----BEGIN CERTIFICATE-----
MIIC2jCCAoCgAwIBAgIUQptHUDXWDxoxbE5Bcajx8QNvCX8wCgYIKoZIzj0EAwIw......
...
-----END CERTIFICATE-----
 among [-----BEGIN CERTIFICATE-----
MIIC2jCCAoCgAwIBAgIURVf1KXLywtuEYtNtgrxU8SVdZO4wCgYIKoZIzj0EAwIw......
...
-----END CERTIFICATE-----
]
2025-01-02 11:49:38.243 UTC [orderer.common.onboarding] TrackChain -> INFO 011 Adding srm-channel to the set of chains to track
2025-01-02 11:49:38.247 UTC [orderer.consensus.etcdraft] detectSelfID -> WARN 012 Could not find -----BEGIN CERTIFICATE-----
MIIC2jCCAoCgAwIBAgIUQptHUDXWDxoxbE5Bcajx8QNvCX8wCgYIKoZIzj0EAwIw......
...
-----END CERTIFICATE-----
 among [-----BEGIN CERTIFICATE-----
MIIC2jCCAoCgAwIBAgIURVf1KXLywtuEYtNtgrxU8SVdZO4wCgYIKoZIzj0EAwIw......
...
-----END CERTIFICATE-----
]
2025-01-02 11:49:38.247 UTC [orderer.common.onboarding] TrackChain -> INFO 013 Adding sys-channel to the set of chains to track
2025-01-02 11:49:38.247 UTC [orderer.commmon.multichannel] Initialize -> INFO 014 Starting system channel 'sys-channel' with genesis block hash a223d58016d149cdf6257cedbe8ad4c42cf01ea93f2b2cd70844d3fc7348c2f3 and orderer type etcdraft
2025-01-02 11:49:38.247 UTC [orderer.common.server] Main -> INFO 015 Starting orderer:
 Version: 2.2.1
 Commit SHA: 344fda6
 Go version: go1.14.4
 OS/Arch: linux/amd64
2025-01-02 11:49:38.247 UTC [orderer.common.server] Main -> INFO 016 Beginning to serve requests
2025-01-02 11:49:48.236 UTC [orderer.common.onboarding] replicateDisabledChains -> INFO 017 Found 2 inactive chains: [sys-channel srm-channel]
2025-01-02 11:49:48.237 UTC [orderer.common.cluster] ReplicateChains -> INFO 018 Will now replicate chains [sys-channel srm-channel]
2025-01-02 11:49:48.239 UTC [orderer.common.cluster] discoverChannels -> INFO 019 Discovered 2 channels: [sys-channel srm-channel]
2025-01-02 11:49:48.239 UTC [orderer.common.cluster] channelsToPull -> INFO 01a Evaluating channels to pull: [sys-channel srm-channel]
2025-01-02 11:49:48.239 UTC [orderer.common.cluster] channelsToPull -> INFO 01b Probing whether I should pull channel sys-channel
2025-01-02 11:49:48.242 UTC [core.comm] ServerHandshake -> ERRO 01c TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=192.168.160.5:58282
2025-01-02 11:49:49.244 UTC [core.comm] ServerHandshake -> ERRO 01d TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=192.168.160.5:58286
2025-01-02 11:49:50.772 UTC [core.comm] ServerHandshake -> ERRO 01e TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=192.168.160.5:58290
2025-01-02 11:49:53.818 UTC [core.comm] ServerHandshake -> ERRO 01f TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=192.168.160.5:58302
2025-01-02 11:49:55.240 UTC [orderer.common.cluster.replication] probeEndpoint -> WARN 020 Failed connecting to {"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=fabric-ca-server,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer.infinichains.com:7050"}: failed to create new connection: context deadline exceeded channel=sys-channel
2025-01-02 11:49:55.241 UTC [orderer.common.cluster.replication] func1 -> WARN 021 Received error of type 'failed to create new connection: context deadline exceeded' from {"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=fabric-ca-server,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer.infinichains.com:7050"} channel=sys-channel
2025-01-02 11:49:55.241 UTC [orderer.common.cluster.replication] HeightsByEndpoints -> INFO 022 Returning the heights of OSNs mapped by endpoints map[] channel=sys-channel
2025-01-02 11:49:55.242 UTC [orderer.common.cluster] channelsToPull -> WARN 023 Could not obtain blocks needed for classifying whether I am in the channel,skipping the retrieval of the chan sys-channel
2025-01-02 11:49:55.242 UTC [orderer.common.cluster] channelsToPull -> INFO 024 Probing whether I should pull channel srm-channel
2025-01-02 11:49:55.245 UTC [core.comm] ServerHandshake -> ERRO 025 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=192.168.160.5:58306
2025-01-02 11:49:56.247 UTC [core.comm] ServerHandshake -> ERRO 026 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=192.168.160.5:60890
2025-01-02 11:49:57.842 UTC [core.comm] ServerHandshake -> ERRO 027 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=192.168.160.5:60894
2025-01-02 11:50:00.164 UTC [core.comm] ServerHandshake -> ERRO 028 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=192.168.160.5:60908
2025-01-02 11:50:02.243 UTC [orderer.common.cluster.replication] probeEndpoint -> WARN 029 Failed connecting to {"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=fabric-ca-server,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer.infinichains.com:7050"}: failed to create new connection: context deadline exceeded channel=sys-channel
2025-01-02 11:50:02.244 UTC [orderer.common.cluster.replication] func1 -> WARN 02a Received error of type 'failed to create new connection: context deadline exceeded' from {"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=fabric-ca-server,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer.infinichains.com:7050"} channel=sys-channel
2025-01-02 11:50:02.244 UTC [orderer.common.cluster.replication] HeightsByEndpoints -> INFO 02b Returning the heights of OSNs mapped by endpoints map[] channel=sys-channel
2025-01-02 11:50:02.244 UTC [orderer.common.cluster] channelsToPull -> WARN 02c Could not obtain blocks needed for classifying whether I am in the channel,skipping the retrieval of the chan srm-channel
2025-01-02 11:50:02.244 UTC [orderer.common.cluster] ReplicateChains -> INFO 02d Found myself in 0 channels out of 2 : {[] [{sys-channel 0xc0000f89c0} {srm-channel 0xc000145800}]}
2025-01-02 11:50:02.244 UTC [orderer.common.cluster] appendBlock -> INFO 02e Skipping commit of block [0] for channel sys-channel because height is at 2
2025-01-02 11:50:02.244 UTC [orderer.common.cluster] appendBlock -> INFO 02f Skipping commit of block [0] for channel srm-channel because height is at 36045
2025-01-02 11:50:02.244 UTC [orderer.common.cluster] PullChannel -> INFO 030 Pulling channel sys-channel
2025-01-02 11:50:02.246 UTC [core.comm] ServerHandshake -> ERRO 031 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=192.168.160.5:60920
2025-01-02 11:50:03.248 UTC [core.comm] ServerHandshake -> ERRO 032 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=192.168.160.5:60924
2025-01-02 11:50:05.071 UTC [core.comm] ServerHandshake -> ERRO 033 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=192.168.160.5:60928
2025-01-02 11:50:07.890 UTC [core.comm] ServerHandshake -> ERRO 034 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=192.168.160.5:47818
2025-01-02 11:50:09.245 UTC [orderer.common.cluster.replication] probeEndpoint -> WARN 035 Failed connecting to {"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=fabric-ca-server,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer.infinichains.com:7050"}: failed to create new connection: context deadline exceeded channel=sys-channel
2025-01-02 11:50:09.245 UTC [orderer.common.cluster.replication] func1 -> WARN 036 Received error of type 'failed to create new connection: context deadline exceeded' from {"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=fabric-ca-server,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer.infinichains.com:7050"} channel=sys-channel
2025-01-02 11:50:09.245 UTC [orderer.common.cluster.replication] HeightsByEndpoints -> INFO 037 Returning the heights of OSNs mapped by endpoints map[] channel=sys-channel
2025-01-02 11:50:09.245 UTC [orderer.common.cluster] ReplicateChains -> PANI 038 Failed pulling system channel: failed obtaining the latest block for channel sys-channel
panic: Failed pulling system channel: failed obtaining the latest block for channel sys-channel

goroutine 52 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000148bb0, 0x0, 0x0, 0x0)
    /go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:230 +0x545
go.uber.org/zap.(*SugaredLogger).log(0xc00012f458, 0xc0006dc504, 0x101fc6d, 0x21, 0xc00036dc40, 0x1, 0x1, 0x0, 0x0, 0x0)
    /go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:234 +0x100
go.uber.org/zap.(*SugaredLogger).Panicf(...)
    /go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:159
github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panicf(...)
    /go/src/github.com/hyperledger/fabric/common/flogging/zap.go:74
github.com/hyperledger/fabric/orderer/common/cluster.(*Replicator).ReplicateChains(0xc00034f8c0, 0xc0000f8e80, 0xc0004fcaf0, 0xc00034f8c0)
    /go/src/github.com/hyperledger/fabric/orderer/common/cluster/replication.go:166 +0x49d
github.com/hyperledger/fabric/orderer/common/onboarding.(*ReplicationInitiator).ReplicateChains(0xc0000ef000, 0xc0000f8e80, 0xc000012f40, 0x2, 0x2, 0x0, 0x0, 0x0)
    /go/src/github.com/hyperledger/fabric/orderer/common/onboarding/onboarding.go:185 +0x1e3
github.com/hyperledger/fabric/orderer/common/onboarding.(*InactiveChainReplicator).replicateDisabledChains(0xc0006f0cc0)
    /go/src/github.com/hyperledger/fabric/orderer/common/onboarding/onboarding.go:312 +0x225
github.com/hyperledger/fabric/orderer/common/onboarding.(*InactiveChainReplicator).Run(0xc0006f0cc0)
    /go/src/github.com/hyperledger/fabric/orderer/common/onboarding/onboarding.go:290 +0x42
created by github.com/hyperledger/fabric/orderer/common/server.initializeEtcdraftConsenter
    /go/src/github.com/hyperledger/fabric/orderer/common/server/main.go:777 +0x218

Thank you!