channel Got error &{FORBIDDEN} while joining new peer to a channel

Question

I created my network with a script like:

docker-compose -f $COMPOSE_FILE up -d $CA 
docker-compose -f $COMPOSE_FILE up -d $ORDERER1 $PEER0 $PEER1
docker-compose -f $COMPOSE_FILE up -d $CLI
docker exec cli peer channel create -o orderer.example.com:7050 -c $CHANNEL_NAME -f /etc/hyperledger/config/channel.tx

The channel is created and if I enter the cli, inside the working directory, I can find with ls the new generated file beerchannel.block. In this directory I also have crypto, which contains genesis block and other config files, and crypto-config that contains msp and certificates. At this point containers logs seems good.

Now I want to join peer0 to the channel with:

docker exec -e $ENV_ADDRESSP0 $CLI peer channel join -b $CHANNEL_NAME.block

As soon as I do this command, I cannot join peer0 to the channel. The strange thing is that running the script return this message:

2019-11-22 10:04:00.868 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2019-11-22 10:04:00.922 UTC [channelCmd] executeJoin -> INFO 002 Successfully submitted proposal to join channel

So, everything seems fine. But when I enter the logs of orderer, I get this message repeated:

2019-11-22 09:59:07.429 UTC [fsblkstorage] newBlockfileMgr -> INFO 009 Getting block information from block storage
2019-11-22 09:59:07.438 UTC [orderer.commmon.multichannel] newChain -> INFO 00a Created and starting new chain beerchannel
2019-11-22 09:59:07.440 UTC [comm.grpc.server] 1 -> INFO 00b streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=172.29.0.6:41778 grpc.code=OK grpc.call_duration=25.385144ms
2019-11-22 10:04:06.923 UTC [common.deliver] deliverBlocks -> WARN 00c [channel: beerchannel] Client authorization revoked for deliver request from 172.29.0.4:48406: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Readers' sub-policies to be satisfied: permission denied
2019-11-22 10:04:06.923 UTC [comm.grpc.server] 1 -> INFO 00d streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=172.29.0.4:48406 grpc.code=OK grpc.call_duration=1.001442ms
2019-11-22 10:04:07.026 UTC [common.deliver] deliverBlocks -> WARN 00e [channel: beerchannel] Client authorization revoked for deliver request from 172.29.0.4:48408: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Readers' sub-policies to be satisfied: permission denied
2019-11-22 10:04:07.026 UTC [comm.grpc.server] 1 -> INFO 00f streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=172.29.0.4:48408 grpc.code=OK grpc.call_duration=582.912µs

Since the peer involved is peer0, I also entered in peer0 logs and found this errors:

2019-11-22 10:04:00.870 UTC [endorser] callChaincode -> INFO 029 [][ec4f5097] Entry chaincode: name:"cscc" 
2019-11-22 10:04:00.870 UTC [ledgermgmt] CreateLedger -> INFO 02a Creating ledger [beerchannel] with genesis block
2019-11-22 10:04:00.874 UTC [fsblkstorage] newBlockfileMgr -> INFO 02b Getting block information from block storage
2019-11-22 10:04:00.896 UTC [kvledger] CommitWithPvtData -> INFO 02c [beerchannel] Committed block [0] with 1 transaction(s) in 16ms (state_validation=0ms block_and_pvtdata_commit=10ms state_commit=2ms) commitHash=[]
2019-11-22 10:04:00.899 UTC [ledgermgmt] CreateLedger -> INFO 02d Created ledger [beerchannel] with genesis block
2019-11-22 10:04:00.902 UTC [gossip.gossip] JoinChan -> INFO 02e Joining gossip network of channel beerchannel with 1 organizations
2019-11-22 10:04:00.902 UTC [gossip.gossip] learnAnchorPeers -> INFO 02f No configured anchor peers of Org1MSP for channel beerchannel to learn about
2019-11-22 10:04:00.917 UTC [gossip.state] NewGossipStateProvider -> INFO 030 Updating metadata information, current ledger sequence is at = 0, next expected block is = 1
2019-11-22 10:04:00.919 UTC [sccapi] deploySysCC -> INFO 031 system chaincode lscc/beerchannel(github.com/hyperledger/fabric/core/scc/lscc) deployed
2019-11-22 10:04:00.919 UTC [cscc] Init -> INFO 032 Init CSCC
2019-11-22 10:04:00.920 UTC [sccapi] deploySysCC -> INFO 033 system chaincode cscc/beerchannel(github.com/hyperledger/fabric/core/scc/cscc) deployed
2019-11-22 10:04:00.920 UTC [qscc] Init -> INFO 034 Init QSCC
2019-11-22 10:04:00.920 UTC [sccapi] deploySysCC -> INFO 035 system chaincode qscc/beerchannel(github.com/hyperledger/fabric/core/scc/qscc) deployed
2019-11-22 10:04:00.920 UTC [sccapi] deploySysCC -> INFO 036 system chaincode (+lifecycle,github.com/hyperledger/fabric/core/chaincode/lifecycle) disabled
2019-11-22 10:04:00.921 UTC [endorser] callChaincode -> INFO 037 [][ec4f5097] Exit chaincode: name:"cscc"  (51ms)
2019-11-22 10:04:00.921 UTC [comm.grpc.server] 1 -> INFO 038 unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.29.0.6:42736 grpc.code=OK grpc.call_duration=51.473337ms
2019-11-22 10:04:06.919 UTC [gossip.election] beLeader -> INFO 039 42a5181dbddcff9d15ae32b05300e849fbcad1cf138e62f3d8b726d7b5db25d3 : Becoming a leader
2019-11-22 10:04:06.919 UTC [gossip.service] func1 -> INFO 03a Elected as a leader, starting delivery service for channel beerchannel
2019-11-22 10:04:06.923 UTC [blocksProvider] DeliverBlocks -> ERRO 03b [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:04:07.026 UTC [blocksProvider] DeliverBlocks -> ERRO 03c [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:04:07.239 UTC [blocksProvider] DeliverBlocks -> ERRO 03d [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:04:07.643 UTC [blocksProvider] DeliverBlocks -> ERRO 03e [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:04:08.445 UTC [blocksProvider] DeliverBlocks -> ERRO 03f [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:04:10.051 UTC [blocksProvider] DeliverBlocks -> ERRO 040 [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:04:13.254 UTC [blocksProvider] DeliverBlocks -> ERRO 041 [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:04:19.657 UTC [blocksProvider] DeliverBlocks -> ERRO 042 [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:04:29.662 UTC [blocksProvider] DeliverBlocks -> ERRO 043 [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:04:39.668 UTC [blocksProvider] DeliverBlocks -> ERRO 044 [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:04:49.671 UTC [blocksProvider] DeliverBlocks -> ERRO 045 [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:04:49.671 UTC [blocksProvider] DeliverBlocks -> ERRO 046 [beerchannel] Wrong statuses threshold passed, stopping block provider
2019-11-22 10:04:49.671 UTC [gossip.election] stopBeingLeader -> INFO 047 42a5181dbddcff9d15ae32b05300e849fbcad1cf138e62f3d8b726d7b5db25d3 Stopped being a leader
2019-11-22 10:04:49.671 UTC [gossip.service] func1 -> INFO 048 Renounced leadership, stopping delivery service for channel beerchannel
2019-11-22 10:05:56.924 UTC [gossip.election] beLeader -> INFO 049 42a5181dbddcff9d15ae32b05300e849fbcad1cf138e62f3d8b726d7b5db25d3 : Becoming a leader
2019-11-22 10:05:56.924 UTC [gossip.service] func1 -> INFO 04a Elected as a leader, starting delivery service for channel beerchannel
2019-11-22 10:05:56.929 UTC [blocksProvider] DeliverBlocks -> ERRO 04b [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:05:57.032 UTC [blocksProvider] DeliverBlocks -> ERRO 04c [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:05:57.235 UTC [blocksProvider] DeliverBlocks -> ERRO 04d [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:05:57.638 UTC [blocksProvider] DeliverBlocks -> ERRO 04e [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:05:58.441 UTC [blocksProvider] DeliverBlocks -> ERRO 04f [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:06:00.044 UTC [blocksProvider] DeliverBlocks -> ERRO 050 [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:06:03.247 UTC [blocksProvider] DeliverBlocks -> ERRO 051 [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:06:09.652 UTC [blocksProvider] DeliverBlocks -> ERRO 052 [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:06:19.656 UTC [blocksProvider] DeliverBlocks -> ERRO 053 [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:06:29.659 UTC [blocksProvider] DeliverBlocks -> ERRO 054 [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:06:39.662 UTC [blocksProvider] DeliverBlocks -> ERRO 055 [beerchannel] Got error &{FORBIDDEN}
2019-11-22 10:06:39.662 UTC [blocksProvider] DeliverBlocks -> ERRO 056 [beerchannel] Wrong statuses threshold passed, stopping block provider
2019-11-22 10:06:39.662 UTC [gossip.election] stopBeingLeader -> INFO 057 42a5181dbddcff9d15ae32b05300e849fbcad1cf138e62f3d8b726d7b5db25d3 Stopped being a leader
2019-11-22 10:06:39.662 UTC [gossip.service] func1 -> INFO 058 Renounced leadership, stopping delivery service for channel beerchannel

It seems something related to permissions but I cannot understand what's wrong in here. The cli contains the beerchannel.block file, successfully generated it and now I just want to add peer0 to the channel.

---

Adding configtx.yaml

Organizations:

    - &OrdererOrg
        Name: OrdererOrg
        ID: OrdererMSP
        MSPDir: crypto-config/ordererOrganizations/c.com/msp

        Policies:
            Readers:
                Type: Signature
                Rule: "OR('OrdererMSP.member')"
            Writers:
                Type: Signature
                Rule: "OR('OrdererMSP.member')"
            Admins:
                Type: Signature
                Rule: "OR('OrdererMSP.admin')"

    - &s
        Name: sMSP
        ID: sMSP
        MSPDir: crypto-config/peerOrganizations/s.c.com/msp

        Policies:
            Readers:
                Type: Signature
                Rule: "OR('sMSP.admin', 'sMSP.peer', 'sMSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('sMSP.admin', 'sMSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('sMSP.admin')"

        AnchorPeers:
            - Host: peer1.s.c.com
              Port: 7051
            - Host: peer2.s.c.com
              Port: 8051

Capabilities:
    Channel: &ChannelCapabilities
        V1_4_3: true
        V1_3: false
        V1_1: false

    Orderer: &OrdererCapabilities
        V1_4_2: true
        V1_1: false

    Application: &ApplicationCapabilities
        V1_4_2: true
        V1_3: false
        V1_2: false
        V1_1: false


Application: &ApplicationDefaults
    Organizations:

    Policies:
        Readers:
            Type: ImplicitMeta
            Rule: "ANY Readers"
        Writers:
            Type: ImplicitMeta
            Rule: "ANY Writers"
        Admins:
            Type: ImplicitMeta
            Rule: "MAJORITY Admins"

    Capabilities:
        <<: *ApplicationCapabilities


Orderer: &OrdererDefaults
    OrdererType: solo

    Addresses:
        - orderer1.c.com:7050

    BatchTimeout: 500ms
    BatchSize:
        MaxMessageCount: 15
        AbsoluteMaxBytes: 99 MB
        PreferredMaxBytes: 512 kb

    Policies:
        Readers:
            Type: ImplicitMeta
            Rule: "ANY Readers"
        Writers:
            Type: ImplicitMeta
            Rule: "ANY Writers"
        Admins:
            Type: ImplicitMeta
            Rule: "MAJORITY Admins"
        BlockValidation:
            Type: ImplicitMeta
            Rule: "ANY Writers"

    Capabilities:
            <<: *OrdererCapabilities

Channel: &ChannelDefaults
    Policies:
        Readers:
            Type: ImplicitMeta
            Rule: "ANY Readers"
        Writers:
            Type: ImplicitMeta
            Rule: "ANY Writers"
        Admins:
            Type: ImplicitMeta
            Rule: "MAJORITY Admins"

    Capabilities:
        <<: *ChannelCapabilities



Profiles:

    OneOrgOrdererGenesis:
        <<: *ChannelDefaults
        Orderer:
            <<: *OrdererDefaults
            Organizations:
                - *OrdererOrg
        Consortiums:
            SampleConsortium:
                Organizations:
                    - *s
    OneOrgChannel:
        <<: *ChannelDefaults
        Consortium: SampleConsortium
        Application:
            <<: *ApplicationDefaults
            Organizations:
                - *s

    SampleMultiNodeEtcdRaft:
        <<: *ChannelDefaults
        Capabilities:
            <<: *ChannelCapabilities
        Orderer:
            <<: *OrdererDefaults
            OrdererType: etcdraft
            EtcdRaft:
                Consenters:
                - Host: orderer1.c.com
                  Port: 7050
                  ClientTLSCert: crypto-config/ordererOrganizations/c.com/orderers/orderer1.c.com/tls/server.crt
                  ServerTLSCert: crypto-config/ordererOrganizations/c.com/orderers/orderer1.c.com/tls/server.crt
                - Host: orderer2.c.com
                  Port: 7050
                  ClientTLSCert: crypto-config/ordererOrganizations/c.com/orderers/orderer2.c.com/tls/server.crt
                  ServerTLSCert: crypto-config/ordererOrganizations/c.com/orderers/orderer2.c.com/tls/server.crt
                - Host: orderer3.c.com
                  Port: 7050
                  ClientTLSCert: crypto-config/ordererOrganizations/c.com/orderers/orderer3.c.com/tls/server.crt
                  ServerTLSCert: crypto-config/ordererOrganizations/c.com/orderers/orderer3.c.com/tls/server.crt
            Addresses:
            - orderer1.c.com:7050
            - orderer2.c.com:7050
            - orderer3.c.com:7050

            Organizations:
            - *OrdererOrg
            Capabilities:
                <<: *OrdererCapabilities
        Application:
            <<: *ApplicationDefaults
            Organizations:
            - <<: *OrdererOrg
        Consortiums:
            SampleConsortium:
                Organizations:
                - *s

Answer 1

This migh be caused by a bad msp folder or bad configuration.

first check if the used msp represents the organisation that has the right to read the configuration/block. Then, check if CORE_PEER_MSPCONFIGPATH is set correctly to your msp example:

export CORE_PEER_MSPCONFIGPATH=${PWD}/organizations/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp

Answer 2

Check for the Reader policies that you have defined in your configtx.yaml this error is generated because of the policy mismatch. You have defined some specific user type(admin, peer, client) in your Reader policies but this specific user type is not passed into certificates that you have generated for your peer.

Edited:

If you want to make it generic and not specific to the identity type then you can edit the s org policies like this:

  - &s
    Name: sMSP
    ID: sMSP
    MSPDir: crypto-config/peerOrganizations/s.c.com/msp

    Policies:
        Readers:
            Type: Signature
            Rule: "OR('sMSP.member')"
        Writers:
            Type: Signature
            Rule: "OR('sMSP.member')"
        Admins:
            Type: Signature
            Rule: "OR('sMSP.admin')"

Answer 3

Check your crypto-config.yaml under peerOrgs section and add EnableNodeOUs property if missing then regenerate the crypto materials. Config example:

PeerOrgs:
 - Name: Org1
   Domain: org1.example.com
   EnableNodeOUs: true