Very high Jitter Buffer Delay When using Opus Encoder with DTX Enabled

Question

I am working on developing an audio conference call server which uses webrtc to send and receive audio packets.

The server stores encoded packets sent by the browser,decodes them and mixes them to generate the output raw frames.

I use opus encoder with DTX enabled to encode the raw frames. During complete silence,the encoder produces a background noise filled packet every 400 ms which i send to the client.

But the jitterBufferDelay shown in webrtc for the packets i send is very high . Why is this happening and how do i reduce it? Webrtc Stats

When i went through the webrtc source code,they are sending the first dtx packet just as a principle to let the decoder know that the encoder has entered dtx mode . So i did the same but the results doesnt vary

Encoder Options that i use:

opus_encoder_ctl(encoder,OPUS_SET_BITRATE((opus_int32)50000));

opus_encoder_ctl(encoder,OPUS_SET_INBAND_FEC(1));

opus_encoder_ctl(encoder,OPUS_SET_PACKET_LOSS_PERC(20));

opus_encoder_ctl(encoder,OPUS_SET_DTX(1));