Eliott Eccidio
Reputation: 11
(WebRTC) Heygen's Avatar Application : Ice Connection Status stays in "checking"
I am building an application implying an LLM and Heygen's API. I want to send the response given by the LLM to a streaming avatar.
I used the documentation provided here: https://docs.heygen.com/reference/repeat-text-copy
Heygen's streaming Avatar is a WebRTC (peer2peer between a server and an app) workflow using ICE to provide the data with an optimise channel of communication.
The sessions are updated through 4 steps: Creation, Starting, Handle the ICE, Close.
I don't know why but when I create and I try to start my session. The Ice Status stays in "checking" and doesn't turn into "connected" as if it didn't find a suited path.
Here is the code that I was using :
Main app.py :
async def main(peerConnection,sessionInfo):
chat = def_chat_chain()
console.print("[cyan]Assistant started! Press Ctrl+C to exit.")
try:
while True:
peerConnection, sessionInfo = await createNewSession(peerConnection,sessionInfo,'medium')
peerConnection_Started, sessionInfo = await startAndDisplaySession(peerConnection,sessionInfo)
# Check if the session is correctly initiated
console.print("[blue]Checking session state...")
session_ready = await check_ice_connection(peerConnection_Started)
....
# Rest not needed
Code snippets used (heygen.py):
#########################################
# WebRTC HOOKS #
#########################################
def print_peer_connection_info(peer_connection):
print("RTCPeerConnection State:")
print(f" Connection State: {peer_connection.connectionState}")
print(f" Signaling State: {peer_connection.signalingState}")
print(f" ICE Gathering State: {peer_connection.iceGatheringState}")
print(f" ICE Connection State: {peer_connection.iceConnectionState}")
async def createNewSession(peerConnection, sessionInfo, quality , avatar = None ,voice = None ):
code = 10007
count =0
while(code == 10007):
print("struggling")
count+=1
sessionInfo = await new_session(quality, AvatarID, VoiceID)
code = sessionInfo['code']
if count == 200 :
return
serverSdp = sessionInfo['data']['sdp']
iceServers = sessionInfo['data']['ice_servers2']
iceServers2 = [DotAccessibleObject(json.dumps(obj)) for obj in iceServers]
# Create a new RTCPeerConnection
config = RTCConfiguration(iceServers=iceServers2)
peerConnection = RTCPeerConnection(configuration=config)
async def on_track(event):
print('Received the track')
frame = await event.recv()
print("frame recu")
# Attach the on_track handler to the peerConnection
peerConnection.on("track", on_track)
# Set the remote description as the offer received from the client
remoteDesc = RTCSessionDescription(serverSdp['sdp'],type=serverSdp['type'])
await peerConnection.setRemoteDescription(remoteDesc)
return peerConnection, sessionInfo
async def startAndDisplaySession(peerConnection, sessionInfo):
if not sessionInfo:
logging.info('Please create a connection first')
return
# Create an answer to the offer
localDescription = await peerConnection.createAnswer()
await peerConnection.setLocalDescription(localDescription)
localDescriptionJson = {
"sdp": localDescription.sdp,
"type": localDescription.type
}
#localDescriptionJson = json.dumps(localDescriptionJson) # Convert to JSON
async def on_ice_candidate(event):
print("on_ice_candidate called")
if event.candidate:
print('Received ICE candidate:', event.candidate)
candidateJson = event.candidate.to_json()
await handle_ice(sessionInfo['data']['session_id'], candidateJson)
else:
print("No more ICE candidates.")
peerConnection.on('icecandidate', on_ice_candidate)
def on_ice_connection_state_change():
new_state = peerConnection.iceConnectionState
print(f"ICE connection state changed to: {new_state}")
peerConnection.on('iceconnectionstatechange', on_ice_connection_state_change)
await start_session(sessionInfo['data']['session_id'], localDescriptionJson)
print_peer_connection_info(peerConnection)
print ("---------------------")
print ("Session started successfully")
logging.info('Session started successfully')
return peerConnection, sessionInfo
These functions are using :
#########################################
# API HEYGEN #
#########################################
# Constants
SERVER_URL = 'https://api.heygen.com'
# Headers for all requests
headers = {
"accept": "application/json",
"content-type": "application/json",
"x-api-key": API_KEY
}
async def new_session(quality, avatar_name, voice_id):
"""
Create a new streaming session.
"""
headers = {
"accept": "application/json",
"content-type": "application/json",
"x-api-key": API_KEY
}
url = f"{SERVER_URL}/v1/streaming.new"
payload = {
"quality": "high",
"avatar_name": avatar_name,
"voice": {"voice_id": voice_id}
}
response = requests.post(url, json=payload, headers=headers)
if response.status_code == 500:
raise Exception("Server error during new session creation")
return response.json()
async def start_session(session_id, sdp):
"""
Start the streaming session.
"""
url = f"{SERVER_URL}/v1/streaming.start"
payload = {"session_id": session_id, "sdp": sdp}
headers = {
"accept": "application/json",
"Content-Type": "application/json",
"X-Api-Key": API_KEY
}
response = requests.post(url, json=payload, headers=headers)
if response.status_code == 500:
raise Exception("Server error during session start")
print("\n\n ------------------ After Start Session --------------- \n\n")
print (payload)
print(response.json())
print("\n-----------------------\n")
return response.json()
async def handle_ice(session_id, candidate):
"""
Handle ICE candidate submission.
"""
url = f"{SERVER_URL}/v1/streaming.ice"
payload = {"session_id": session_id, "candidate": candidate}
response = requests.post(url, json=payload, headers=headers)
if response.status_code == 500:
raise Exception("Server error during ICE handling")
The output I am getting :
## Beginning of the Trace
Assistant started! Press Ctrl+C to exit.
RTCPeerConnection State:
Connection State: new
Signaling State: have-remote-offer
ICE Gathering State: new
ICE Connection State: new
Received the track
Received the track
INFO:aioice.turn:TURN allocation created ('18.195.48.253', 46416) (expires in 600 seconds)
(Stays for ten seconds)
------------------ After Start Session ---------------
### This error appears after the call of the start_session function :
{'code': 400, 'message': 'peer start failed,please check the answer'}
-----------------------
Session started successfully
RTCPeerConnection State:
Connection State: new
Signaling State: stable
ICE Gathering State: complete
ICE Connection State: new
INFO:root:Session started successfully
--------------------- PeerConnection after StartandDisplay -----------------
RTCPeerConnection State:
Connection State: new
Signaling State: stable
ICE Gathering State: complete
ICE Connection State: new
Checking session state...
INFO:root:Current ICE State: new
ICE connection state changed to: checking
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.12.38', 56830) -> ('192.168.91.23', 39101)) State.FROZEN -> State.WAITING
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.12.38', 56830) -> ('172.17.0.1', 50991)) State.FROZEN -> State.WAITING
INFO:aioice.ice:Connection(0) Check CandidatePair(('18.195.48.253', 46416) -> ('192.168.91.23', 39101)) State.FROZEN -> State.WAITING
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.12.38', 56830) -> ('192.168.91.23', 39101)) State.WAITING -> State.IN_PROGRESS
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.12.38', 56830) -> ('172.17.0.1', 50991)) State.WAITING -> State.IN_PROGRESS
INFO:aioice.ice:Connection(0) Check CandidatePair(('18.195.48.253', 46416) -> ('192.168.91.23', 39101)) State.WAITING -> State.IN_PROGRESS
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-35' coro=<TurnClientMixin.send_data() done, defined at /opt/homebrew/anaconda3/envs/sncf/lib/python3.10/site-packages/aioice/turn.py:271> exception=TransactionFailed(Message(message_method=Method.CHANNEL_BIND, message_class=Class.ERROR, transaction_id=b'\x95\xf8(\x11\xb9\xa7\xefV\x95V(p'))>
Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/sncf/lib/python3.10/site-packages/aioice/turn.py", line 290, in send_data
await self.channel_bind(channel, addr)
File "/opt/homebrew/anaconda3/envs/sncf/lib/python3.10/site-packages/aioice/turn.py", line 111, in channel_bind
await self.request_with_retry(request)
File "/opt/homebrew/anaconda3/envs/sncf/lib/python3.10/site-packages/aioice/turn.py", line 243, in request_with_retry
response, addr = await self.request(request)
File "/opt/homebrew/anaconda3/envs/sncf/lib/python3.10/site-packages/aioice/turn.py", line 230, in request
return await transaction.run()
File "/opt/homebrew/anaconda3/envs/sncf/lib/python3.10/site-packages/aioice/stun.py", line 302, in run
return await self.__future
aioice.stun.TransactionFailed: STUN transaction failed (403 - Forbidden IP)
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.12.38', 56830) -> ('18.119.113.6', 52576)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.12.38', 56830) -> ('18.119.113.6', 34712)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.12.38', 56830) -> ('18.119.113.6', 43161)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.12.38', 56830) -> ('18.119.113.6', 60432)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.12.38', 56830) -> ('18.119.113.6', 49343)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.ice:Connection(0) Check CandidatePair(('18.195.48.253', 46416) -> ('172.17.0.1', 50991)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.turn:TURN channel bound 16385 ('172.17.0.1', 50991)
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.12.38', 56830) -> ('34.203.251.179', 43096)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.12.38', 56830) -> ('34.203.251.179', 41561)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.ice:Connection(0) Check CandidatePair(('192.168.12.38', 56830) -> ('34.203.251.179', 39500)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.ice:Connection(0) Check CandidatePair(('18.195.48.253', 46416) -> ('18.119.113.6', 52576)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.turn:TURN channel bound 16386 ('18.119.113.6', 52576)
INFO:aioice.ice:Connection(0) Check CandidatePair(('18.195.48.253', 46416) -> ('18.119.113.6', 34712)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.turn:TURN channel bound 16387 ('18.119.113.6', 34712)
INFO:aioice.ice:Connection(0) Check CandidatePair(('18.195.48.253', 46416) -> ('18.119.113.6', 43161)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.turn:TURN channel bound 16388 ('18.119.113.6', 43161)
INFO:aioice.ice:Connection(0) Check CandidatePair(('18.195.48.253', 46416) -> ('18.119.113.6', 60432)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.turn:TURN channel bound 16389 ('18.119.113.6', 60432)
INFO:aioice.ice:Connection(0) Check CandidatePair(('18.195.48.253', 46416) -> ('18.119.113.6', 49343)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.turn:TURN channel bound 16390 ('18.119.113.6', 49343)
INFO:aioice.ice:Connection(0) Check CandidatePair(('18.195.48.253', 46416) -> ('34.203.251.179', 43096)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.turn:TURN channel bound 16391 ('34.203.251.179', 43096)
INFO:aioice.ice:Connection(0) Check CandidatePair(('18.195.48.253', 46416) -> ('34.203.251.179', 41561)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.turn:TURN channel bound 16392 ('34.203.251.179', 41561)
INFO:aioice.ice:Connection(0) Check CandidatePair(('18.195.48.253', 46416) -> ('34.203.251.179', 39500)) State.FROZEN -> State.IN_PROGRESS
INFO:aioice.turn:TURN channel bound 16393 ('34.203.251.179', 39500)
INFO:root:Current ICE State: checking
INFO:root:Current ICE State: checking
INFO:root:Current ICE State: checking
INFO:root:Current ICE State: checking
End of Trace
I think it might be because the on_ice_candidate is never being called but I don't know. I am not very knowledgeable on this subject.
If you have any clue on why the ICE state stays in checking and not connected it would be very helpful !
Let me know if you need any more information.
Many thanks in advance !
Upvotes: 1
Views: 348