CODEWITHSUNDEEP
pythonsocketsirctwitch
Turtoise
Turtoise

Reputation: 27

Joining twitch irc through helix api, (5000+ channels) connection reset error

To start off im using an anonymous connection joining the channels which means there are no JOIN limits, I have tried different variations of sleeping, I started off just joining from a text however that had a lot of problems because it was connecting all the sockets before joining so I couldnt see what caused it. However this is the best version I have created so far, its pretty scuffed but I am just trying to understand what the issue is. If anyone has any insight on doing a big task like this I would appreciate it a lot!

(oauth and helix headers are from a random alt account I made for testing and its trying to join 10k channels in the example but stops around 2k-3k max)

import requests
import socket 
import time
import threading 
import random 

connections_made = 0
sockets = []


def connect():
    global sockets
    global connections_made
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    print("CONNECTING TO IRC")
    sock.connect(('irc.chat.twitch.tv', 6667))
    sock.send(bytes('PASS oauth:'+ '\r\n', 'utf-8'))
    sock.send(bytes('NICK justinfan' + str(random.randint(10000,99999)) + '\r\n', 'utf-8'))
    sockets.append(sock)
    connections_made += 1
    print(f"socket: {len(sockets)}")


for i in range(2):
    connect() # initial for .recv reading

helix_headers = {'client-id': 'q6batx0epp608isickayubi39itsckt', 'authorization': 'Bearer rk0ixn6169ar7y5xey9msvk1h8zrs8'}
def request(channels_to_join,cursor):
    
    request_amount = int(channels_to_join / 100)   # 100 requests = 10000 channels   

    user_list = []
    sock_numb = 0
    total_chans_joined = 0 
    count_every_request  = 0
    for i in range(request_amount):
        time.sleep(1)
        # 3k channels with time.sleep(1) 1.5k channels with time.sleep(2)  30 seconds then connection reset error (when bulk joining 100 channels and waiting for the next request)
        # waiting 30 seconds  doesnt fix this either stop at about 500  channels so lasted 2.5minutes? 
        # waiting 60 seconds at 500 channels breaks

        if count_every_request == 1: # for every 100 channels 
            connect() 
            count_every_request = 0 


        r = requests.get("https://api.twitch.tv/helix/streams?first=100&after=" + cursor,headers=helix_headers)
        cursor = r.json()['pagination']['cursor']

        count_every_request += 1


        for everything in r.json()['data']:
            user_list.append(everything['user_login'])
            channel = everything['user_login']
            # join channel

            if sock_numb == connections_made: # makes it so when joining sockets it joins up to the amount of sockets that there are and then loops back
                sock_numb = 0 
            print(f"JOINING  #{channel} with socket: {sock_numb} total joined: {total_chans_joined}")
            sockets[sock_numb].send(bytes('JOIN #' + channel + '\r\n', 'utf-8'))
            total_chans_joined += 1
            sock_numb += 1



def loop():    
    print("Looping")
    try:
        while True:
            time.sleep(0.1)
            for i in range(connections_made): 
                data = sockets[i].recv(4096).decode("utf-8",errors='replace').strip()
                if data == "":
                    continue

                print(data)

                if "PING :tmi.twitch.tv" in data:
                    print("PONG")
                    sockets[i].send(bytes('PONG :tmi.twitch.tv' + '\r\n', 'utf-8'))

    except Exception as e:
        print(str(e) + " error in loop ")
        pass 



thread_loop = threading.Thread(target=loop)
thread_loop.start()


request(channels_to_join=10000,cursor = "eyJiIjp7IkN1cnNvciI6ImV5SnpJam80T0RrMU1TNDVNRFkwTWpnd09URTVNU3dpWkNJNlptRnNjMlVzSW5RaU9uUnlkV1Y5In0sImEiOnsiQ3Vyc29yIjoiZXlKeklqbzFNakF6TGpJM056UTFPVEUzT1RReE1Td2laQ0k2Wm1Gc2MyVXNJblFpT25SeWRXVjkifX0")

Upvotes: 0

Views: 533

Answers (1)

Barry Carlyon
Barry Carlyon

Reputation: 1058

The likely problem is that your bot can't keep up with the message send buffer.

So you connect to many channels, but are not processing the incoming chat messages in a timely fashion. So the "queue" of messages to send from Twitch to You exceeds Twitch's buffer. And it DC's you

Or as per the IRC Rate limit guide you are sneding too many commands and getting Disconnected from the server.

Large chat bots will often split groups of channels over multiple connections to solve this issue.

Upvotes: 2

Related Questions