Java Voice Chat - Mixdown Incoming Data for Single Output

Question

I am trying to write a Java voice chat application, and have achieved echo capabilities, but when trying to connect multiple clients, I am stuck. I understand that you cannot iterate through sockets and send the data to everyone connected without mixing-down the data. (I have tried and it sounds nothing like how it should). I am not quite sure what to do, and I am using a very simple byte-buffered echo server as the server (where I would like to perform the mixdown). I also have a client that takes microphone input, sends it to the server, takes data from the server, and plays that data out of a speaker.

NOTE: The client is composed of 2 classes (Program and SoundReceiver). I am using the javax.sound.sampled library.

Echo Server: http://pastebin.com/c9KiaTpJ

import java.net.*;
import java.io.*;
import java.util.*;

public class Echo
{
    public static void main(String[] args) throws Exception
    {
        ServerSocket serverSocket = new ServerSocket(3000);
        while(true){Thread echoThread = new Thread(new EchoThread(serverSocket.accept()));
                    echoThread.start();}
    }
}

class EchoThread implements Runnable
{
    public static Collection<Socket> sockets = new ArrayList<Socket>();
    Socket connection = null;
    DataInputStream dataIn = null;
    DataOutputStream dataOut = null;

    public EchoThread(Socket conn) throws Exception
    {
        connection = conn;
        dataIn = new DataInputStream(connection.getInputStream());
        dataOut = new DataOutputStream(connection.getOutputStream());
        sockets.add(connection);
    }

    public void run()
    {
        int bytesRead = 0;
        byte[] inBytes = new byte[1];
        while(bytesRead != -1)
        {
            try{bytesRead = dataIn.read(inBytes, 0, inBytes.length);}catch (IOException e){}
            if(bytesRead >= 0)
            {
                sendToAll(inBytes, bytesRead);
            }
        }
        sockets.remove(connection);
    }

    public static void sendToAll(byte[] byteArray, int q)
    {
        Iterator<Socket> sockIt = sockets.iterator();
        while(sockIt.hasNext())
        {
            Socket temp = sockIt.next();
            DataOutputStream tempOut = null;
            try
            {
                tempOut = new DataOutputStream(temp.getOutputStream());
            } catch (IOException e1)
            {
                // TODO Auto-generated catch block
                e1.printStackTrace();
            }
            try{tempOut.write(byteArray, 0, q);}catch (IOException e){}
        }
    }
}

Client Program Class: http://pastebin.com/v24CYwXE

import java.io.DataOutputStream;
import java.net.*;
import javax.sound.sampled.*;

public class Program
{
    public static void main(String[] args) throws Exception
    {
        AudioFormat af = new AudioFormat(8000.0f,8,1,true,false);
        DataLine.Info info = new DataLine.Info(TargetDataLine.class, af);
        TargetDataLine microphone = (TargetDataLine)AudioSystem.getLine(info);
        microphone.open(af);
        Socket conn = new Socket("localhost",3000);
        microphone.start();
        DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
        int bytesRead = 0;
        byte[] soundData = new byte[1];
        Thread inThread = new Thread(new SoundReceiver(conn));
        inThread.start();
        while(bytesRead != -1)
        {
            bytesRead = microphone.read(soundData, 0, soundData.length);
            if(bytesRead >= 0)
            {
                dos.write(soundData, 0, bytesRead);
            }
        }
        System.out.println("IT IS DONE.");
    }
}

Client SoundReceiver Class: http://pastebin.com/2tt0Jucv

import java.net.*;
import java.io.*;

import javax.sound.sampled.*;

public class SoundReceiver implements Runnable
{
    Socket connection = null;
    DataInputStream soundIn = null;
    SourceDataLine inSpeaker = null;

    public SoundReceiver(Socket conn) throws Exception
    {
        connection = conn;
        soundIn = new DataInputStream(connection.getInputStream());
        AudioFormat af = new AudioFormat(8000.0f,8,1,true,false);
        DataLine.Info info = new DataLine.Info(SourceDataLine.class, af);
        inSpeaker = (SourceDataLine)AudioSystem.getLine(info);
        inSpeaker.open(af);
    }

    public void run()
    {
        int bytesRead = 0;
        byte[] inSound = new byte[1];
        inSpeaker.start();
        while(bytesRead != -1)
        {
            try{bytesRead = soundIn.read(inSound, 0, inSound.length);} catch (Exception e){}
            if(bytesRead >= 0)
            {
                inSpeaker.write(inSound, 0, bytesRead);
            }
        }
    }
}

Basically, I'd like to merge all incoming bytes into one byte array while keeping everyone's voice full (just like a 3-way phone call).

Answer 1

I think you need to create a check in the server side. its like if sendAll is being called from EChoThread that has connection instance just pass this to sendAll and there compare sockIt with connection if they are the same then this is the same socket that is sending the date and there is no need to send data to it self so just skip it and move to the next socket.

the following changes should be made at server side:

     public void run()
{
    int bytesRead = 0;
    byte[] inBytes = new byte[1];
    while(bytesRead != -1)
    {
        try{bytesRead = dataIn.read(inBytes, 0, inBytes.length);}catch (IOException e)       {}
        if(bytesRead >= 0)
        {
            sendToAll(connection, inBytes, bytesRead);
        }
    }
    sockets.remove(connection);
}


public static void sendToAll(Socket connection, byte[] byteArray, int q)
{
    Iterator<socket> sockIt = sockets.iterator();
    while(sockIt.hasNext())
    {
        Socket temp = sockIt.next();
        if(connection == temp){
            continue;
        }
        DataOutputStream tempOut = null;
        try
        {
            tempOut = new DataOutputStream(temp.getOutputStream());
        } catch (IOException e1)
        {
            // TODO Auto-generated catch block
            e1.printStackTrace();
        }
        try{tempOut.write(byteArray, 0, q);}catch (IOException e){}
      }
  }




 public void run()
{
    int bytesRead = 0;
    byte[] inBytes = new byte[1];
    while(bytesRead != -1)
    {
        try{bytesRead = dataIn.read(inBytes, 0, inBytes.length);}catch (IOException e)       {}
        if(bytesRead >= 0)
        {
            sendToAll(connection, inBytes, bytesRead);
        }
    }
    sockets.remove(connection);
}


public static void sendToAll(Socket connection, byte[] byteArray, int q)
{
    Iterator<socket> sockIt = sockets.iterator();
    while(sockIt.hasNext())
    {
        Socket temp = sockIt.next();
        if(connection == temp){
            continue;
        }
        DataOutputStream tempOut = null;
        try
        {
            tempOut = new DataOutputStream(temp.getOutputStream());
        } catch (IOException e1)
        {
            // TODO Auto-generated catch block
            e1.printStackTrace();
        }
        try{tempOut.write(byteArray, 0, q);}catch (IOException e){}
      }
  }

Answer 2

It's this line:

try
{
    tempOut.write(byteArray, 0, q);
}
catch (IOException e){
} 

in server side which i think sends the data back to the client due which there is an echo. I think you should omit the line.

Answer 3

setting the limit to the serverSocket may help, eg new ServerSocket(3000,101); something like backlog or queue length..