Execute SELECT sql by randomly picking tables

Question

I am working on a project in which I have two tables in a different database with different schemas. So that means I have two different connection parameters for those two tables to connect using JDBC-

Let's suppose below is the config.property file-

TABLES: table1 table2

#For Table1
table1.url: jdbc:mysql://localhost:3306/garden
table1.user: gardener
table1.password: shavel
table1.driver: jdbc-driver
table1.percentage: 80



#For Table2
table2.url: jdbc:mysql://otherhost:3306/forest
table2.user: forester
table2.password: axe
table2.driver: jdbc-driver
table2.percentage: 20

Below method will read the above config.property file and make a ReadTableConnectionInfo object for each tables.

private static HashMap<String, ReadTableConnectionInfo> tableList = new HashMap<String, ReadTableConnectionInfo>();

private static void readPropertyFile() throws IOException {

    prop.load(Read.class.getClassLoader().getResourceAsStream("config.properties"));

    tableNames = Arrays.asList(prop.getProperty("TABLES").split(" "));

    for (String arg : tableNames) {

        ReadTableConnectionInfo ci = new ReadTableConnectionInfo();

        String url = prop.getProperty(arg + ".url");
        String user = prop.getProperty(arg + ".user");
        String password = prop.getProperty(arg + ".password");
        String driver = prop.getProperty(arg + ".driver");
        double percentage = Double.parseDouble(prop.getProperty(arg + ".percentage"));

        ci.setUrl(url);
        ci.setUser(user);
        ci.setPassword(password);
        ci.setDriver(driver);
        ci.setPercentage(percentage);

        tableList.put(arg, ci);
    }

}

Below is the ReadTableConnectionInfo class that will hold all the table connection info for a particular table.

public class ReadTableConnectionInfo {

    public String url;
    public String user;
    public String password;
    public String driver;
    public String percentage;

    public String getUrl() {
        return url;
    }

    public void setUrl(String url) {
        this.url = url;
    }

    public String getUser() {
        return user;
    }

    public void setUser(String user) {
        this.user = user;
    }

    public String getPassword() {
        return password;
    }

    public void setPassword(String password) {
        this.password = password;
    }

    public String getDriver() {
        return driver;
    }

    public void setDriver(String driver) {
        this.driver = driver;
    }

    public double getPercentage() {
        return percentage;
    }

    public void setPercentage(double percentage) {
        this.percentage = percentage;
    }
}

Now I am creating ExecutorService for specified number of threads and passing this tableList object to constructor of ReadTask class-

        // create thread pool with given size
        ExecutorService service = Executors.newFixedThreadPool(10);

        for (int i = 0; i < 10; i++) {
            service.submit(new ReadTask(tableList));
        }

Below is my ReadTask that implements Runnable interface in which each thread is supposed to make a connection for each tables.

class ReadTask implements Runnable {

    private final HashMap<String, XMPReadTableConnectionInfo> tableLists;

public ReadTask(HashMap<String, ReadTableConnectionInfo> tableList) {
    this.tableLists = tableList;
}


@Override
public void run() {

    int j = 0;
    dbConnection = new Connection[tableLists.size()];
    statement = new Statement[tableLists.size()];

    //loop around the map values and make the connection list
    for (ReadTableConnectionInfo ci : tableLists.values()) {

        dbConnection[j] = getDBConnection(ci.getUrl(), ci.getUser(), ci.getPassword(), ci.getDriver());
        statement[j] = dbConnection[j].createStatement();

        j++;
    }

    while (System.currentTimeMillis() <= 60 minutes) {

    /* Generate random number and check to see whether that random number
     * falls between 1 and 80, if yes, then choose table1
     * and then use table1 connection and statement that I made above and do a SELECT * on that table.
     * If that random numbers falls between 81 and 100 then choose table2 
     * and then use table2 connection and statement and do a SELECT * on that table
     */

    ResultSet rs = statement[what_table_statement].executeQuery(selectTableSQL);

    }
     }
}

Currently I have two tables, that means each thread will make two connections for each table and then use that particular table connection for doing SELECT * on that table depending on the random generation number.

Algorithm:-

1. Generate Random number between 1 and 100.
2. If that random number is less than table1.getPercentage() then choose table1 and then use table1 statement object to make a SELECT sql call to that database.
3. else choose table2 and then use table2 statement object to make a SELECT sql call to that database.

My Question-

I am having hard time in figuring out how should apply the above algorithm and how should I compare the random number with each tables percentage and then decide which table I need to use and after that figure out which table connection and statements I need to use to make a SELECT sql call.

So that means I need to check getPercentage() method of each table and them compare with the Random Number.

Right now I have only two tables, in future I can have three tables, with percentage distribution might be as 80 10 10.

UPDATE:-

class ReadTask implements Runnable {

    private Connection[] dbConnection = null;
    private ConcurrentHashMap<ReadTableConnectionInfo, Connection> tableStatement = new ConcurrentHashMap<ReadTableConnectionInfo, Connection>();

    public ReadTask(LinkedHashMap<String, XMPReadTableConnectionInfo> tableList) {
        this.tableLists = tableList;
    }


    @Override
    public run() {

    int j = 0;
    dbConnection = new Connection[tableLists.size()];

    //loop around the map values and make the connection list
    for (ReadTableConnectionInfo ci : tableLists.values()) {

    dbConnection[j] = getDBConnection(ci.getUrl(), ci.getUser(), ci.getPassword(), ci.getDriver());
    tableStatement.putIfAbsent(ci, dbConnection[j]);

    j++;
    }

      Random random = new SecureRandom();

      while ( < 60 minutes) {

        double randomNumber = random.nextDouble() * 100.0;
        ReadTableConnectionInfo table = selectRandomConnection(randomNumber);

        for (Map.Entry<ReadTableConnectionInfo, Connection> entry : tableStatement.entrySet()) {

            if (entry.getKey().getTableName().equals(table.getTableName())) {

                final String id = generateRandomId(random);
                final String selectSql = generateRandomSQL(table);

                preparedStatement = entry.getValue().prepareCall(selectSql);
                preparedStatement.setString(1, id);

                rs = preparedStatement.executeQuery();
            }
        }
      }
    }



        private String generateRandomSQL(ReadTableConnectionInfo table) {

        int rNumber = random.nextInt(table.getColumns().size());

        List<String> shuffledColumns = new ArrayList<String>(table.getColumns());
        Collections.shuffle(shuffledColumns);

        String columnsList = "";

        for (int i = 0; i < rNumber; i++) {
            columnsList += ("," + shuffledColumns.get(i));
        }

        final String sql = "SELECT ID" + columnsList + "  from "
                + table.getTableName() + " where id = ?";

        return sql;
    }


    private ReadTableConnectionInfo selectRandomConnection(double randomNumber) {

        double limit = 0;
        for (ReadTableConnectionInfo ci : tableLists.values()) {
            limit += ci.getPercentage();
            if (random.nextDouble() < limit) {
                return ci;
            }
            throw new IllegalStateException();
        }
        return null;
    }
    }

Answer 1

You could think of it as a loop over the available connections, something like the following:

public run() {
  ...
  Random random = new SecureRandom();

  while ( < 60 minutes) {
    double randomNumber = random.nextDouble() * 100.0;
    ReadTableConnectionInfo tableInfo = selectRandomConnection(randomNumber);

    // do query...
  }
}


private ReadTableConnectionInfo selectRandomConnection(double randomNumber) {
  double limit = 0;
  for (ReadTableConnectionInfo ci : tableLists.values()) {
    limit += ci.getPercentage();
    if (randomNumber < limit) {
      return ci;
  }
  throw new IllegalStateException();
}

As long as randomNumber has a maximum value of less then sum(percentage), that'll do the job.

One other thing I thought of: if you're going to end up having so many possible queries that the a looping lookup becomes an issue, you could build a lookup table: create an array such that the total size of the array contains enough entries so that the relative weightings of the queries can be represented with integers.

For your example of three queries, 80:10:10, have a 10-entry array of ReadTableConnectionInfo with eight references pointing to table1, one to table2, and one to table3. Then simply scale your random number to be 0 <= rand < 10 (eg (int)(Math.random() * 10), and use it to index in to your array.

Answer 2

You can build a lookup table which contains the table name and its weight:

class LookupTable {
    private int[]    weights;
    private String[] tables;
    private int      size = 0;

    public LookupTable(int n) {
        this.weights = new int[n];
        this.tables = new String[n];
    }

    public void addTable(String tableName, int r) {
        this.weights[size] = r;
        this.tables[size] = tableName;
        size++;
    }

    public String lookupTable(int n) {
        for (int i = 0; i < this.size; i++) {
            if (this.weights[i] >= n) {
                return this.tables[i];
            }
        }
        return null;
    }
}

The code to initialize the table:

    LookupTable tr = new LookupTable(3);
    // make sure adds the range from lower to upper!
    tr.addTable("table1", 20);
    tr.addTable("table2", 80);
    tr.addTable("table3", 100);

The test code:

    Random r = new Random(System.currentTimeMillis());
    for (int i = 0; i < 10; i++) {
        // r.nextInt(101) + 1 would return a number of range [1~100]. 
        int n = r.nextInt(101) + 1;
        String tableName = tr.lookupTable(n);
        System.out.println(n + ":" + tableName);
    }

Answer 3

Regardless of how many tables you have, their percentages will always add up to 100. The easiest way to conceptualize how you would choose is to think of each table as representing a range of percentages.

For instance, with three tables that have the percents you mentioned (80%, 10%, 10%), you could conceptualize them as:

Random Number From To == Table == 0.0000 0.8000 Table_1 0.8000 0.9000 Table_2 0.9000 1.0000 Table_3

So, generate a Random # between 0.0000 and 1.0000 and then go down the ordered list and see which range fits, and therefore which table to use.

(BTW: I'm not sure why you have two connections for each table.)