CODEWITHSUNDEEP
javaxmlbasex
manish payasi
manish payasi

Reputation: 87

inserting millions of xml files into basex

Is it possible to store a million of records(xml files) into Basex database?

for(int j=1;j<1000000;j++) {
     String id=String.valueOf(j);
     String[] elems={"firstname","lastname","nickname","salary","permanentAddress","currentAddress","contactNo","email","alternateMail","FacebookName","Color"};
     String[] childs = {id,"yong"+id,"mook kim"+id,"mkyong"+id,"100000"+id,"Bhopal"+id,"pune"+id,"999999999"+id ,"test"+id+"@basex.com","testA"+id+"@basex.com","Tom"+id,"grey"+id};

    try {
        xmlFile= x.CreateXMLDoc("test","transperfect",elems,childs);
    } catch (TransformerConfigurationException exception) { 
        exception.printStackTrace(); 
    }

    storeIntoBaseXDB(j,id,xmlFile);         
}

//StoreIntoBaseXDB method

public static void storeIntoBaseXDB(int i,String id,String xmlFile) throws BaseXException {

    if(i==1)
    {   
        System.out.println("=========================Store into Database=========================");


        //System.out.println("Check database existence");
        con.getDatabaseConnection(dbNmae);
        con.executeInsertQuery(dbNmae,id,xmlFile);
    }
    else
    {
        con.executeInsertQuery(dbNmae,id,xmlFile);

    }
}

//executeInsertQuery() method

          public long executeInsertQuery(String dbname,String id,String xmlFile) throws BaseXException
    {

           //System.out.println(path);
            new Open(dbname).execute(context);



            try 
            {


                /* String query = "db:add('db','C:/Users/manish/Desktop/BaseX65/xml/books.xml')";
                 QueryProcessor proc = new QueryProcessor(query, context);
                    proc.execute();
                    proc.close();
                    context.close();*/

                new Add(id,xmlFile).execute(context);

                //System.out.println(new Find(id));


            }

            catch (Exception e)
            {
                System.out.println("ERROR executing query: ");
                e.printStackTrace();
            }

}

Upvotes: 3

Views: 1547

Answers (2)

dirkk
dirkk

Reputation: 6218

Adding to the answer @icza provided with the correct statistics for BaseX, I'd like to add a few things:

The upper limits on number of files, nodes, etc. pp. always apply per database. This means you can simply create another database to circumvent these restrictions. In many situations it might even be beneficial to split a large database up into several small ones, as locking is done on a database level, i.e. if you have one large database and you do a write, everything is locked. If you split it up into several databases, only a smaller portion will be locked.

As XQuery can easily access multiple databases in one query, split-up databases can easily be merged together in your queries.

Upvotes: 2

icza
icza

Reputation: 418505

Yes, it is possible. Why wouldn't it be (a million is a relatively small number)?

The limit is 2^29 which is 546,870,912 (the number of stored XML documents).

The limit for XML elements is 2^31 which is 2,147,483,648 (although this includes all nodes including attributes, texts, etc.).

Here is a web page on BaseX statistics regarding example existing databases, the first row of the table also contains the limits (#Files represents the number of stored XML documents, #Nodes represents the number of XML nodes like elements, attributes, texts, etc.):

http://docs.basex.org/wiki/Statistics

Upvotes: 2

Related Questions