CODEWITHSUNDEEP
android
Samy Louize Hanna
Samy Louize Hanna

Reputation: 831

read large text file from server android

I worked in android app that read large text file from server (20 MB) line by line , After each line insert into sqlite DB some data extracted from this line and so on .

the problem is, this application takes one hour approximately to finish reading this file . note : there are 400,000 line in this file .

how could i fast this operation to take 3 or 4 minutes ? Thanks for your help .

    public void readArData() {
    try {
        URL url = new URL("http://10.0.2.2/xy.txt");
        BufferedReader in = new BufferedReader(new   
    inputStreamReader(url.openStream()));
        String str;
        while ((str = in.readLine()) != null) {
            // insert into Sqlite DB
        }
        in.close();
    } catch (MalformedURLException e) {
    } catch (IOException e) {
    }

}

Upvotes: 1

Views: 2262

Answers (3)

Moog
Moog

Reputation: 10193

Text files are highly compressible so use the gzip compression available on Android.

Other options include:

  • splitting your dataset into multiple files and again using gzip download them separately

  • seeding the database during the install and creating a synchronization method to update newer records and bring the mobile version up to date.\

  • remove redundant data using an efficient database design and only downloading what's required

UPDATE

To clarify:

  1. zip the txt file on the server.
  2. change your android code to download the zip file to local storage on device.
  3. unzip the local copy to local storage.

You should now have a copy of your txt file on local storage so you can delete the local zip file.

  1. Read the local txt file and insert database records.

The major differences in this approach are

a. that you are downloading less data b. your download is not interrupted by sql inserts.

You can also try using a transaction to improve insert speed as described in Android SQLite database: slow insertion

db.beginTransaction();
for (entry : listOfEntries) {
    db.insert(entry);
}
db.setTransactionSuccessful();
db.endTransaction();

Upvotes: 1

tacone
tacone

Reputation: 11441

  1. make sure you have 20mb of space in the internal storage or sd card.
  2. dowload the file all at once, better if zipped. (as it will weight circa 3-4mb)
  3. then you can open it and parse it. Use queries that insert more than row a time (check the INSERT sqlite syntax).
  4. delete the file.

this approach will also allow you to resume an interrupted download, which is not bad.

Upvotes: 0

Grambot
Grambot

Reputation: 4514

Are you sure 3 - 4 minutes is an achievable amount of time? I ask because you have to:

1) Download the file contents 2) Parse the file contents 3) Insert into a database

Some of these operations can occur simultaneously but #1 is highly dependent on the network speed and 2/3 are highly dependent on what sort of system resources you have available.

Instead, can we ask why you need to pull all this data down to the phone and drop it in a database? Is there some issue with your app querying a database hosted on the webserver that holds the .txt? What sort of access to the .txt do you have? Can you modify it on the server so parsing can be done quicker?

In short, I think we need more data to analyze your issue. Can you post some examples of lines from the .txt and the code you use to parse it?

Upvotes: 2

Related Questions