Cannot parse file using JDBC

Question

Im trying to parse a pipe delimited file and insert fields into a table. when i start the application nothing happens in my DB. My DB has 4 columns (account_name, command_name, and system_name, CreateDt). The file i am parsing has the date in the first row then extra data. The rows following i only need the first 3 fields in each the rest is extra data. the last row is the row count. i skipped the inserting date because for now but want to get back to it after at least able to insert the first 3 fields. I have little experience with parsing a file and storing data in a DB and have looked through jdbc examples to get to this point but im struggling and am sure there is a better way.

File Example

20200310|extra|extra|extra||
Mn1223|01192|windows|extra|extra|extra||
Sd1223|02390|linux|extra|extra|extra||
2

table format

account_name      command_name   system_name    createDt
Mn1223            01192          windows        20200310
Sd1223            02390          linux          20200310

Code to parse and insert into DB

public List insertZygateData (List parseData) throws Exception {
        String filePath = "C:\DEV\Test_file.xlsx";


        List lines = Files.readAllLines(Paths.get(filePath));

        // remove date and amount
        lines.remove(0);
        lines.remove(lines.size() - 1);

        for (ZygateEntity zygateInfo : parseData){
                    new MapSqlParameterSource("account_name", zygateInfo.getAccountName())
                            .addValue("command_name", zygateInfo.getCommandName())
                            .addValue("system_name", zygateInfo.getSystemName())
                            .getValues();

        }
        return lines.stream()
                .map(s -> s.split("[|]")).map(val -> new ZygateEntity(val[0],val[1],val[2])).collect(Collectors.toList());


    }


    public boolean cleantheTable() throws SQLException {

        String sql = "INSERT INTO Landing.midrange_xygate_load (account_name,command_name,system_name)"+
                "VALUES (:account_name,:command_name,:system_name)";

        boolean truncated = false;
        Statement stmt = null;
        try {
            String sqlTruncate = "truncate table Landing.midrange_xygate_load";
            jdbcTemplate.execute(sqlTruncate);
            truncated = true;

        } catch (Exception e) {
            e.printStackTrace();
            truncated = false;
            return truncated;
        } finally {
            if (stmt != null) {
                jdbcTemplate.execute(sql);
                stmt.close();
            }

        }
        log.info("Clean the table return value :" + truncated);
        return truncated;
    }
}

Entity/Model

public ZygateEntity(String accountName, String commandName, String systemName){
        this.accountName=accountName;
        this.commandName=commandName;
        this.systemName=systemName;
    }

   //getters and setters
    @Override
    public String toString() {
        return "ZygateEntity [accountName=" + accountName + ", commandName=" + commandName + ", systemName=" + systemName + ", createDt=" + createDt +"]";
    }
}

Luke Woodward · Accepted Answer

Taking a look at what you've provided, it seems you have a jumbled collection of bits of code, and while most of it is there, it's not all there and not quite all in the right order.

To get some kind of clarity, try to break down what it is you're doing into separate steps, and have a method that focuses on each step. In particular, you write

Im trying to parse a pipe delimited file and insert fields into a table

This naturally breaks down into two parts:

parsing the pipe-delimited file, and
inserting fields into a table.

For the first part, you seem to have most of the parts already in your insertZygateData method. In particular, this line reads all the lines of a file into a list:

    List lines = Files.readAllLines(Paths.get(filePath));

These lines then remove the first and last lines from the list of lines read:

    // remove date and amount
    lines.remove(0);
    lines.remove(lines.size() - 1);

You then have some code that looks a bit out of place: this seems to be something to do with inserting into the database, but we haven't created our list of ZygateEntity objects as we haven't yet finished reading the file. Let's put this for loop to one side for the moment.

Finally, we take the list of lines we read, split them using pipes, create ZygateEntity objects from the parts and create a List of these objects, which we then return.

    return lines.stream()
            .map(s -> s.split("[|]")).map(val -> new ZygateEntity(val[0],val[1],val[2])).collect(Collectors.toList());

Putting this lot together, we have a useful method that parses the file, completing the first part of the task:

    private List parseZygateData() throws IOException {
        String filePath = "C:\DEV\Test_file.xlsx";

        List lines = Files.readAllLines(Paths.get(filePath));

        // remove date and amount
        lines.remove(0);
        lines.remove(lines.size() - 1);

        return lines.stream()
                .map(s -> s.split("[|]")).map(val -> new ZygateEntity(val[0],val[1],val[2])).collect(Collectors.toList());
    }

(Of course, we could add a parameter for the file path to read, but in the interest of getting something working, it's OK to stick with the current hard-coded file path.)

So, we've got our list of ZygateEntity objects. How do we write a method to insert them into the database?

We can find a couple of the ingredients we need in your code sample. First, we need the SQL statement to insert the data. This is in your cleanThetable method:

    String sql = "INSERT INTO Landing.midrange_xygate_load (account_name,command_name,system_name)"+
            "VALUES (:account_name,:command_name,:system_name)";

We then have this loop:

    for (ZygateEntity zygateInfo : parseData){
                new MapSqlParameterSource("account_name", zygateInfo.getAccountName())
                        .addValue("command_name", zygateInfo.getCommandName())
                        .addValue("system_name", zygateInfo.getSystemName())
                        .getValues();
    }

This loop creates a MapSqlParameterSource out of each ZygateEntity object, and then converts it to a Map by calling the getValues() method. But then it does nothing with this value. Effectively you're creating these objects and getting rid of them again without doing anything with them. This isn't ideal.

A MapSqlParameterSource is used with a Spring NamedParameterJdbcTemplate. Your code mentions a jdbcTemplate, which appears to be a field within the class that parses data and inserts into the database, but you don't show the full code of this class. I'm going to have to assume it's a NamedParameterJdbcTemplate rather than a 'plain' JdbcTemplate.

A NamedParameterJdbcTemplate contains a method update that takes a SQL string and a SqlParameterSource. We have a SQL string, and we're creating MapSqlParameterSource objects, so we can use these to carry out the insert. There's not a lot of point in creating one of these MapSqlParameterSource objects only to convert it to a map, so let's remove the call to getValues().

So, we now have a method to insert the data into the database:

public void insertZygateData(List parseData) {
    String sql = "INSERT INTO Landing.midrange_xygate_load (account_name,command_name,system_name)"+
            "VALUES (:account_name,:command_name,:system_name)";

    for (ZygateEntity zygateInfo : parseData){
        SqlParameterSource source = new MapSqlParameterSource("account_name", zygateInfo.getAccountName())
                .addValue("command_name", zygateInfo.getCommandName())
                .addValue("system_name", zygateInfo.getSystemName());
        jdbcTemplate.update(sql, source);
    }
}

Finally, let's take a look at your cleanThetable method. As with the others, let's keep it focused on one task: it looks like at the moment you're trying to delete the data out of the table and then insert it in the same method, but let's have it just focus on deleting the data as we've now got a method to insert the data.

We can't immediately get rid of the String sql = ... line, because the finally block in your code uses it. If stmt is not null, then you attempt to run the INSERT statement and then close stmt.

However, stmt is never assigned any value other than null, so it remains null. stmt != null is therefore always false, so the INSERT statement never runs. Your finally block never does anything, so you would be best off removing it altogether. With your finally block gone, you can also get rid of your local variable stmt and the sql string, leaving us with a method whose focus is to truncate the table:

public boolean cleantheTable() throws SQLException {
    boolean truncated = false;
    try {
        String sqlTruncate = "truncate table Landing.midrange_xygate_load";
        jdbcTemplate.execute(sqlTruncate);
        truncated = true;

    } catch (Exception e) {
        e.printStackTrace();
        truncated = false;
        return truncated;
    }
    log.info("Clean the table return value :" + truncated);
    return truncated;
}

I'll leave it up to you to write the code that calls these methods. I wrote some code for this purpose, and it ran successfully and inserted into a database.

So, in summary, no data was being written to your database because you were never making a call to the database to insert any. In your insertZygateData method you were creating the parameter-source objects but not doing anything useful with them, and in your cleanThetable method, it looked like you were trying to insert data, but your line jdbcTemplate.execute(sql) that attempted to do this never ran. Even if stmt wasn't null, this line wouldn't work as you didn't pass the parameter values in anywhere: you would get an exception from the database as it would be expecting values for the parameters but you never gave it any.

Hopefully my explanation gives you a way of getting your code working and helps you understand why it wasn't.

Cannot parse file using JDBC

Answers (1)

Related Questions