CODEWITHSUNDEEP
sqlapache-beambeam
ah_ben
ah_ben

Reputation: 105

Apache Beam : java.lang.IllegalStateException when reading from MSSQL table

I'm having a beam pipeline that reads from MSSQL table using a simple query :

    return "SELECT "
        + "U.ID as userid, "
        + "U.firstname as firstname, "
        + "U.lastname as lastname, "
        + "email as email, "
        + "U.IP as ip "
        + "FROM users U WITH (NOLOCK) "
        + "OPTION (MAXDOP 4)";

My (simplified) users table looks like :

CREATE TABLE users (
    [ID]                   INT           IDENTITY (1000000, 1) NOT NULL,
    [Firstname]                   NVARCHAR (30) CONSTRAINT [DF_iddb_Nom]  DEFAULT ('') NOT NULL,
    [Lastname]                NVARCHAR (30) CONSTRAINT [DF_iddb_Prenom]  DEFAULT ('') NOT NULL,
    [Email]                   NVARCHAR (40) NOT NULL,
    [IP]           VARCHAR (15)  NULL
)

My processor is :

public static class DataReadFn extends DoFn<Row, User> {

    @ProcessElement
    public void processElement(ProcessContext c) {
      Row row = c.element();

      User user = new User()
          .setUserId(format(String.valueOf(row.getInt32("userid"))))
          .setFirstName(format(row.getString("firstname")))
          .setLastName(format(row.getString("lastname")))
          .setEmail(format(row.getString(“email”))
          .setIp(format(row.getString("ip"))); // last

      c.output(record);
    }

private String format(String str) {
  return str == null ? "" : str.trim();
}

}

When I execute this pipeline on Dataflow, I get following exception only when I include LastLoginIP in the query. Without LastLoginIP, it works fine.

java.lang.IllegalStateException
        at org.apache.beam.sdk.util.Preconditions.checkStateNotNull(Preconditions.java:452)
        at org.apache.beam.sdk.io.jdbc.SchemaUtil.lambda$createLogicalTypeExtractor$83184fac$1(SchemaUtil.java:307)
        at org.apache.beam.sdk.io.jdbc.SchemaUtil$BeamRowMapper.mapRow(SchemaUtil.java:380)
        at org.apache.beam.sdk.io.jdbc.SchemaUtil$BeamRowMapper.mapRow(SchemaUtil.java:358)
        at org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn.processElement(JdbcIO.java:1498)

It probably has something to do with fact that LastLoginIP is nullable but cannot tell exactly. Any ideas ? Thanks.

Upvotes: 0

Views: 157

Answers (1)

Bill Smith
Bill Smith

Reputation: 1

I ran into the same problem. It looks like that bug was fixed in Apache Beam 2.47.0. Compare the 2.46.0 and 2.47.0 versions of SchemaUtil.createLogicalTypeExtractor.

Upvotes: 0

Related Questions