CODEWITHSUNDEEP
gocassandracqlcassandra-2.0gocql
Dalpapa
Dalpapa

Reputation: 177

gocql SELECT * doesn't return all columns

I came across this weird behaviour while trying to implement some counters for my application. Basically, I did a counter table like so :

CREATE TABLE stats_dev.log_counters (
  date text PRIMARY KEY,
  all counter
);

Then I have some specific types of message I want to count as well, so in my Go app, I ALTER the table to add the column I didn't have before.

My app is growing, and I start to have more than 30 columns (shouldn't be more than 50) and when I want to retrieve all those counters, some columns are missing in the result.

query := s.Query(`SELECT * FROM `+_apiCountersTable+` WHERE date IN ?`, dates)
res, err := query.Iter().SliceMap()

This returns me something like 30 over 34 columns. Although, when I do the request on CQLSH :

cqlsh:stats_dev> SELECT * FROM api_counters WHERE date = 'total';

I get the proper full result. So :

  1. Does that come from my request which should be different ?
  2. Could that come from gocql driver ?
  3. Is that pattern completely stupid ?

My temporary solution is to SELECT the column names from the system.schema_columns table and to strings.Join() all of that to my SELECT query ...

Thank you very much for your help.

Upvotes: 2

Views: 2604

Answers (2)

Dalpapa
Dalpapa

Reputation: 177

Thanks Andy for your help.

At first, I thought that considering what you told me, I would rather do a SELCT column_name on the system.schema_columns sometimes and refresh it when I alter my table. I would just then strings.join() the columns in my SELECT FROM api_counters. It worked but if I had 2 different instances, and one would update the schema and the other would receive a GET request, this one would not know the new column still.

And then I rearranged my ideas and found out that there was obviously an other way of doing that and I simply change for this schema : CREATE TABLE stats_dev.api_counters ( date text, description text, all counter, PRIMARY KEY (date, description) ); and I am updating the field based on the description I am expecting. So far so good.

I knew it was definitely Option 3 : my pattern was not the best one.

Upvotes: 2

Andy Tolbert
Andy Tolbert

Reputation: 11638

I'm not familiar with the gocql library, but it sounds like you may be running into a combination of not repreparing your statements and CASSANDRA-7910.

Whenever a request is Prepared (like what is being done in Select * from ___ where date in ?), it sends a request to cassandra, which responds with the column metadata for that table, so when you get a response to a query back from cassandra, you know what columns are available to look for. It looks like gocql has a feature called Automatic query preparation which may be treating your request as a prepared statement.

When you alter a table, the prepared statement does not get updated on your client side, so really the only way to fix this is to reprepare your statement (not sure if you have that level of control from gocql). However this still doesn't work since there is a bug in cassandra (CASSANDRA-7910) where it does not return the new columns since it is itself caching the prepared statement on it's side and not invalidating it when the schema changes. This issue is fixed in 2.1.3 (coming soon), it may be worth trying this against the cassandra-2.1 branch in git to see if that resolves your issue.

It's not an abnormal pattern to alter your schema when your application is running, so this is a scenario that should work, but unfortunately doesn't. I'd look into seeing if there is a way to reprepare statements in gocql.

I see there is a stmtsLRU var in cluster.go. If you could somehow get to that you could invalidate prepared statements. If there isn't a way to do this, would be good to open up an issue against gocql as you can reprepare statements again in other drivers. I know that the java driver allows you to do this, but gives you a warning. I suppose this could be a big difference between gocql and the other drivers in that in the other drivers you explicitly use a prepared statement object, where in gocql it gets handled for you automatically in the library.

With the cassandra bug outstanding, I think you should stick to not using prepared statements and instead make queries like: SELECT * FROM api_counters WHERE date = 'total';

Upvotes: 2

Related Questions