CODEWITHSUNDEEP
kdb+q-lang
mchen
mchen

Reputation: 10156

How to remove consecutive duplicate rows in KDB?

For instance, if I have the below table, then I want to remove the 3rd row:

Stock   Price
-------------------
GOOG    101
GOOG    102
GOOG    102     <- want to remove this
GOOG    101

Note: even though row 4 is a duplicate of row 1, I don't want to remove it as it is not a consecutive duplicate. That is, it is not a duplicate of the row immediately above.

I would also want to do check for duplicates across multiple fields, not just Price.

Upvotes: 5

Views: 3081

Answers (2)

JPC
JPC

Reputation: 1919

You can also use differ

q)t:([]stock:4#`GOOG; price:101 102 102 101)
q)differ t
1101b
q)t where differ t
stock price
-----------
GOOG  101
GOOG  102
GOOG  101

now let's suppose there is a time column, as you indicate in your comment above

q)t:update time:til count i from t
q)t
stock price time
----------------
GOOG  101   0
GOOG  102   1
GOOG  102   2
GOOG  101   3
q)t where differ `stock`price#t
stock price time
----------------
GOOG  101   0
GOOG  102   1
GOOG  101   3

Now going back to the t without a time column, for simplicity. This gives you speed up over the alternative method proposed by @jgleeson (which I think is great, but a speed up is always welcomed so thought I'd share this regardless)

q)\ts do[10000;r:t where differ t]
31 1184j
q)\ts do[10000;r2:t where not t~'prev t]
62 1488j
q)r~r2
1b

Upvotes: 5

jgleeson
jgleeson

Reputation: 955

d:([]Stock:4#`GOOG;Price:101 102 102 101)
q)d
Stock Price
-----------
GOOG  101
GOOG  102
GOOG  102
GOOG  101

q)d where not d~'prev d
Stock Price
-----------
GOOG  101
GOOG  102
GOOG  101

Upvotes: 8

Related Questions