CODEWITHSUNDEEP
precisionmatrix-multiplicationkdb+q-lang
curious
curious

Reputation: 75

q kdb matrix multiplication precision

I am doing the matrix multiplication / dot product of two vectors, but losing significant precision in the process. My two tables are:

A:flip (`WORLD`MOMENTUM`VOLATIL`VALUE`SIZE`SIZENONL`GROWTH`LIQUID`LEVERAGE`ENERGY`OILGAS`OILEXPL`CHEMICAL`CONSTPP`DIVMETAL`PRECMETL`STEEL`CAPGOODS`COMMSVCS`TRANSPRT`AIRLINES`AUTOCOMP`CONSDUR`CONSVCS`MEDIA`RETAIL`FOODRETL`FOODPRD`HSHLDPRD`HEALTH`BIOTECH`PHARMAC`BANKS`DIVFINAN`INSURAN`REALEST`INTERNET`SOFTWARE`COMMUNIC`COMPUTER`SEMICOND`TELECOM`UTILITY`ARE`ARG`AUS`AUT`BEL`BHR`BRA`CAN`CHE`CHL`CHN`CHX`COL`CZE`DEU`DNK`EGY`ESP`FIN`FRA`GBR`GRC`HKG`HUN`IDN`IND`IRL`ISR`ITA`JOR`JPN`KOR`KWT`MAR`MEX`MYS`NLD`NOR`NZL`OMN`PAK`PER`PHL`POL`PRT`QAT`RUS`SAU`SGP`SWE`THA`TUR`TWN`USA`ZAF`AREC`ARGC`AUSC`AUTC`BELC`BHRC`BRAC`CANC`CHEC`CHLC`CHNC`COLC`CZEC`DEUC`DNKC`EGYC`EMUC`ESPC`FINC`FRAC`GBRC`GRCC`HKGC`HUNC`IDNC`INDC`IRLC`ISRC`ITAC`JORC`JPNC`KORC`KWTC`MARC`MEXC`MYSC`NLDC`NORC`NZLC`OMNC`PAKC`PERC`PHLC`POLC`PRTC`QATC`RUSC`SAUC`SGPC`SWEC`THAC`TURC`TWNC`USAC`ZAFC)!(enlist 610216.7;enlist -167790.7;enlist -61640.11;enlist 1211006f;enlist -805318.3;enlist -144186.8;enlist 484780.3;enlist 729160.8;enlist -1223112f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist -63589.8;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 47259.38;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist -42577.22;enlist 0f;enlist 721972.9;enlist 0f;enlist -32170.51;enlist 0f;enlist 0f;enlist 0f;enlist -152106.2;enlist 131428.1;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 610216.7;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 0f;enlist 610216.7;enlist 0f;enlist 0f;enlist 0f);
B:flip (enlist `WORLD)!(enlist 48.29499 -5.301547 17.17246 2.146016 1.69432 0.295386 1.296268 1.381681 1.931021 7.972107 1.824557 10.18463 6.161642 4.141302 20.12705 -44.44252 22.11896 6.354314 -0.455013 -1.28214 -5.993917 9.041717 -1.749305 -5.074565 -2.914975 -4.31613 -9.272181 -11.28901 -8.72011 -7.808923 -0.938631 -8.579992 6.898276 8.94749 6.497003 -1.967439 -2.090085 -0.254925 1.320048 5.30489 5.746626 -8.236259 -14.77134 -9.308569 -14.18904 -7.501979 3.634038 -3.201371 -32.09656 -5.156152 -8.858147 0.763125 -20.76443 -16.71714 -2.192779 -22.27704 -5.303177 3.503334 -2.870094 -1.125126 6.617578 -1.169103 7.511715 -4.384823 10.17381 0.769835 -3.488056 -18.70038 -6.739745 -4.994236 -7.670161 7.711241 -29.75733 5.311001 -22.09107 -33.23776 -38.32805 -17.19578 -14.23223 6.033368 -2.48355 -7.678366 -17.67733 -27.92534 -15.13988 -11.78941 -11.99404 -0.087656 -12.29577 -6.127644 -8.919363 2.165941 -0.196319 -9.859482 -6.289163 -12.44774 3.835522 -11.39685 6.8e-005 10.45363 30.1068 3.294173 3.294173 0.11509 33.12787 25.92562 -1.497061 20.91031 3.470695 32.76403 5.528537 3.294173 3.31778 -0.172639 3.294173 3.294173 3.294173 3.294173 13.11109 3.294173 0.708191 14.87056 8.111332 11.23322 3.294173 10.9295 3.294173 0.056315 -25.3915 26.04028 -0.344381 3.045744 41.80897 15.30789 3.294173 18.46745 24.13028 0.065611 0.511451 7.17594 9.577834 19.91554 3.294173 0.448114 33.1457 0.036037 9.321848 13.95029 7.793591 30.57135 11.67946 0 44.67686);

I have tried a few combinations of mmu and $, including:

(raze value flip A) mmu flip value flip B
(flip value flip A) mmu flip value flip B

The result I get is 41,756,840, the correct result is 41,756,847. I cannot see where the problem is, any ideas welcome, thank you.

Upvotes: 0

Views: 273

Answers (2)

SJT
SJT

Reputation: 1097

I’ve failed to reproduce your problem. I’m using V3.6 32-bit on macOS Sierra. 

q)\P 10
q)A
WORLD    MOMENTUM  VOLATIL   VALUE   SIZE      SIZENONL  GROWTH   LIQUID   LE..
-----------------------------------------------------------------------------..
610216.7 -167790.7 -61640.11 1211006 -805318.3 -144186.8 484780.3 729160.8 -1..
q)3#B
WORLD
---------
48.29499
-5.301547
17.17246
q)(raze value flip A) mmu flip value flip B
,41756847.3
q)(flip value flip A) mmu flip value flip B
41756847.3

You don’t of course need mmu here. You have two vectors and want the sum of their products. 

q)a:610216.7 -167790.7 -61640.11 1211006 -805318.3 -144186.8 484780.3 729160.8 -1223112 0 0 0 0 -63589.8 0 0 0 0 0 0 0 0 47259.38 0 0 0 0 0 0 0 0 0 0 -42577.22 0 721972.9 0 -32170.51 0 0 0 -152106.2 131428.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 610216.7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 610216.7 0 0 0
q)b:48.29499 -5.301547 17.17246 2.146016 1.69432 0.295386 1.296268 1.381681 1.931021 7.972107 1.824557 10.18463 6.161642 4.141302 20.12705 -44.44252 22.11896 6.354314 -0.455013 -1.28214 -5.993917 9.041717 -1.749305 -5.074565 -2.914975 -4.31613 -9.272181 -11.28901 -8.72011 -7.808923 -0.938631 -8.579992 6.898276 8.94749 6.497003 -1.967439 -2.090085 -0.254925 1.320048 5.30489 5.746626 -8.236259 -14.77134 -9.308569 -14.18904 -7.501979 3.634038 -3.201371 -32.09656 -5.156152 -8.858147 0.763125 -20.76443 -16.71714 -2.192779 -22.27704 -5.303177 3.503334 -2.870094 -1.125126 6.617578 -1.169103 7.511715 -4.384823 10.17381 0.769835 -3.488056 -18.70038 -6.739745 -4.994236 -7.670161 7.711241 -29.75733 5.311001 -22.09107 -33.23776 -38.32805 -17.19578 -14.23223 6.033368 -2.48355 -7.678366 -17.67733 -27.92534 -15.13988 -11.78941 -11.99404 -0.087656 -12.29577 -6.127644 -8.919363 2.165941 -0.196319 -9.859482 -6.289163 -12.44774 3.835522 -11.39685 6.8e-005 10.45363 30.1068 3.294173 3.294173 0.11509 33.12787 25.92562 -1.497061 20.91031 3.470695 32.76403 5.528537 3.294173 3.31778 -0.172639 3.294173 3.294173 3.294173 3.294173 13.11109 3.294173 0.708191 14.87056 8.111332 11.23322 3.294173 10.9295 3.294173 0.056315 -25.3915 26.04028 -0.344381 3.045744 41.80897 15.30789 3.294173 18.46745 24.13028 0.065611 0.511451 7.17594 9.577834 19.91554 3.294173 0.448114 33.1457 0.036037 9.321848 13.95029 7.793591 30.57135 11.67946 0 44.67686
q)sum a*b
41756847.3

A possible source of trouble is the location of backticks in the symbol vector in A, which can’t be seen directly in your posted question.

Upvotes: 0

Alexander Belopolsky
Alexander Belopolsky

Reputation: 2268

The correct answer given your data is neither 41,756,840 nor 41,756,847. If you increase the display precision, you will see

q)\P 16
q)(flip value flip A) mmu flip value flip B
41756847.29598875

It looks like you are misinterpreting the limited precision display somewhere.

I would also like to note that your problem becomes more manageable if you notice that only a few entries are nonzero in A

q)show a:(where 0<>a)#a:first A
WORLD   | 610216.7
MOMENTUM| -167790.7
VOLATIL | -61640.11
VALUE   | 1211006
SIZE    | -805318.3
SIZENONL| -144186.8
GROWTH  | 484780.3
LIQUID  | 729160.8
LEVERAGE| -1223112
CONSTPP | -63589.8
CONSDUR | 47259.38
DIVFINAN| -42577.22
REALEST | 721972.9
SOFTWARE| -32170.51
TELECOM | -152106.2
UTILITY | 131428.1
TUR     | 610216.7
TURC    | 610216.7

The corresponding values in B can be found as

q)show b:(key a)!B.WORLD where 0<>value first A
WORLD   | 48.29499
MOMENTUM| -5.301547
VOLATIL | 17.17246
VALUE   | 2.146016
SIZE    | 1.69432
SIZENONL| 0.295386
GROWTH  | 1.296268
LIQUID  | 1.381681
LEVERAGE| 1.931021
CONSTPP | 4.141302
CONSDUR | -1.749305
DIVFINAN| 8.94749
REALEST | -1.967439
SOFTWARE| -0.254925
TELECOM | -8.236259
UTILITY | -14.77134
TUR     | -6.289163
TURC    | 30.57135

and the dot product is

q)(value a) mmu value b
41756847.29598875

Upvotes: 1

Related Questions