CODEWITHSUNDEEP
rapache-sparkmachine-learningapache-spark-sqlsparkr
keypoint
keypoint

Reputation: 2318

how to train a naiveBayes model with Libsvm dataset format like Mnist?

I downloaded Mnist data here and its forart is like below https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass.html#mnist

$ head -1 mnist
5 153:3 154:18 155:18 156:18 157:126 158:136 159:175 160:26 161:166 162:255 163:247 164:127 177:30 178:36 179:94 180:154 181:170 182:253 183:253 184:253 185:253 186:253 187:225 188:172 189:253 190:242 191:195 192:64 204:49 205:238 206:253 207:253 208:253 209:253 210:253 211:253 212:253 213:253 214:251 215:93 216:82 217:82 218:56 219:39 232:18 233:219 234:253 235:253 236:253 237:253 238:253 239:198 240:182 241:247 242:241 261:80 262:156 263:107 264:253 265:253 266:205 267:11 269:43 270:154 290:14 291:1 292:154 293:253 294:90 320:139 321:253 322:190 323:2 348:11 349:190 350:253 351:70 377:35 378:241 379:225 380:160 381:108 382:1 406:81 407:240 408:253 409:253 410:119 411:25 435:45 436:186 437:253 438:253 439:150 440:27 464:16 465:93 466:252 467:253 468:187 494:249 495:253 496:249 497:64 519:46 520:130 521:183 522:253 523:253 524:207 525:2 545:39 546:148 547:229 548:253 549:253 550:253 551:250 552:182 571:24 572:114 573:221 574:253 575:253 576:253 577:253 578:201 579:78 597:23 598:66 599:213 600:253 601:253 602:253 603:253 604:198 605:81 606:2 623:18 624:171 625:219 626:253 627:253 628:253 629:253 630:195 631:80 632:9 649:55 650:172 651:226 652:253 653:253 654:253 655:253 656:244 657:133 658:11 677:136 678:253 679:253 680:253 681:212 682:135 683:132 684:16

Then I tried R naiveBayes package, but got error

naiveBayes formula interface handles data frames or arrays only

below is what I did:

Load R interface: ./bin/sparkR --master "local[2]"

> training <- loadDF(sqlContext, "~/Downloads/mnist", "libsvm")

> training
SparkDataFrame[label:double, features:vector]

> head(training)
  label                      features
1     0 <environment: 0x7fa8ef236630>
2     1 <environment: 0x7fa8edc3d708>
3     1 <environment: 0x7fa8edc561b8>
4     1 <environment: 0x7fa8edc6aa30>
5     1 <environment: 0x7fa8edc72aa0>
6     0 <environment: 0x7fa8edc79ab0>

> model <- naiveBayes(label ~ features, training)

Error in naiveBayes.formula(label ~ features, training) :
  naiveBayes formula interface handles data frames or arrays only

I guess it's because the "feature" is vector type and R cannot parse it. Is there anything I'm doing wrong? Or how to handle the vector type in R to get a naiveBayes model properly?

Thank you very much.

Upvotes: 0

Views: 140

Answers (1)

keypoint
keypoint

Reputation: 2318

I think I found the issue here, when building naiveBayes model. I imported R package for naiveBayes by library(e1071)

But if R package, then I should somehow convert the > training SparkDataFrame[label:double, features:vector] to a R recognizable data format which is array or data frames.

When I switch to spark built-in method spark.naiveBayes() to build model, then it's generating the model successfully as follows:

model <- spark.naiveBayes(training, label ~ features)

since built-in spark.naiveBayes() comes in couple with SparkDataFrame type

Upvotes: 0

Related Questions