CODEWITHSUNDEEP
apache-sparkspark-streaming
catapan
catapan

Reputation: 161

In spark or spark streaming, how to configure log4j properities so that no too much logs are printing in the console?

When I try to run a spark streaming program, there is too much logs printed out in the console. Can you recommend a good configuration so I can get only helpful in formation only?

Upvotes: 2

Views: 1303

Answers (2)

Pinela
Pinela

Reputation: 1

I had the same issue and changed the log4j.properties as indicated. the output then turned into this :D

-------------------------------------------
Time: 2015-11-06 00:13:05
-------------------------------------------
()
15/11/06 00:13:06 WARN BlockManager: Block input-0-1446768785800 replicated to only 0 peer(s) instead of 1 peers
-------------------------------------------
Time: 2015-11-06 00:13:10
-------------------------------------------
(u'hello', 1)
()
15/11/06 00:13:14 WARN BlockManager: Block input-0-1446768794000 replicated to only 0 peer(s) instead of 1 peers
-------------------------------------------
Time: 2015-11-06 00:13:15
-------------------------------------------
(u'', 1)
(u'you', 2)
(u'how', 1)
(u'are', 1)
()

looks great :D thanks

Upvotes: 0

jkgeyti
jkgeyti

Reputation: 2404

Edit conf/log4j.properties to suit your needs.

You most likely just want to change

log4j.rootCategory=INFO, console

to

log4j.rootCategory=WARN, console

Upvotes: 2

Related Questions