How to fix Mechanize::ChunkedTerminationError?

Question

I just received a Mechanize::ChunkedTerminationError, which I have never encountered before. All it says in the Documentation is:

Raised when Mechanize detects the chunked transfer-encoding may be incorrectly terminated.

Does anybody has an example of under what circumstances this happens and how I could go about fixing it?

Answer 1

agent = Mechanize.new
agent.ignore_bad_chunking = true

Answer 2

I'm still trying to figure out the exact cause, but I had the same error and was able to avoid it by forcing the encoding, in my case to utf-8, before parsing the page contents.

page = agent.get(url)
page.encoding = 'utf-8'

I think this may happen when the contents of the page uses some character not supported by the character set specified by the web page.