CODEWITHSUNDEEP
sslopensslscrapypyopenssl
Facundo Fabre
Facundo Fabre

Reputation: 228

Scrapy. SSL Error "DH KEY TOO SMALL" when trying to crawl site

I'm getting an the following error after retry when trying to crawl a website.

[<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_check_cert_and_algorithm', 'dh key too small')]>]

I tried with every SSL method available in Scrapy with similar result. When I go to the site in Chrome it seems the page is insecure (broken HTTPS), but I still can bypass the error. Same behaviour using python requests (I can get the site content by setting verify to False).

Is there any workaround? Can't I just turn off SSL validations the way I do in python requests?

P.S. Share the site URL makes no sense since it only allows requests from whitelisted IPs.

Upvotes: 1

Views: 3299

Answers (1)

Steffen Ullrich
Steffen Ullrich

Reputation: 123531

Disabling validation will not help since this is not a problem of the certificate validation. What could help is a change of the cipher used, i.e. disable DH ciphers so that the code affected by weak DH keys (logjam attack) gets not used. What would also help is use of an older version of OpenSSL which does not yet protect against the logjam attack.

Unfortunately there seems to be no obvious way to specify the cipher set to use in Scrapy. Maybe one could find one if hooking into twisted or OpenSSL libraries.

Upvotes: -1

Related Questions