CODEWITHSUNDEEP
pythonpycharmpython-typingurlopen
alecxe
alecxe

Reputation: 474281

Passing a list as a url value to urlopen

Motivation

Motivated by this problem - the OP was using urlopen() and accidentally passed a sys.argv list instead of a string as a url. This error message was thrown:

AttributeError: 'list' object has no attribute 'timeout'

Because of the way urlopen was written, the error message itself and the traceback is not very informative and may be difficult to understand especially for a Python newcomer:

Traceback (most recent call last):
  File "test.py", line 15, in <module>
    get_category_links(sys.argv)
  File "test.py", line 10, in get_category_links
    response = urlopen(url)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 420, in open
    req.timeout = timeout
AttributeError: 'list' object has no attribute 'timeout'

Problem

Here is the shortened code I'm working with:

try:
    from urllib.request import urlopen
except ImportError:
    from urllib2 import urlopen

import sys


def get_category_links(url):
    response = urlopen(url)
    # do smth with response
    print(response)


get_category_links(sys.argv)

I'm trying to think whether this kind of an error can be caught statically with either smart IDEs like PyCharm, static code analysis tools like flake8 or pylint, or with language features like type annotations.

But, I'm failing to detect the problem:

Question

Is it possible to catch this problem statically (without actually executing the code)?

Upvotes: 9

Views: 1295

Answers (2)

Supratim Samantray
Supratim Samantray

Reputation: 136

Firstly, you need to check whether the url type is string or not and if string then check for ValueError exception(Valid url)

import sys
from urllib2 import urlopen

def get_category_links(url):
  if type(url) != type(""):  #Check if url is string or not
      print "Please give string url"
      return
  try:
      response = urlopen(url)
      # do smth with response
      print(response)
  except ValueError:        #If url is string but invalid
      print "Bad URL"

get_category_links(sys.argv)

Upvotes: 0

Ashwini Chaudhary
Ashwini Chaudhary

Reputation: 251196

Instead of keeping it editor specific, you can use mypy to analyze your code. This way it will run on all dev environments instead of just for those who use PyCharm.

from urllib.request import urlopen
import sys


def get_category_links(url: str) -> None:
    response = urlopen(url)
    # do smth with response


get_category_links(sys.argv)
response = urlopen(sys.argv)

The issues pointed out by mypy for the above code:

error: Argument 1 to "get_category_links" has incompatible type List[str]; expected "str"
error: Argument 1 to "urlopen" has incompatible type List[str]; expected "Union[str, Request]"

Mypy here can guess the type of sys.argv because of its definition in its stub file. Right now some standard library modules are still missing from typeshed though, so you will have to either contribute them or ignore the errors related till they get added :-).

When to run mypy?

  1. To catch such errors you can run mypy on the files with annotations with your tests in your CI tool. Running it on all files in project may take some time, for a small project it is your choice.

  2. Add a pre-commit hook that runs mypy on staged files and points out issues right away(could be a little annoying to the dev if it takes a while).

Upvotes: 6

Related Questions