CODEWITHSUNDEEP
pythonjsonscrapy
Jerry Guo
Jerry Guo

Reputation: 1

json.dumps() can not convert a dict whose value is list

When I learn the web crawler by scrapy, and save the data to json. I found that json.dumps () can not work on a list of values for the list,like this:

def __init__(self):
    self.file = codecs.open("mydata2.json","wb",encoding="utf-8")

def process_item(self, item, spider):

    line = json.dumps(i,ensure_ascii=False) + '\n'
    print(line)
    self.file.write(line)
    return item
def close_spider(self,spider):
    self.file.close()

it doesn't work.Then i modify the code

def __init__(self):
    self.file = codecs.open("mydata2.json","wb",encoding="utf-8")

def process_item(self, item, spider):
    i = dict(item)
    for key in i.keys():
        i[key] = str(i[key])
    print(i)
    line = json.dumps(i,ensure_ascii=False) + '\n'
    print(line)
    self.file.write(line)
    return item
def close_spider(self,spider):
    self.file.close()

Everything is all right.So how is this going?

Upvotes: 0

Views: 705

Answers (1)

GuangshengZuo
GuangshengZuo

Reputation: 4687

Maybe There are some data in origin type which can not be serialized. But if you convert all to string, it can be serialized.

You can define your custom JsonEncoder to fix this.

To use a custom JSONEncoder subclass (e.g. one that overrides the default() method to serialize additional types), specify it with the cls kwarg; otherwise JSONEncoder is used.

Upvotes: 1

Related Questions