CODEWITHSUNDEEP
pythonjsonscrapy
user2240632
user2240632

Reputation: 51

Scrapy dynamic creation of objects + json export

I created a new spider to crawl a website. This crawler get each video game of liste on website and create an object for it :

class gameInfos(Item):
    title = Field()
    desc = Field()
    kind = Field()

for each game, the website contains a variable list of resellers. I get each reseller in object :

class buyInfos(Item):
    name = Field()
    address = Field()
    price = Field()

Now, my problem :

I want put buyInfos object(s) inside gameInfos object and that my json file looks :

[
   {
    "title": "BF3",
    "desc": "a beautiful game",
    "kind" : "FPS",
    "buy" : 
           [
              {name : "cdiscount", "address" : "example", "price" : "45 €"},
              {name : "amazon", "address" : "example amazon", "price" : "40 €"},
               //... other resellers
           ]
   },
   {
    "title": "COD 42",
    "desc": "a game",
    "kind" : "FPS",
    "buy" :
   }, 
   //... other games
]

So I tried to create an object in my main object. It works but finally, I have only one object to fill, whereas I want create some objects inside my main object.

Thanks for your help

Upvotes: 2

Views: 745

Answers (2)

user2240632
user2240632

Reputation: 51

The solution was simple. Create an object as:

class GameInfo(Item):
    title = Field()
    desc = Field()
    kind = Field()
    listeBuys = Field()

Then, in your spider, instances GameInfo:

gameInfo = GameInfo()

Then, instances python list for desired field:

gameInfo['listeBuys'] = []

Finally, add as you want:

gameInfo['listeBuys'].append(asyouwant)

Thanks dm03514 for his help !

Upvotes: 1

dm03514
dm03514

Reputation: 55962

Scrapy Field class is a subclass of dict. If you want to make one of the fields able to contain a list of Items I believe you could create a List field.

class ListField(list):
  pass

class GameInfo(Item):
    title = Field()
    desc = Field()
    kind = Field()
    buys = ListField()

Now in your spider you can create the gameInfos and it will be able to contain all the relevant buyInfos

game_info = GameInfo()
# create your buy info and append to game info
game_info['buys'].append(new_buy_info)

Upvotes: 1

Related Questions