CODEWITHSUNDEEP
goweb-scraping
AliReza
AliReza

Reputation: 55

How to get multiple element on a page with go colly

I have an struct as following: 

Type Post struct{
     ID int64
     Title string
     Content string
}

I Curl a web page for receive data with Go Colly, I have two OnHtml method as following:

func main() {
    c := colly.NewCollector()

    c.OnHTML("p", func(e *colly.HTMLElement) {
        Post := Post{
           Content: e.Text
        }
        db.Create(&Post)
    })
    c.OnHTML("h", func(e *colly.HTMLElement) {
        Post := Post{
           Title: e.Text
        }
        db.Create(&Post)        
    })

    c.Visit("http://go-colly.org/")
}

The above code works well but this create two row in database as following:

+--------------+---------------+---------------+
|      id      |     title     |    content    |
+--------------+---------------+---------------+
|       1      |      Hello    |      Null     |
+--------------+---------------+---------------+
|       2      |      Null     | Mycontent ... |
+--------------+---------------+---------------+

i want to create it :

+--------------+---------------+---------------+
|      id      |     title     |    content    |
+--------------+---------------+---------------+
|       1      |      Hello    | Mycontent ... |
+--------------+---------------+---------------+

how can I get two element and save in one row in go colly?

Upvotes: 0

Views: 4468

Answers (1)

Kangoo13
Kangoo13

Reputation: 371

You should read this example: http://go-colly.org/docs/examples/coursera_courses/ at the line where there is detailCollector.OnHTML("div[id=rendered-content]", func(e *colly.HTMLElement) {

The example set the onHTML on an element (here a div) that encapsulates the whole thing, so for you, you need to find the element that encapsulates every post containing the title + the content and then do an e.ForEach to parse every post.

EDIT: http://go-colly.org/docs/examples/factbase/ is also a good example for your use-case. Taking the body and then parses every topic with a speaker and a text.

Is that clear?

Upvotes: 6

Related Questions