Mikołaj
Mikołaj

Reputation: 385

Rvest binding all comments from forum thread into one data.frame

I want to collect forum commentaries to the data.frame. I want to collect all commentaries from one url to one row. Number of commentaries will not be equal (maximum commentaries = 3) so I want to get "NA" in any cells without commentaries.

   library(rvest)
    library(xml2)
    library(qdapRegex)
    url_page <- c("http://medyczka.pl/forum-gastrologiczne/")
    web <- read_html(url_page)
    post_url <- web %>% html_nodes(css='.title') %>%
      html_attr('href') %>%
      as.character()
    post_url <-data.frame(post_url)

#Prepare df for all possible commentaries
posts_all <-data.frame()
#Let' make a simple function
for(i in 1:5){
  web2<-read_html(as.character(post_url[i,1]))
  posts <- web2 %>% html_nodes(css='.restore') %>%
    html_text() %>%
    as.character()
  posts <- rm_between(posts,'\r','\t', replacement="")
  posts_df <-data.frame(posts)
  posts_all <-rbind(posts_all,posts_df)
}
str(posts_all)

However I got 1 column with 92 rows, instead of 5 rows up to 30 commentaries (columns).

#> str(posts_all)
#'data.frame':  92 obs. of  1 variable:
 #$ posts: Factor w/ 89 levels "Do tego potrzeba wiedzy a tu nie ma lekarzy. Radzę zapytać na innym forum medycznym.",..: 14 4 11 3 1 6 12 2 7 10 ...

What I made wrong? How to properly collect the data?

Upvotes: 0

Views: 90

Answers (1)

ryanhnkim
ryanhnkim

Reputation: 269

I'd suggest you looking into tidyverse along with rvest. This should be a good starter for you;

library(tidyverse)
library(rvest)

pg <- read_html('http://medyczka.pl/forum-gastrologiczne/')

tbl <- tibble(
        title = pg %>% html_nodes('.title') %>% html_text(),
        title_link = pg %>% html_nodes('.title') %>% html_attr('href')
        )

get_comment <- function(url) {

pg <- read_html(url)
tbl <- tibble( posts = pg %>% html_nodes('.restore') %>% html_text() %>% str_trim(),
               link = url
                ) %>% mutate(post_seq = seq_along(posts))
}

posts_all <- NULL

posts_all <-  map(tbl$title_link, get_comment) %>% bind_rows()

tbl_posts_all <- tbl %>% left_join(posts_all, by = c('title_link' = 'link'))

Upvotes: 1

Related Questions