Reputation: 385
I want to collect forum commentaries to the data.frame. I want to collect all commentaries from one url to one row. Number of commentaries will not be equal (maximum commentaries = 3) so I want to get "NA" in any cells without commentaries.
library(rvest)
library(xml2)
library(qdapRegex)
url_page <- c("http://medyczka.pl/forum-gastrologiczne/")
web <- read_html(url_page)
post_url <- web %>% html_nodes(css='.title') %>%
html_attr('href') %>%
as.character()
post_url <-data.frame(post_url)
#Prepare df for all possible commentaries
posts_all <-data.frame()
#Let' make a simple function
for(i in 1:5){
web2<-read_html(as.character(post_url[i,1]))
posts <- web2 %>% html_nodes(css='.restore') %>%
html_text() %>%
as.character()
posts <- rm_between(posts,'\r','\t', replacement="")
posts_df <-data.frame(posts)
posts_all <-rbind(posts_all,posts_df)
}
str(posts_all)
However I got 1 column with 92 rows, instead of 5 rows up to 30 commentaries (columns).
#> str(posts_all)
#'data.frame': 92 obs. of 1 variable:
#$ posts: Factor w/ 89 levels "Do tego potrzeba wiedzy a tu nie ma lekarzy. Radzę zapytać na innym forum medycznym.",..: 14 4 11 3 1 6 12 2 7 10 ...
What I made wrong? How to properly collect the data?
Upvotes: 0
Views: 90
Reputation: 269
I'd suggest you looking into tidyverse
along with rvest
.
This should be a good starter for you;
library(tidyverse)
library(rvest)
pg <- read_html('http://medyczka.pl/forum-gastrologiczne/')
tbl <- tibble(
title = pg %>% html_nodes('.title') %>% html_text(),
title_link = pg %>% html_nodes('.title') %>% html_attr('href')
)
get_comment <- function(url) {
pg <- read_html(url)
tbl <- tibble( posts = pg %>% html_nodes('.restore') %>% html_text() %>% str_trim(),
link = url
) %>% mutate(post_seq = seq_along(posts))
}
posts_all <- NULL
posts_all <- map(tbl$title_link, get_comment) %>% bind_rows()
tbl_posts_all <- tbl %>% left_join(posts_all, by = c('title_link' = 'link'))
Upvotes: 1