CODEWITHSUNDEEP
rtidytext
Kreitz Gigs
Kreitz Gigs

Reputation: 379

Opposite of unnest_tokens after creating dummy variable

library(NLP)
library(tm)
library(tidytext)
library(tidyverse)
library(topicmodels)
library(dplyr)
library(stringr)
library(purrr)
library(tidyr)
#sample dataset
tags <- c("product, productdesign, electronicdevice")
web <- c("hardware, sunglasses, eyeware")
tags2 <- data_frame(tags, web, stringsAsFactors = FALSE)
#tokenize the words
toke <- tags2 %>%
  unnest_tokens(word, tags)
toke
#create a dummy variable
toke2 <- toke%>% mutate(
  product = ifelse(str_detect(word, "^product$"), "1", "0"))
#unnest the toke
nested_toke <- toke2 %>%
  nest(word) %>%
  mutate(text = map(data, unlist), 
         text = map_chr(text, paste, collapse = " "))

nested_toke %>%
  select(text)

When I nest the column of tokenized words after creating the dummy variable based on the string "product" it seems to be inserting "product" into a new row below the original row where "product" was located.

product underlined should be in the row above

Upvotes: 0

Views: 234

Answers (1)

Julia Silge
Julia Silge

Reputation: 11643

When you add a new column after unnesting, you have to think about what to do with it if you want to nest again. Let's work through it and see what we're talking about.

library(tidyverse)
tags <- c("product, productdesign, electronicdevice")
web <- c("hardware, sunglasses, eyeware")
tags2 <- data_frame(tags, web)

library(tidytext)
tidy_tags <- tags2 %>%
    unnest_tokens(word, tags)
tidy_tags
#> # A tibble: 3 x 2
#>   web                           word            
#>   <chr>                         <chr>           
#> 1 hardware, sunglasses, eyeware product         
#> 2 hardware, sunglasses, eyeware productdesign   
#> 3 hardware, sunglasses, eyeware electronicdevice

So that is your data set unnested, converted to a tidy form. Next, let's add the new column that detects whether the word "product" is in the word column. 

tidy_product <- tidy_tags %>% 
    mutate(product = ifelse(str_detect(word, "^product$"), 
                            TRUE, 
                            FALSE))
tidy_product
#> # A tibble: 3 x 3
#>   web                           word             product
#>   <chr>                         <chr>            <lgl>  
#> 1 hardware, sunglasses, eyeware product          T      
#> 2 hardware, sunglasses, eyeware productdesign    F      
#> 3 hardware, sunglasses, eyeware electronicdevice F

Now think about what your options are for nesting again. If you nest again without taking into account the new column (nest(word)) the structure has a NEW COLUMN and will have to make a NEW ROW to account for the two different values that can take. You could instead do something like nest(word, product) but then the TRUE/FALSE values will end up in your text string. If you are wanting to get back to the original text format, you need to remove the new column you created, because having it there changes the relationships between rows and columns. 

nested_product <- tidy_product %>%
    select(-product) %>%
    nest(word) %>%
    mutate(text = map(data, unlist), 
           text = map_chr(text, paste, collapse = ", "))

nested_product
#> # A tibble: 1 x 3
#>   web                           data             text                     
#>   <chr>                         <list>           <chr>                    
#> 1 hardware, sunglasses, eyeware <tibble [3 × 1]> product, productdesign, …

Created on 2018-02-22 by the reprex package (v0.2.0).

Upvotes: 1

Related Questions