CODEWITHSUNDEEP
rdplyrlapply
dhbrand
dhbrand

Reputation: 162

Get column names with zero variance using dplyr

I'm trying to find any variables in my data that have zero variance (i.e. constant continuous variables). I figured out how to do it with lapply but I would like to use dplyr as I'm trying to follow tidy data principles. I can create a vector of just the variances using dplyr but its the last step where I find the values not equal to zero and return the variable names that confusing me.

Here's the code 

library(PReMiuM)
library(tidyverse)
#> ── Attaching packages ───────────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──
#> ✔ ggplot2 2.2.1     ✔ purrr   0.2.4
#> ✔ tibble  1.4.2     ✔ dplyr   0.7.4
#> ✔ tidyr   0.7.2     ✔ stringr 1.2.0
#> ✔ readr   1.2.0     ✔ forcats 0.2.0
#> ── Conflicts ──────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
#> ✖ dplyr::filter() masks stats::filter()
#> ✖ dplyr::lag()    masks stats::lag()


setwd("~/Stapleton_Lab/Projects/Premium/hybridAnalysis/")

# read in data from analysis script
df <- read_csv("./hybrid.csv")
#> Parsed with column specification:
#> cols(
#>   .default = col_double(),
#>   Exp = col_character(),
#>   Pedi = col_character(),
#>   Harvest = col_character()
#> )
#> See spec(...) for full column specifications.

# checking for missing variable
# df %>% 
#     select_if(function(x) any(is.na(x))) %>% 
    # summarise_all(funs(sum(is.na(.))))


# grab month for analysis
may <- df %>% 
    filter(Month==5)
june <- df %>% 
    filter(Month==6)
july <- df %>% 
    filter(Month==7)
aug <- df %>% 
    filter(Month==8)
sept <- df %>% 
    filter(Month==9)
oct <- df %>% 
    filter(Month==10)

# check for zero variance in continuous covariates
numericVars <- grep("Min|Max",names(june))

zero <- which(lapply(june[numericVars],var)==0,useNames = TRUE)

noVar <- june %>% 

    select(numericVars) %>% 

    summarise_all(var) %>% 

    filter_if(all, all_vars(. != 0))
#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

#> Warning in .p(.tbl[[vars[[i]]]], ...): coercing argument of type 'double'
#> to logical

Upvotes: 7

Views: 5259

Answers (4)

Agile Bean
Agile Bean

Reputation: 7161

The answers here are all good, but as dplyr 1.0.0 deprecated the scoped variants (e.g. select_if, select_at, filter_all), here is an update using the same repex data given by @Benjamin:

mtcars2 <- mtcars
mtcars2$mpg <- mtcars2$qsec <- 7

mtcars2 %>% 
  map_df( ~ var(.)) %>% 
  select(where( ~ . == 0))

gives

# A tibble: 1 x 2
    mpg  qsec
  <dbl> <dbl>
1     0     0

or after %>% names:

[1] "mpg"  "qsec"

Upvotes: 0

Benjamin
Benjamin

Reputation: 17279

With a reproducible example, I think what you are aiming for is below. Please note that as pointed out by Colin, I have not dealt with the issue of you selecting variables with a character variable. See his answer for details on that.

# reproducible data
mtcars2 <- mtcars
mtcars2$mpg <- mtcars2$qsec <- 7

library(dplyr)

mtcars2 %>% 
  summarise_all(var) %>% 
  select_if(function(.) . == 0) %>% 
  names()
# [1] "mpg"  "qsec"

Personally, I think that obfuscates what you are doing. One of the following using the purrr package (if you wish to remain in the tidyverse) would be my preference, with a well written comment.

library(purrr)

# Return a character vector of variable names which have 0 variance
names(mtcars2)[which(map_dbl(mtcars2, var) == 0)]
names(mtcars2)[map_lgl(mtcars2, function(x) var(x) == 0)]

If you'd like to optimize it for speed, stick with base R

# Return a character vector of variable names which have 0 variance
names(mtcars2)[vapply(mtcars2, function(x) var(x) == 0, logical(1))]

Upvotes: 6

zx8754
zx8754

Reputation: 56219

Select columns if unique count is 1 then get column names, using @Benjamin's example data mtcars2:

mtcars2 %>% 
  select_if(function(.) n_distinct(.) == 1) %>% 
  names()
# [1] "mpg"  "qsec"

Upvotes: 2

GcL
GcL

Reputation: 615

You have two problems.

1. Passing names of columns as a variable to select()

The vignette about that is here. programming with dplyr. The solution here is to use the select_at() scoped variant of the select function.

2. Variance equals 0

noVar <- june %>% 
    select_at(.vars=numericVars) %>% 
    summarise_all(.funs=var) %>%
    filter_all(any_vars(. == 0))

Upvotes: 3

Related Questions