Duplicated columns in Julia Dataframes

Question

In Python Pandas and R one can get rid of duplicated columns easily - just load the data, assign the column names, and select those that are not duplicated.

What is the best practices to deal with such data with Julia Dataframes? Assigning duplicated column names is not allowed here. I understand that only way would be to massage incoming data more, and get rid of such data before constructing a Dataframe?

The thing is that it is almost always easier to deal with duplicated columns in the dataframe that is already constructed, rather than in incoming data.

UPD: I meant the duplicated column names. I build dataframe from raw data, where columns names (and thus data) could be repeated.

UPD2: Python example added.

>>> import numpy as np
>>> import pandas as pd
>>> df = pd.DataFrame(np.hstack([np.zeros((4,1)), np.ones((4,2))]), columns=["a", "b", "b"])
>>> df
     a    b    b
0  0.0  1.0  1.0
1  0.0  1.0  1.0
2  0.0  1.0  1.0
3  0.0  1.0  1.0
>>> df.loc[:, ~df.columns.duplicated()]
     a    b
0  0.0  1.0
1  0.0  1.0
2  0.0  1.0
3  0.0  1.0

I build my Julia Dataframe from a Float32 matrix and then assign column names from a vector. That is where I need to get rid of columns that have duplicated names (already present in dataframe). That is the nature of underlying data, sometimes it has dups, sometimes not, I have no control on its creation.

Answer 1

Is this something you are looking for (I was not 100% sure from your description - if this is not what you want then please update the question with an example):

julia> df = DataFrame([zeros(4,3) ones(4,5)])
4×8 DataFrame
│ Row │ x1      │ x2      │ x3      │ x4      │ x5      │ x6      │ x7      │ x8      │
│     │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │
├─────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ 1   │ 0.0     │ 0.0     │ 0.0     │ 1.0     │ 1.0     │ 1.0     │ 1.0     │ 1.0     │
│ 2   │ 0.0     │ 0.0     │ 0.0     │ 1.0     │ 1.0     │ 1.0     │ 1.0     │ 1.0     │
│ 3   │ 0.0     │ 0.0     │ 0.0     │ 1.0     │ 1.0     │ 1.0     │ 1.0     │ 1.0     │
│ 4   │ 0.0     │ 0.0     │ 0.0     │ 1.0     │ 1.0     │ 1.0     │ 1.0     │ 1.0     │

julia> DataFrame(unique(last, pairs(eachcol(df))))
4×2 DataFrame
│ Row │ x1      │ x4      │
│     │ Float64 │ Float64 │
├─────┼─────────┼─────────┤
│ 1   │ 0.0     │ 1.0     │
│ 2   │ 0.0     │ 1.0     │
│ 3   │ 0.0     │ 1.0     │
│ 4   │ 0.0     │ 1.0     │

EDIT

To deduplicate column names use makeunique keyword argument:

julia> DataFrame(rand(3,4), [:x, :x, :x, :x], makeunique=true)
3×4 DataFrame
│ Row │ x         │ x_1      │ x_2      │ x_3       │
│     │ Float64   │ Float64  │ Float64  │ Float64   │
├─────┼───────────┼──────────┼──────────┼───────────┤
│ 1   │ 0.410494  │ 0.775563 │ 0.819916 │ 0.0520466 │
│ 2   │ 0.0503997 │ 0.427499 │ 0.262234 │ 0.965793  │
│ 3   │ 0.838595  │ 0.996305 │ 0.833607 │ 0.953539  │

EDIT 2

So you seem to have access to column names when creating a data frame. In this case I would do:

julia> mat = [ones(3,1) zeros(3,2)]
3×3 Array{Float64,2}:
 1.0  0.0  0.0
 1.0  0.0  0.0
 1.0  0.0  0.0

julia> cols = ["a", "b", "b"]
3-element Array{String,1}:
 "a"
 "b"
 "b"

julia> df = DataFrame(mat, cols, makeunique=true)
3×3 DataFrame
│ Row │ a       │ b       │ b_1     │
│     │ Float64 │ Float64 │ Float64 │
├─────┼─────────┼─────────┼─────────┤
│ 1   │ 1.0     │ 0.0     │ 0.0     │
│ 2   │ 1.0     │ 0.0     │ 0.0     │
│ 3   │ 1.0     │ 0.0     │ 0.0     │

julia> select!(df, unique(cols))
3×2 DataFrame
│ Row │ a       │ b       │
│     │ Float64 │ Float64 │
├─────┼─────────┼─────────┤
│ 1   │ 1.0     │ 0.0     │
│ 2   │ 1.0     │ 0.0     │
│ 3   │ 1.0     │ 0.0     │