CODEWITHSUNDEEP
pythonpython-3.xlist
Nastja Kr
Nastja Kr

Reputation: 169

split list of lists into 3 parts by percentage

I have list of lists like:

list = [[[bad, good],"Antonyms"], [[good, nice],"Synonyms"]]

I need to split this data into train, development and test:60%, 20%, 20% And I have no idea how to do it. The similar questions doesnt give me an answer for my case. Maybe somboody have an idea?

Thank you

Upvotes: 0

Views: 2052

Answers (2)

cph_sto
cph_sto

Reputation: 7587

Train, development and test will be the three final lists generated.

import random
l = [[['bad0', 'good0'], 'Antonyms0'], [['good0', 'nice0'], 'Synonyms0'],
 [['bad1', 'good1'], 'Antonyms1'], [['good1', 'nice1'], 'Synonyms1'],
 [['bad2', 'good2'], 'Antonyms2'], [['good2', 'nice2'], 'Synonyms2'],
 [['bad3', 'good3'], 'Antonyms3'], [['good3', 'nice3'], 'Synonyms3'],
]

#Initializing the three lists.
train = []
development = []
test = []

r = random.uniform(0, 1) # Random number generator between 0 & 1.
for i in l:
    if r <= 0.6:
        train = train + i
    elif r <= 0.8:
        development = development + i
    else:
        test = test + i

train

[['good1', 'nice1'],
 'Synonyms1',
 ['bad3', 'good3'],
 'Antonyms3',
 ['good3', 'nice3'],
 'Synonyms3']

development

 [['bad0', 'good0'],
 'Antonyms0',
 ['good0', 'nice0'],
 'Synonyms0',
 ['bad1', 'good1'],
 'Antonyms1',
 ['bad2', 'good2'],
 'Antonyms2',
 ['good2', 'nice2'],
 'Synonyms2']

test
  []

Upvotes: 1

Venkatachalam
Venkatachalam

Reputation: 16966

I am assuming that Antonyms, synonyms are some kind of categories for you. Using train_test_split from sklearn we can do the data splitting.

Note: I have changed the bad, good,etc into string. Hope that is the case with your dataset as well.

import numpy as np
from sklearn.model_selection import train_test_split

my_list = [[['bad', 'good'],"Antonyms"], [['good', 'nice'],"Synonyms"],
           [['good', 'nice'],"Synonyms"],[['good', 'nice'],"Synonyms"],
           [['good', 'nice'],"Synonyms"]]

data=np.array(my_list)

print(data.shape)
#(5, 2)

X,y=data[:,0],data[:,1]

#split the data to get 60% train and 40% test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)
#split the test again to get 20% dev and 20% test
X_dev, X_test, y_dev, y_test = train_test_split(X_test, y_test, test_size=0.5, random_state=42)

print(y_train.shape,y_dev.shape,y_test.shape)
#(3,) (1,) (1,)

Upvotes: 1

Related Questions