CODEWITHSUNDEEP
pythonpandas
Robin Svensson
Robin Svensson

Reputation: 83

How to combine different columns into one table in Python with Pandas

I've collected some sensor-values for the Cloud(csv file) and noticed a problem which I think I can go around by combining different columns into one table. I have the following table:

Sensor1x, Sensor1y, Sensor1z, Sensor2x, Sensor2y, Sensor2z

12.1,       0.1,       0.2,        2.2,       3.3,       2.2

NaN,       NaN,        NaN,        NaN,        NaN,      4.5,

etc.

When I try to remove the NaN values from let's say the Sensor1 columns, for some reason, all values in Sensor 2 columns are set to 0, while all NaN values are removed from Sensor1 columns (so at least the NaN part worked for the specific columns). Another strange thing is that I only chose to remove NaN values from one column (Sensor1x for example), but they are removed in all x, y and z columns.

So, since this results in the first 3 columns being filled and the last 3 being null, I figured I could save the removed Nan Columns in 1 variable, and do the same for my Sensor2 columns, which would give the following two variables:

First Variable:

Sensor1x, Sensor1y, Sensor1z, Sensor2x, Sensor2y, Sensor2z

12.1,       0.1,       0.2,        0,       0,       0,

etc.

Second variable:

Sensor1x, Sensor1y, Sensor1z, Sensor2x, Sensor2y, Sensor2z

0,          0,         0,          2.2,       3.3,       2.2,

etc.

How would I combine these two variables so I get the full table?

Simple code I'm currently using in Jupyter Notebook:

SDO = pd.read_csv('SensorOutputData202021.csv')
SDO = SDO[SDO['Ax'].notna()]
SDO

The simple 3 lines I'm using to print the dataframe and remove the na values. And also the lines of code that for some reason nullifies the values from the rest of my three columns.

Image of my current sensor table:

enter image description here

Here is what happends when I use notna function on Ax:

enter image description here

Upvotes: 0

Views: 388

Answers (1)

Roman Zh.
Roman Zh.

Reputation: 1049

At the moment you are using the notna() method which returns the mask of your column ('Ax'): Trues at values and Falses at NaNs. So then you can call:

SDO = SDO[SDO['Ax'].notna()]

You are removing each row where Ax is None.

You cannot "just delete" Nones from the table as you should keep its structure.

I suggest you to look for the following methods in addition to notna():

  • replace() - replaces target value by another value;
  • dropna() - remove rows or columns with missing values;
  • fillna() - filling missing values using specific logic;

Also you can read this part of the pandas documentation

So, depending on what you are going to do with your data you should or remove rows with NaN data completely or replace these NaNs with something that you can treat in your code in a meaningful way.

Upvotes: 1

Related Questions