Create a matrix from the dataframe of all permutations

Question

I have a dataframe for all possible combinations of 2 stations in Caltrain and fare for the trip between them. I want to make a matrix of values but cannot understand how. Data looks like that:

+=========================+=========================+========+
|       SOURCE_STN        |        DEST_STN         | RC1($) |
+=========================+=========================+========+
| 22nd Street             | Atherton                |   8.25 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Redwood City            |      6 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Millbrae Transit Center |      6 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Stanford                |   8.25 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Santa Clara             |   10.5 |
+-------------------------+-------------------------+--------+
| 22nd Street             | California Ave.         |   8.25 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Tamien                  |   10.5 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Lawrence                |   10.5 |
+-------------------------+-------------------------+--------+
| 22nd Street             | San Antonio             |   8.25 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Burlingame              |      6 |
+-------------------------+-------------------------+--------+
| 22nd Street             | San Mateo               |      6 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Hayward Park            |      6 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Blossom Hill            |  12.75 |
+-------------------------+-------------------------+--------+
| 22nd Street             | South San Francisco     |   3.75 |
+-------------------------+-------------------------+--------+
| 22nd Street             | San Carlos              |      6 |
+-------------------------+-------------------------+--------+
| 22nd Street             | San Jose Diridon        |   10.5 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Gilroy                  |     15 |
+-------------------------+-------------------------+--------+
| 22nd Street             | College Park            |   10.5 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Menlo Park              |   8.25 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Palo Alto               |   8.25 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Belmont                 |      6 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Menlo Park              |   8.25 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Broadway                |      6 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Sunnyvale               |   8.25 |
+-------------------------+-------------------------+--------+
| 22nd Street             | San Martin              |     15 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Morgan Hill             |     15 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Hillsdale               |      6 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Capitol                 |  12.75 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Bayshore                |   3.75 |
+-------------------------+-------------------------+--------+
| 22nd Street             | Mountain View           |   8.25 |
+-------------------------+-------------------------+--------+
| 22nd Street             | San Francisco           |   3.75 |
+-------------------------+-------------------------+--------+
| Atherton                | Redwood City            |      6 |
+-------------------------+-------------------------+--------+
| Atherton                | Millbrae Transit Center |      6 |
+-------------------------+-------------------------+--------+
| Atherton                | Stanford                |   8.25 |
+-------------------------+-------------------------+--------+
| Atherton                | Santa Clara             |   10.5 |
+-------------------------+-------------------------+--------+
| Atherton                | California Ave.         |   8.25 |
+-------------------------+-------------------------+--------+
| Atherton                | Tamien                  |   10.5 |
+-------------------------+-------------------------+--------+
| Atherton                | Lawrence                |   10.5 |
+-------------------------+-------------------------+--------+
| Atherton                | San Antonio             |   8.25 |
+-------------------------+-------------------------+--------+
| Atherton                | Burlingame              |      6 |
+-------------------------+-------------------------+--------+
| Atherton                | San Mateo               |      6 |
+-------------------------+-------------------------+--------+
| Atherton                | Hayward Park            |      6 |
+-------------------------+-------------------------+--------+
| Atherton                | Blossom Hill            |  12.75 |
+-------------------------+-------------------------+--------+
| Atherton                | South San Francisco     |   3.75 |
+-------------------------+-------------------------+--------+
| Atherton                | San Carlos              |      6 |
+-------------------------+-------------------------+--------+
| Atherton                | San Jose Diridon        |   10.5 |
+-------------------------+-------------------------+--------+
| Atherton                | Gilroy                  |     15 |
+-------------------------+-------------------------+--------+
| Atherton                | College Park            |   10.5 |
+-------------------------+-------------------------+--------+
| Atherton                | Menlo Park              |   8.25 |
+-------------------------+-------------------------+--------+
| Atherton                | Palo Alto               |   8.25 |
+-------------------------+-------------------------+--------+
| Atherton                | Menlo Park              |   8.25 |
+-------------------------+-------------------------+--------+
| Atherton                | 22nd Street             |   3.75 |
+-------------------------+-------------------------+--------+
| Atherton                | Broadway                |      6 |
+-------------------------+-------------------------+--------+
| Atherton                | Sunnyvale               |   8.25 |
+-------------------------+-------------------------+--------+
| Atherton                | San Martin              |     15 |
+-------------------------+-------------------------+--------+
| Atherton                | Belmont                 |      6 |
+-------------------------+-------------------------+--------+
| Atherton                | Morgan Hill             |     15 |
+-------------------------+-------------------------+--------+
| Atherton                | Hillsdale               |      6 |
+-------------------------+-------------------------+--------+
| Atherton                | Capitol                 |  12.75 |
+-------------------------+-------------------------+--------+
| Atherton                | Bayshore                |   3.75 |
+-------------------------+-------------------------+--------+
| Atherton                | Mountain View           |   8.25 |
+-------------------------+-------------------------+--------+
| Atherton                | San Francisco           |   3.75 |
+-------------------------+-------------------------+--------+
| Bayshore                | Atherton                |   8.25 |
+-------------------------+-------------------------+--------+
| Bayshore                | Redwood City            |      6 |
+-------------------------+-------------------------+--------+
| Bayshore                | Millbrae Transit Center |      6 |
+-------------------------+-------------------------+--------+
| Bayshore                | Stanford                |   8.25 |
+-------------------------+-------------------------+--------+
| Bayshore                | Santa Clara             |   10.5 |
+-------------------------+-------------------------+--------+
| Bayshore                | Menlo Park              |   8.25 |
+-------------------------+-------------------------+--------+
| Bayshore                | 22nd Street             |   3.75 |
+-------------------------+-------------------------+--------+
| Bayshore                | Broadway                |      6 |
+-------------------------+-------------------------+--------+
| Bayshore                | Sunnyvale               |   8.25 |
+-------------------------+-------------------------+--------+
| Bayshore                | San Martin              |     15 |
+-------------------------+-------------------------+--------+
| Bayshore                | California Ave.         |   8.25 |
+-------------------------+-------------------------+--------+
| Bayshore                | Tamien                  |   10.5 |
+-------------------------+-------------------------+--------+
| Bayshore                | Lawrence                |   10.5 |
+-------------------------+-------------------------+--------+
| Bayshore                | San Antonio             |   8.25 |
+-------------------------+-------------------------+--------+
| Bayshore                | Burlingame              |      6 |
+-------------------------+-------------------------+--------+
| Bayshore                | San Mateo               |      6 |
+-------------------------+-------------------------+--------+
| Bayshore                | Hayward Park            |      6 |
+-------------------------+-------------------------+--------+
| Bayshore                | Blossom Hill            |  12.75 |
+-------------------------+-------------------------+--------+
| Bayshore                | South San Francisco     |   3.75 |
+-------------------------+-------------------------+--------+
| Bayshore                | San Carlos              |      6 |
+-------------------------+-------------------------+--------+
| Bayshore                | San Jose Diridon        |   10.5 |
+-------------------------+-------------------------+--------+
| Bayshore                | Gilroy                  |     15 |
+-------------------------+-------------------------+--------+
| Bayshore                | College Park            |   10.5 |
+-------------------------+-------------------------+--------+
| Bayshore                | Menlo Park              |   8.25 |
+-------------------------+-------------------------+--------+
| Bayshore                | Palo Alto               |   8.25 |
+-------------------------+-------------------------+--------+
| Bayshore                | Belmont                 |      6 |
+-------------------------+-------------------------+--------+
| Bayshore                | Morgan Hill             |     15 |
+-------------------------+-------------------------+--------+
| Bayshore                | Hillsdale               |      6 |
+-------------------------+-------------------------+--------+
| Bayshore                | Capitol                 |  12.75 |
+-------------------------+-------------------------+--------+
| Bayshore                | Mountain View           |   8.25 |
+-------------------------+-------------------------+--------+
| Bayshore                | San Francisco           |   3.75 |
+-------------------------+-------------------------+--------+
| Belmont                 | Atherton                |   8.25 |
+-------------------------+-------------------------+--------+
| Belmont                 | Redwood City            |      6 |
+-------------------------+-------------------------+--------+
| Belmont                 | Millbrae Transit Center |      6 |
+-------------------------+-------------------------+--------+
| Belmont                 | Stanford                |   8.25 |
+-------------------------+-------------------------+--------+
| Belmont                 | Santa Clara             |   10.5 |
+-------------------------+-------------------------+--------+
| Belmont                 | Menlo Park              |   8.25 |
+-------------------------+-------------------------+--------+
| Belmont                 | 22nd Street             |   3.75 |
+-------------------------+-------------------------+--------+
| Belmont                 | Broadway                |      6 |
+-------------------------+-------------------------+--------+
| Belmont                 | Sunnyvale               |   8.25 |
+-------------------------+-------------------------+--------+
| Belmont                 | San Martin              |     15 |
+-------------------------+-------------------------+--------+
| Belmont                 | California Ave.         |   8.25 |
+-------------------------+-------------------------+--------+
| Belmont                 | Tamien                  |   10.5 |
+-------------------------+-------------------------+--------+
| Belmont                 | Lawrence                |   10.5 |
+-------------------------+-------------------------+--------+
| Belmont                 | San Antonio             |   8.25 |
+-------------------------+-------------------------+--------+
| Belmont                 | Burlingame              |      6 |
+-------------------------+-------------------------+--------+
| Belmont                 | San Mateo               |      6 |
+-------------------------+-------------------------+--------+
| Belmont                 | Hayward Park            |      6 |
+-------------------------+-------------------------+--------+
| Belmont                 | Blossom Hill            |  12.75 |
+-------------------------+-------------------------+--------+
| Belmont                 | South San Francisco     |   3.75 |
+-------------------------+-------------------------+--------+
| Belmont                 | San Carlos              |      6 |
+-------------------------+-------------------------+--------+
| Belmont                 | San Jose Diridon        |   10.5 |
+-------------------------+-------------------------+--------+
| Belmont                 | Gilroy                  |     15 |
+-------------------------+-------------------------+--------+
| Belmont                 | College Park            |   10.5 |
+-------------------------+-------------------------+--------+
| Belmont                 | Menlo Park              |   8.25 |
+-------------------------+-------------------------+--------+
| Belmont                 | Palo Alto               |   8.25 |
+-------------------------+-------------------------+--------+
| Belmont                 | Morgan Hill             |     15 |
+-------------------------+-------------------------+--------+
| Belmont                 | Hillsdale               |      6 |
+-------------------------+-------------------------+--------+
| Belmont                 | Capitol                 |  12.75 |
+-------------------------+-------------------------+--------+
| Belmont                 | Bayshore                |   3.75 |
+-------------------------+-------------------------+--------+
| Belmont                 | Mountain View           |   8.25 |
+-------------------------+-------------------------+--------+
| Belmont                 | San Francisco           |   3.75 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Atherton                |   8.25 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Redwood City            |      6 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Millbrae Transit Center |      6 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Stanford                |   8.25 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Santa Clara             |   10.5 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Menlo Park              |   8.25 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | 22nd Street             |   3.75 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Broadway                |      6 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Sunnyvale               |   8.25 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | San Martin              |     15 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | California Ave.         |   8.25 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Tamien                  |   10.5 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Lawrence                |   10.5 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | San Antonio             |   8.25 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Burlingame              |      6 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | San Mateo               |      6 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Hayward Park            |      6 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | South San Francisco     |   3.75 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | San Carlos              |      6 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | San Jose Diridon        |   10.5 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Gilroy                  |     15 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | College Park            |   10.5 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Menlo Park              |   8.25 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Palo Alto               |   8.25 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Belmont                 |      6 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Morgan Hill             |     15 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Hillsdale               |      6 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Capitol                 |  12.75 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Bayshore                |   3.75 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | Mountain View           |   8.25 |
+-------------------------+-------------------------+--------+
| Blossom Hill            | San Francisco           |   3.75 |
+-------------------------+-------------------------+--------+
| Broadway                | Atherton                |   8.25 |
+-------------------------+-------------------------+--------+
| Broadway                | Redwood City            |      6 |
+-------------------------+-------------------------+--------+
| Broadway                | Millbrae Transit Center |      6 |
+-------------------------+-------------------------+--------+
| Broadway                | Stanford                |   8.25 |
+-------------------------+-------------------------+--------+
| Broadway                | Santa Clara             |   10.5 |
+-------------------------+-------------------------+--------+
| Broadway                | Menlo Park              |   8.25 |
+-------------------------+-------------------------+--------+
| Broadway                | 22nd Street             |   3.75 |
+-------------------------+-------------------------+--------+
| Broadway                | Sunnyvale               |   8.25 |
+-------------------------+-------------------------+--------+
| Broadway                | San Martin              |     15 |
+-------------------------+-------------------------+--------+
| Broadway                | California Ave.         |   8.25 |
+-------------------------+-------------------------+--------+
| Broadway                | Tamien                  |   10.5 |
+-------------------------+-------------------------+--------+
| Broadway                | Lawrence                |   10.5 |
+-------------------------+-------------------------+--------+
| Broadway                | San Antonio             |   8.25 |
+-------------------------+-------------------------+--------+
| Broadway                | Burlingame              |      6 |
+-------------------------+-------------------------+--------+
| Broadway                | San Mateo               |      6 |
+-------------------------+-------------------------+--------+
| Broadway                | Hayward Park            |      6 |
+-------------------------+-------------------------+--------+
| Broadway                | Blossom Hill            |  12.75 |
+-------------------------+-------------------------+--------+
| Broadway                | South San Francisco     |   3.75 |
+-------------------------+-------------------------+--------+
| Broadway                | San Carlos              |      6 |
+-------------------------+-------------------------+--------+
| Broadway                | San Jose Diridon        |   10.5 |
+-------------------------+-------------------------+--------+
| Broadway                | Gilroy                  |     15 |
+-------------------------+-------------------------+--------+
| Broadway                | College Park            |   10.5 |
+-------------------------+-------------------------+--------+
| Broadway                | Menlo Park              |   8.25 |
+-------------------------+-------------------------+--------+
| Broadway                | Palo Alto               |   8.25 |
+-------------------------+-------------------------+--------+
| Broadway                | Morgan Hill             |     15 |
+-------------------------+-------------------------+--------+
| Broadway                | Belmont                 |      6 |
+-------------------------+-------------------------+--------+
| Broadway                | Hillsdale               |      6 |
+-------------------------+-------------------------+--------+
| Broadway                | Capitol                 |  12.75 |
+-------------------------+-------------------------+--------+
| Broadway                | Bayshore                |   3.75 |
+-------------------------+-------------------------+--------+
| Broadway                | Mountain View           |   8.25 |
+-------------------------+-------------------------+--------+
| Broadway                | San Francisco           |   3.75 |
+-------------------------+-------------------------+--------+

I tried to do something with pd.Dataframe(index = , columns =) but it does not work. It must be like that:

+---------------+----------------+-----------+
|    station    |  San Francisco |  Broadway |
+---------------+----------------+-----------+
| San Francisco |  0             |  value    |
| Broadway      | value          |  0        |
+---------------+----------------+-----------+

Thank you!

Answer 1

You can achieve this by using pivot_table in pandas.

data_frame.pivot_table('RC1($)', 'SOURCE_STN', 'DEST_STN').fillna(0)

Explanation:

The matrix you expect could be achieved by the following code as the permutations are still retained while also managing duplicates by obtaining the mean of duplicates.

import pandas as pd
import numpy as np

route_table = pd.read_csv("path_to_file.csv")

route_matrix = route_table.pivot_table('RC1($)', 'SOURCE_STN', 'DEST_STN', aggfunc=np.mean).fillna(0)
print(route_matrix)

Output:

DEST_STN                 22nd Street  Atherton  ...  Sunnyvale  Tamien
SOURCE_STN                                      ...                   
22nd Street                     0.00      8.25  ...       8.25    10.5
Atherton                        3.75      0.00  ...       8.25    10.5
Bayshore                        3.75      8.25  ...       8.25    10.5
Belmont                         3.75      8.25  ...       8.25    10.5
Blossom Hill                    3.75      8.25  ...       8.25    10.5
Broadway                        3.75      8.25  ...       8.25    10.5
Burlingame                      3.75      8.25  ...       8.25    10.5
California Ave.                 3.75      8.25  ...       8.25    10.5
Capitol                         3.75      8.25  ...       8.25    10.5
College Park                    3.75      8.25  ...       8.25    10.5
Gilroy                          3.75      8.25  ...       8.25    10.5
Hillsdale                       3.75      8.25  ...       8.25    10.5

---

For a different perspective of the pivoted table to get a better understanding:

Alternate Code

route_matrix = route_table.pivot_table('RC1($)', 'DEST_STN', 'SOURCE_STN', aggfunc=np.mean).fillna(0)
print(route_matrix)

Output

SOURCE_STN               22nd Street  Atherton  ...  Sunnyvale  Tamien
DEST_STN                                        ...                   
22nd Street                     0.00      3.75  ...       3.75    3.75
Atherton                        8.25      0.00  ...       8.25    8.25
Bayshore                        3.75      3.75  ...       3.75    3.75
Belmont                         6.00      6.00  ...       6.00    6.00
Blossom Hill                   12.75     12.75  ...      12.75   12.75
Broadway                        6.00      6.00  ...       6.00    6.00
Burlingame                      6.00      6.00  ...       6.00    6.00
California Ave.                 8.25      8.25  ...       8.25    8.25
Capitol                        12.75     12.75  ...      12.75   12.75
College Park                   10.50     10.50  ...      10.50   10.50
Gilroy                         15.00     15.00  ...      15.00   15.00

Answer 2

You need to use df.pivot().

df.pivot(index='SOURCE_STN', columns='DEST_STN', values='RC1($)').fillna(0)

Edit:
For dataframes with duplicates you need to use df.pivot_table() instead and specify an aggregation function in aggfunc parameter.

import numpy as np

df.pivot_table(index='SOURCE_STN', columns='DEST_STN', values='RC1($)', aggfunc=np.sum).fillna(0)