CODEWITHSUNDEEP
python
JSRB
JSRB

Reputation: 2623

How to iterate over a `groupBy` returned object in Python?

I have the following data input as a single string:

data = ['historical_trades', '1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '343727655', '2019.11.14 21:24:01', '2019.11.14 21:24:01', '6', '0.01', '', '0.00000', '0.00000', '0.00000', '0.00000', '0.00', '0.00', '100000.00', '< >', 'historical_trades', '1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '344377716', '2019.11.27 21:07:22', '2019.12.06 17:14:03', '1', '0.20', 'GBPUSD', '1.29118', '1.31069', '0.00000', '0.00000', '0.00', '4.24', '-353.20', '<>', 'historical_trades', '1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '344377723', '2019.11.27 21:07:56', '2019.12.06 17:14:02', '1', '0.20', 'GBPUSD', '1.29117', '1.31070', '0.00000', '0.00000', '0.00', '4.24', '-353.57', '<>', 'historical_trades', '1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '344377724', '2019.11.27 21:08:00', '2019.12.02 18:38:14', '1', '0.20', 'GBPUSD', '1.29118', '1.29444', '0.00000', '0.00000', '0.00', '2.36', '-58.87', '<>', 'historical_trades', '1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '345200110', '2019.12.06 17:14:08', '2019.12.06 17:22:43', '0', '0.20', 'EURUSD', '1.10474', '1.10479', '0.00000', '0.00000', '0.00', '0.00', '0.91', '<>', 'historical_trades', '1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '345200125', '2019.12.06 17:14:21', '2019.12.06 17:22:45', '0', '4.00', 'EURUSD', '1.10483', '1.10479', '0.00000', '0.00000', '0.00', '0.00', '-14.48', '<>', '']

I use the Python itertool groupBy to extract the repeating strings/objects. I then want to push these extracted objects to the SQL database while each object represents one row.

            for key, group in groupby(data, key=lambda s: s != "historical_trades"):
                if key:

                    print(list(group))

                    # print output:

                    ['1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '343727655', '2019.11.14 21:24:01', '2019.11.14 21:24:01', '6', '0.01', '', '0.00000', '0.00000', '0.00000', '0.00000', '0.00', '0.00', '100000.00', '< >']
                    ['1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '344377716', '2019.11.27 21:07:22', '2019.12.06 17:14:03', '1', '0.20', 'GBPUSD', '1.29118', '1.31069', '0.00000', '0.00000', '0.00', '4.24', '-353.20', '<>']
                    ['1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '344377723', '2019.11.27 21:07:56', '2019.12.06 17:14:02', '1', '0.20', 'GBPUSD', '1.29117', '1.31070', '0.00000', '0.00000', '0.00', '4.24', '-353.57', '<>']
                    ['1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '344377724', '2019.11.27 21:08:00', '2019.12.02 18:38:14', '1', '0.20', 'GBPUSD', '1.29118', '1.29444', '0.00000', '0.00000', '0.00', '2.36', '-58.87', '<>']
                    ['1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '345200110', '2019.12.06 17:14:08', '2019.12.06 17:22:43', '0', '0.20', 'EURUSD', '1.10474', '1.10479', '0.00000', '0.00000', '0.00', '0.00', '0.91', '<>']
                    ['1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '345200125', '2019.12.06 17:14:21', '2019.12.06 17:22:45', '0', '4.00', 'EURUSD', '1.10483', '1.10479', '0.00000', '0.00000', '0.00', '0.00', '-14.48', '<>', '']

                    # Insert iteration logic here, to iterate returned (printed) object to execute the following statements.

                    [...]
                    [...]
                    s.th. like: "for every row in list(group):" ??

                    version = data[1]
                    DID = uuid.UUID(data[2])
                    accountNumber = int(data[3])
                    orderTicket = data[4]
                    orderOpenTime = data[5]
                    orderCloseTime = data[6]
                    orderType = float(data[7])
                    orderLots = float(data[8])
                    orderSymbol = data[9]
                    orderOpenPrice = float(data[10])
                    orderClosePrice = float(data[11])
                    orderStopLoss = float(data[12])
                    orderTakeProfit = float(data[13])
                    orderCommission = float(data[14])
                    orderSwap = float(data[15])
                    orderProfit = float(data[16])
                    orderComment = data[17]


                    # push the manipulated data to the PostgreSQL DB using `Trades` model
                    # If orderTicket exists, update values, if not create new entry

                    obj, created = Trades.objects.update_or_create(
                        orderTicket=orderTicket,
                        defaults={
                            'version': version,
                            'DID': DID,
                            'accountNumber': accountNumber,
                            'orderTicket': orderTicket,
                            'orderOpenTime': orderOpenTime,
                            'orderCloseTime': orderCloseTime,
                            'orderType': orderType,
                            'orderLots': orderLots,
                            'orderSymbol': orderSymbol,
                            'orderOpenPrice': orderOpenPrice,
                            'orderClosePrice': orderClosePrice,
                            'orderStopLoss': orderStopLoss,
                            'orderTakeProfit': orderTakeProfit,
                            'orderCommission': orderCommission,
                            'orderSwap': orderSwap,
                            'orderProfit': orderProfit,
                            'orderComment': orderComment
                        }
                    )

Now how can I access the groupby returned object and iterate over it to populate the database?

Working solution:

I tried to adapt @Tobi's solution into my code and it works properly now.

            for key, group in groupby(data, key=lambda s: s != "live_trades"):
                if key:

                    trades = list(group)
                    print(trades)

                    for data in trades:

                        version = trades[0]
                        print(version)
                        DID = uuid.UUID(trades[1])
                        accountNumber = int(trades[2])
                        print(accountNumber)
                        orderTicket = trades[3]
                        print(orderTicket)
                        orderOpenTime = trades[4]
                        orderCloseTime = trades[5]
                        orderType = float(trades[6])
                        orderLots = float(trades[7])
                        orderSymbol = trades[8]
                        orderOpenPrice = float(trades[9])
                        orderClosePrice = float(trades[10])
                        orderStopLoss = float(trades[11])
                        orderTakeProfit = float(trades[12])
                        orderCommission = float(trades[13])
                        orderSwap = float(trades[14])
                        orderProfit = float(trades[15])
                        orderComment = trades[16]
                        print(orderTicket)


                        # convert dateTime strings into Python comfort dateTime

                        orderOpenTime = datetime.strptime(orderOpenTime, '%Y.%m.%d %H:%M:%S')
                        orderCloseTime = datetime.strptime(orderCloseTime, '%Y.%m.%d %H:%M:%S')

                        # push the manipulated data to the PostgreSQL DB using `Trades` model
                        # If DID exists, update values, if not create new entry

                        obj, created = Trades.objects.update_or_create(
                            orderTicket=orderTicket,
                            defaults={
                                'version': version,
                                'DID': DID,
                                'accountNumber': accountNumber,
                                'orderTicket': orderTicket,
                                'orderOpenTime': orderOpenTime,
                                'orderCloseTime': orderCloseTime,
                                'orderType': orderType,
                                'orderLots': orderLots,
                                'orderSymbol': orderSymbol,
                                'orderOpenPrice': orderOpenPrice,
                                'orderClosePrice': orderClosePrice,
                                'orderStopLoss': orderStopLoss,
                                'orderTakeProfit': orderTakeProfit,
                                'orderCommission': orderCommission,
                                'orderSwap': orderSwap,
                                'orderProfit': orderProfit,
                                'orderComment': orderComment
                            }
                        )

Upvotes: 1

Views: 178

Answers (1)

tobias_k
tobias_k

Reputation: 82929

The following should work just fine:

for data in group:
    version = data[0]
    ...

Two things to note:

  • indexing starts with 0, so the first element in data is [0], the second is [1] and so on

  • not only the result from groupby, but also the indivisual groups within it are iterators, so once you consume them, e.g. with print(list(group)), they are exhausted (I guess that was the problem when you tried your loop)

    >>> k, g = next(groupby([1,1,1,2,2,2]))
>>> list(g) # prints fine at first...
[1, 1, 1]
>>> list(g) # ... nothing happens in the loop afterwards?
[]

    Instead, you could print the current group in your inner loop, or collect all the groups in a list, i.e. group = list(group)

Also, instead of storing all the fields in data in 17 different variables and then writing them back to a dictionary, you could use zip to combine them with the correct names and types directly from data. (Only works if you need all the fields from data, but this seems to be the case here.)

fields = ('version','DID','accountNumber','orderTicket','orderOpenTime',
        'orderCloseTime','orderType','orderLots','orderSymbol','orderOpenPrice',
        'orderClosePrice','orderStopLoss','orderTakeProfit','orderCommission',
        'orderSwap','orderProfit','orderComment')

types = (str,uuid.UUID,int,str,str,str,float,float,str,float,float,float,
        float,float,float,float,str)

defaults = {f: t(d) for f,t,d in zip(fields, types, data)}
#{'orderSwap': 0.0, 'orderCloseTime': '2019.12.06 17:22:45', 'orderComment': '<>', 
# 'orderTakeProfit': 0.0, 'accountNumber': 1925323, 'DID': UUID('30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f'), 
# 'orderType': 0.0, 'orderCommission': 0.0, 'orderLots': 4.0, 'orderOpenTime': '2019.12.06 17:14:21', 
# 'orderSymbol': 'EURUSD', 'orderOpenPrice': 1.10483, 'version': '1.00', 'orderProfit': -14.48, 
# 'orderStopLoss': 0.0, 'orderClosePrice': 1.10479, 'orderTicket': '345200125'}

Upvotes: 1

Related Questions