CODEWITHSUNDEEP
pysparkapache-spark-sqlsasdatastep
S K SINGH
S K SINGH

Reputation: 1

Create columns by iterating over rows in pyspark dataframe - SAS code to be converted into pyspark

data work2(drop=i K);
    set work1;
    ARRAY History(36) $ History_1-History_36;
    array History_new(36)$ History_new1-History_new36;

    if INTCK('MONTH',DateReported,MIS_DATE)>=0 then
        VINTAGE_DateReported=
        INTCK('MONTH',DateReported,MIS_DATE)+1;
    else VINTAGE_DateReported=1;
    I=0;

    do k=VINTAGE_DateReported to 36;
        I+1;
        History_new(k)=History(I);
    END;
run;

above is my SAS code which has to be converted into pyspark.

I have tried this below code in pyspark, but it's not working.

work1 = work1.rdd.map(lambda x: (x.MemberReference, x.(for i in range (x.VINTAGE_DateReported_trades.collect()[0][0], 37, 1): x.withColumn('History__new' + str(i), x['History_'][i])))

Upvotes: 0

Views: 36

Answers (0)

Related Questions