CODEWITHSUNDEEP
filterelasticsearchlogstash
Sagnik Sinha
Sagnik Sinha

Reputation: 873

Parsing a string as date in logstash while inserting in elasticsearch

One record in my csv file looks like 

,No,FMN1116CD,Holiday Order,2782427,Mr Vijay Ran ,9/6/2014,17/11/2014,,To Deliver,0,S,FALSE,2726149,-1,-1,,0,,,,-1,Delhi Inbound,NEW DELHI,Basic Hotel Order Details,Custom Package,Double,,Others,25500,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,890,0,0,3280,0,29670,,,FALSE,not paid,287894747,,TRUE,,5,-301767,0,50764,,,,,,,,,,,,-3065174,,,,,,,,,,,,,,,,,,,,2,India,22/11/2014,,Manual,Custom Package,26889,Balasore,Callcenter,0,0,0,0

My conf file looks like

input {  
  file {
      path => "/home/sagnik/Work/logstash-1.4.2/bin/ho.csv"
      start_position => "beginning"
  }
}



filter {
    date {
       match => ["Travel_Date", "dd/MM/YYYY"]

    }
    csv {
       columns =>     ["Comm_Plan","Queue_Booking","Order_Reference","Multi_Ordertype","Order_Item_Id","Pax_Name","Generation_Date","Travel_Date","Desk_ID","Status","SalesID","UserRole","Group_Booking","Agent_ID","Admin_ID","Partner_ID","Partner_Name","AgencyAdmin_Id","Supp_Pmt_Ref","Supp_Pmt_Acc","Supp_Pmt_Status","Distributor","Agent_Name","State","Supplier_Code","Secondary_Supplier_Code","Supplier_Number","PNR","Ticket_Number","Basic","Taxes","OCTax","Meal_Price","Cab_Price","Handling","PLB","Deposit_Incentive","Subagent_Handling","Subagent_Plb","Subagent_Deposit_Incentive","Dist_Comm","Stax_Air","Booking_Surcharge","TDS","SubAgent_TDS","Dist_TDS","Dist_Service_Tax","STax_Bas","Partner_Booking_Fee","Old_Payment_Fee","Transaction_Fee_Rcvd","Transaction_Fee_Givn","Net_Amount","Vouchers","CC","Dist_Credit","Partner_Payment_Status","Call_CenterId","Linked_Order","Is_Holiday","Child_Ordertype","Room_Nights","Payment_Sum","Credit_Outstanding","Payment_Fee","DepositCharge","DepositComm_Cr","CreditCharge","CreditComm_Cr","Distributor_CreditCharge","Distributor_CreditComm_Cr","Vendor_7Charge","CCICICI_MOTO_3DCharge","IPSPCharge","NetBanking_TPSCharge","CCICICI_EMICharge","NetBanking_CITRUSCharge","CCHDFC_MOTOCharge","ACharge","CCAMEXCharge","NetBanking_4Charge","NetBanking_PayUCharge","Ccivrscharge","Vch_LossVoucher","Vch_StaffTravel","Vch_DiscountB2C","Vch_ViaPointsRedemption","Vch_DealVoucher","Vch_BonusRedemption","Vch_Loss","Vch_MultiOrder","Vch_SME","Vch_TripCard","Vch_NetPayments","Vch_OfferPromo","Vch_HotelPromotion","No_Of_Pax","Hotel_CountryName","Checkout_Date","Hotel_Booking_Code","Hotel_Type","Hotel_Name","Hotel_Id","Hotel_City","Hotel_Booked_By","Hotel_Net","Hotel_Taxes","Hotel_Gross","Hotel_Supplier_Commission"]
       separator => ","
    }
}



output {  
    elasticsearch {
        host => "localhost"
        index => "hotel"
    }
    stdout { codec => rubydebug }
}

But after insertion the Travel_Date is coming as a string and not a date. As a result I am unable to do any navigation with it. Please help

Upvotes: 2

Views: 5727

Answers (2)

Brijesh Mishra
Brijesh Mishra

Reputation: 2748

you need to use target for this case

 date {
       match => ["Travel_Date", "dd/MM/YYYY"]
        target => "New_Field_Name"
    }

Upvotes: 3

Ban-Chuan Lim
Ban-Chuan Lim

Reputation: 7890

I think you have misunderstanding date plugin. date plugin is used to parse a field value and match it to @timestamp field.

If you need convert a field value from string to date, you can use Ruby plugin to do it. With the below conf I can parse the Travel_Date to date format and navigate it in elasticsearch. Have a look. 

input {  
  file {
      path => "/home/sagnik/Work/logstash-1.4.2/bin/ho.csv"
      start_position => "beginning"
  }
}



filter {

    csv {
       columns =>     ["Comm_Plan","Queue_Booking","Order_Reference","Multi_Ordertype","Order_Item_Id","Pax_Name","Generation_Date","Travel_Date","Desk_ID","Status","SalesID","UserRole","Group_Booking","Agent_ID","Admin_ID","Partner_ID","Partner_Name","AgencyAdmin_Id","Supp_Pmt_Ref","Supp_Pmt_Acc","Supp_Pmt_Status","Distributor","Agent_Name","State","Supplier_Code","Secondary_Supplier_Code","Supplier_Number","PNR","Ticket_Number","Basic","Taxes","OCTax","Meal_Price","Cab_Price","Handling","PLB","Deposit_Incentive","Subagent_Handling","Subagent_Plb","Subagent_Deposit_Incentive","Dist_Comm","Stax_Air","Booking_Surcharge","TDS","SubAgent_TDS","Dist_TDS","Dist_Service_Tax","STax_Bas","Partner_Booking_Fee","Old_Payment_Fee","Transaction_Fee_Rcvd","Transaction_Fee_Givn","Net_Amount","Vouchers","CC","Dist_Credit","Partner_Payment_Status","Call_CenterId","Linked_Order","Is_Holiday","Child_Ordertype","Room_Nights","Payment_Sum","Credit_Outstanding","Payment_Fee","DepositCharge","DepositComm_Cr","CreditCharge","CreditComm_Cr","Distributor_CreditCharge","Distributor_CreditComm_Cr","Vendor_7Charge","CCICICI_MOTO_3DCharge","IPSPCharge","NetBanking_TPSCharge","CCICICI_EMICharge","NetBanking_CITRUSCharge","CCHDFC_MOTOCharge","ACharge","CCAMEXCharge","NetBanking_4Charge","NetBanking_PayUCharge","Ccivrscharge","Vch_LossVoucher","Vch_StaffTravel","Vch_DiscountB2C","Vch_ViaPointsRedemption","Vch_DealVoucher","Vch_BonusRedemption","Vch_Loss","Vch_MultiOrder","Vch_SME","Vch_TripCard","Vch_NetPayments","Vch_OfferPromo","Vch_HotelPromotion","No_Of_Pax","Hotel_CountryName","Checkout_Date","Hotel_Booking_Code","Hotel_Type","Hotel_Name","Hotel_Id","Hotel_City","Hotel_Booked_By","Hotel_Net","Hotel_Taxes","Hotel_Gross","Hotel_Supplier_Commission"]
        separator => ","
    }
    ruby {
            code => "
                    event['Travel_Date'] = Date.parse(event['Travel_Date']);
            "
    }
}



output {  
    elasticsearch {
        host => "localhost"
        index => "hotel"
    }
    stdout { codec => rubydebug }
}

Hope this can help you.

Upvotes: 2

Related Questions