CODEWITHSUNDEEP
javac++scalatime-seriesresampling
jbssm
jbssm

Reputation: 7161

Time series ressampling of historical trade data

I have some historical trade date in a csv file in the format: unixtime, price, volume I I want to analyze that data.

I managed to do it in Python, but it's painfully slow (takes me about 2 days to run the algorithm for a 30 day data test).

I'm trying to do it in c/c++ or even Java or Scala, but my main issue is that I have no way to resample the data. I need to resample this data to the format: date time, open, high, low, close, volume for 15 minutes intervals but I can't find any way to do that in c/c++

In Python this does what I want (it uses a pandas Dataframe):

def resample_data(raw_data, time_frame):
    # resamples the ticker data in ohlc
    resampledData = raw_data.copy()
    ohlc_dict = {
        'open':'first',
        'high':'max',
        'low':'min',
        'close':'last',
        'price':'first'
        }

    resampledData = resampledData.resample(time_frame, how={'price':ohlc_dict, 'amount':'sum'})
    resampledData.amount = resampledData['amount']['sum'].fillna(0.0)
    resampledData['price']['close'] = resampledData['price']['close'].fillna(method='pad')
    resampledData = resampledData.apply(lambda x: x.fillna(resampledData['price']['close']))

    return resampledData

Any ideas (or a library) that does this in c/c++/Java/scala?

Upvotes: 0

Views: 2401

Answers (2)

Gavin
Gavin

Reputation: 6490

Try looking at Saddle, for data manipulation. I have only just found this myself so am not sure of full capabilities but it is inspired by Pandas.

Upvotes: 0

yǝsʞǝla
yǝsʞǝla

Reputation: 16422

Just a quick example of what you can do with standard Scala libraries. This code can be run in Scala REPL:

// not importing external libraries like Joda time and its Scala wrappers
import java.util.Date
import scala.annotation.tailrec

case class Sample(value: Double, timeMillis: Long)
case class SampleAggregate(startTimeMillis: Long, endTimeMillis: Long,
  min: Sample, max: Sample)

val currentMillis = System.currentTimeMillis
val inSec15min = 15 * 60
val inMillis15min = inSec15min * 1000
// sample each second:
val data = (1 to inSec15min * 100).map { i =>
  Sample(i, currentMillis + i*1000) }.toList

@tailrec
def aggregate(xs: List[Sample], intervalDurationMillis: Long,
  accu: List[SampleAggregate]): List[SampleAggregate] =
  xs match {
    case h :: t =>
      val start = h.timeMillis
      val (slice, rest) = xs.span(_.timeMillis < (start + intervalDurationMillis))
      val end = slice.last.timeMillis
      val aggr = SampleAggregate(start, end, slice.minBy(_.value),
        slice.maxBy(_.value))
      aggregate(rest, intervalDurationMillis, aggr :: accu)
    case Nil =>
      accu.reverse
  }

val result = aggregate(data, inMillis15min, Nil)

Fake data:

data.take(10).foreach(println)
Sample(1.0,1388809630677)
Sample(2.0,1388809631677)
Sample(3.0,1388809632677)
Sample(4.0,1388809633677)
Sample(5.0,1388809634677)
Sample(6.0,1388809635677)
Sample(7.0,1388809636677)
Sample(8.0,1388809637677)
Sample(9.0,1388809638677)
Sample(10.0,1388809639677)

Results:

result.foreach(println)
SampleAggregate(1388809630677,1388810529677,Sample(1.0,1388809630677),Sample(900.0,1388810529677))
SampleAggregate(1388810530677,1388811429677,Sample(901.0,1388810530677),Sample(1800.0,1388811429677))
SampleAggregate(1388811430677,1388812329677,Sample(1801.0,1388811430677),Sample(2700.0,1388812329677))
SampleAggregate(1388812330677,1388813229677,Sample(2701.0,1388812330677),Sample(3600.0,1388813229677))
SampleAggregate(1388813230677,1388814129677,Sample(3601.0,1388813230677),Sample(4500.0,1388814129677))
SampleAggregate(1388814130677,1388815029677,Sample(4501.0,1388814130677),Sample(5400.0,1388815029677))
SampleAggregate(1388815030677,1388815929677,Sample(5401.0,1388815030677),Sample(6300.0,1388815929677))

We can pass a function into span that will define intervals (hours or days). This can also be transformed into a Stream as it's being read from a file.

Upvotes: 1

Related Questions