LightningChart JS - Corruption of LineSeries / Progressive X

Question

I'm running into an interesting problem with LightningChart where it seems to be corrupting or otherwise decimating my data depending on how far it is from the DateOrigin of the chart. My data is 1000 samples per second and I am trying to display 1-2 weeks of data at a time. I am using the ChartXY class, the x-axis type is set to "linear-highPrecision" which should have 1 ms accuracy, which is all I need and I don't need any more, I am creating a LineSeries and it's data pattern is 'ProgressiveX' and regularProgressiveStep: true.

Here's what it looks like when the data is plotted near the DateOrigin.

Here's what it looks like zoomed in on the data near the DateOrigin.

That looks fantastic! And lightning chart is doing exactly what I want!

However, I would like this data to be offset correctly to show it's true absolute time.

Here's what it looks like when I offset this data by 14 days. My code to set the relative offset looks like this.

ds.addArrayY(curve.data,step=1,start=14*24*60*60*1000)

Ok, it looks alright zoomed out, but what if we zoom in?

It's gone haywire! It looks like the X axis values are being coerced to some larger step of the X axis. It gets worse the further that you go out from the DateOrigin. My fear is that this is some built-in behavior of the engine and I am expecting too much, however, it says it has 1ms resolution, so I expect that to be respected.

Here's how I create the chart.

// Create a Line Chart.
const PumpsChart = lightningChart().ChartXY({
    // Set the chart into a div with id, 'target'. 
    // Chart's size will automatically adjust to div's size. 
    theme: Themes.lightGradient,
    container: 'PumpsChart',
    defaultAxisX: {
        type: 'linear-highPrecision'
    }
}).setTitle('') // Set chart title
.setTitleFont(new FontSettings({
    family: waveChartFontFamily,
    size: 20
}))
.setMouseInteractionWheelZoom(false)

axisPumpsChartDateTime = PumpsChart.getDefaultAxisX()
.setTickStrategy(
    AxisTickStrategies.DateTime, 
    (tickStrategy) => tickStrategy.setDateOrigin(waveDateOrigin))

axisPumpsChartPressurePSI = PumpsChart.getDefaultAxisY()
    .setTitle("Pressure (PSI)")
    .setInterval(0,10000,0,true)

Here's how I create the LineSeries

newDataSeries = targetChart.chart.addLineSeries(
                                    { 
                                        yAxis: targetChart.axis,
                                        dataPattern: {
                                            pattern: 'ProgressiveX',
                                            regularProgressiveStep: true,
                                        }
                                     }
                                );

Here's how I add data to the chart:

ds.addArrayY(curve.data,step=1,start=14*24*60*60*1000)

I would prefer not to use the AxisTickStrategies.DateTime over AxisTickStrategies.DateTime for a few reasons, my data spans weeks, 100 hours is too little, I am just fine with millisecond resolution, I don't need more than that, and I need to present my data in relative and not absolute time.

Hopefully there's some parameter that I missing that I can adjust to achieve this.

EDIT

Well, this corruption is also happening with Time tick strategy s well when the data is offset relative to the origin -636 hours. I tried this with and without ProgressiveX set as DataPattern.pattern.

**** EDIT 2 ****

Well, I even tried downSampling to 20 samples per second, and changed this back to AxisTickStrategies.DateTime, it's "squishing" all the points to this magic .25 second interval for some reason.

Answer 1

Could it be 32 bit floating point precision loss? Two weeks in seconds amounts to 1 209 600, and log2(1_209_600) = 20.2 = 21 bits of storage space. The mantissa of 32-bit binary floats is just 23 bits, leaving you with 2 bits for the fractionnal part, which would explain the 0.25 increments.

So if i'm correct, you would need to have the X position precision bumped to 64-bit floats. You're already using "linear-highPrecision" axis mode, though, which would have been my first guess for a solution, so as far as I can tell it doesn't actually increase the data precision to 64 bits, only the axis. Unless there's another solution I've missed, you would probably need to split your data into separate series.

EDIT: I'm not actually sure that's a problem on LightningChart's end, now that I've looked it up. OpenGL ES 3.0 (upon which WebGL 2 relies, and in turn LightningChart) requires that "high-precision" float parameters be stored in binary32 IEEE 754 standard floats.

Official spec, p. 53:

highp floating point values are stored in IEEE 754 single precision floating point format. Mediump and lowp floating point values have minimum range and precision requirements as detailed below and have maximum range and precision as defined by IEEE 754.

So given that information, it seems to be a bug in LCjs caused a WebGL technical limitation.

Answer 2

I tried to produce a reference application in a similar situation - using DateTime ticks with high resolution time data (1000 Hz). The below snippet should open right here in stack overflow, generating a quite large set of test data (might take a while), and using date origin to show it with high zoom range.

Does this work for you? If yes, then maybe there is something off in your application which you could spot with the help of this reference application.

const { lightningChart, AxisTickStrategies } = lcjs
const { createProgressiveTraceGenerator } = xydata

createProgressiveTraceGenerator()
    .setNumberOfPoints(1000 * 60 * 60 * 4)
    .generate()
    .toPromise()
    .then((data) => {
        // Just for generating data set
        const dataStart = Date.now()
        return data.map((p) => ({ x: dataStart - 1000 * 60 * 60 * 4 + p.x, y: p.y }))
    })
    .then((data) => {
        // 1. Offset data X by date origin
        const dateOrigin = new Date()
        const tDateOrigin = dateOrigin.getTime()
        data = data.map((p) => ({ x: p.x - tDateOrigin, y: p.y }))

        const chart = lightningChart().ChartXY({ disableAnimations: true })
        const axisX = chart.getDefaultAxisX().setTickStrategy(AxisTickStrategies.DateTime, (ticks) =>
            ticks
                // 2. Inform Tick Strategy of date origin
                .setDateOrigin(dateOrigin),
        )

        const series = chart.addLineSeries({
            dataPattern: {
                pattern: 'ProgressiveX',
                regularProgressiveStep: true,
            },
        })

        series.add(data)
        chart.addLegendBox().add(chart)
    })

<script src="https://unpkg.com/@arction/lcjs@3.4.0/dist/lcjs.iife.js"></script>
<script src="https://unpkg.com/@arction/xydata@1.2.1/dist/xydata.iife.js"></script>

Looking back at your question, perhaps the original issue was on the tick formatting? When you zoom in this case, it ultimately shows just seconds on each tick rather than a full time. Was this the problem? If yes, then the issue is about formatting axis ticks and I'll have to see what can be done about that.