Yet another module memory loss issue

Question

This problem is driving me nuts. Its consumed way too much time and I could do with some help, please! Basically, Im downloading weather data from my server using Get(). This works well. All I want to do is display it on a 16x2 i2c lcd screen, however I am plagued with 'out of memory' issues. I have tried several modules provided by the community and all seem to suffer the same problems, so Im convinced the problem lays with my code / calling convention. Im currently using module lcd.lua, a slight variation on Vladimir Dronnikov's offering. and calling it in the prescribed manor

<pre>
   i2c.setup(0, 3, 4, i2c.SLOW)
   lcd = dofile("lcd.lua")()
</pre>

the calling lua script is liberally sprinkled with print statements:

<pre>
    lcd.put(lcd:locate(0, 0),"Mast Temp: ")
</pre>

My file structure looks like this

<pre>
    Init.lua -- opportunity to prevent script execution. Calls wifiConnect
    wifiConnect.lua -- connects to wifi and calls dofile(DetServerData())
    GetServerData.lua -- Fetches data and displays on lcd module.
</pre>

When the wifiConnect calls dofile against GetServerData it will usually fall over with 'not enough memory' sometimes the script will run 2 or 3 times before encountering 'out of memory' errors. I have however had it run 8000-9000 times before falling over. (only one or twice.) If I allow wifiConnect to run GetFileData it will stop with

<pre>
    PANIC: unprotected error in call to Lua API (not enough memory)
</pre>

loading GetServerData seems to consume 13816 bytes on the heap.. leaving 18800. surely enough to run. I have tried require(getServerData), slightly different numbers, same result.

GetServerData is as follows

     --  0 DS18B20_temp_C
     --  1 WindSpeed_kmph
     --  2 WindBearing;
     --  3 WindDir
     --  4 BMP_temp
     --  5 BMP_pressure
      -- BMP_altitude --------------REMOVED
     --  6 DHT22_temperature
     --  7 DHT22_humidity
     -- DH22_dewPoint--------------REMOVED
     -- DHT22_heatIndex--------------REMOVED

    local WDPin = 1
    local LCDscreenPin = 2
    local LCDmaxScreen = 4
    local cycleCount=0
    local WDogLED = false
    local dataTable={} ; i=1
    local connClosed = true
    local LCDpageID = 1

    function GetData()
        if connClosed == true then
            local conn=net.createConnection(net.TCP, 0)
            print("Open connection...")
            WatchDog()
            if conn == nil then
                print("Connection error")
            else
                conn:connect(80,'31.220.16.114')
                connClosed = false
                tmr.alarm(4,3000,0,ResetConn)
                conn:send("GET /test.txt? HTTP/1.1\r\n")
                conn:send("Host:   theoldschool.esy.es\r\n")
                conn:send("Accept: */*\r\n")
                conn:send("User-Agent: Mozilla/4.0 (compatible; esp8266 Lua; Windows NT 5.1)\r\n")
                conn:send("\r\n")
                conn:on("disconnection", function(conn)
                connClosed = true
                WatchDog()
                print("Disconnected...")
                end) -- on:"disconection"
               conn:on("receive",function(conn,payload)
                   cleanData = extractWeatherData(payload)
                   DbugGarbageCollect()
                   fillDataTable(cleanData,'\r\n')
                   for k,v in pairs(dataTable) do print(k,v) end
                       cycleCount = cycleCount +1
                   print(cycleCount)
                   end)-- on recieve
              end-- if conn == nil
       end -- if connClosed
    end--GetData

    function ResetConn()
        print("Stuck")
        connClosed = true
    end -- ResetConn

    function fillDataTable(inputstr, sep)
        local i=0
        for str in string.gmatch(inputstr, "([^"..sep.."]+)") do
            dataTable[i] = str
            i = i + 1
        end--for
     end--splitCmd

    function stripMarkers(str, chrs)
       local s = str:gsub("["..chrs:gsub("%W","%%%1").."]", '')
       return s
    end

    function extractWeatherData (payload)
        local startChar = string.find(payload, '%')
        local s = string.sub(payload, startChar,endChar)
        s = stripMarkers(s, "")
        return s
    end -- extractWeatherData

    function WatchDog()
        if WDogLED == false then
            WDogLED = true
            gpio.write(WDPin, gpio.HIGH)
        else
            WDogLED = false
            gpio.write(WDPin, gpio.LOW)
        end --if
    end--WatchDog

    function DbugGarbageCollect()
        local before = collectgarbage("count")
        collectgarbage()
        print(before - collectgarbage("count"))
        print(node.heap())
    end --DbugGarbageCollect

    function LCDdisplay()
        lcd.clear()
        if LCDpageID == 1 then
            lcd.put(lcd:locate(0, 0),"Mast Temp: ")
            lcd.put(lcd:locate(1, 0),"Shade Tmp: ")
        elseif LCDpageID == 2 then
            lcd.put(lcd:locate(0, 0),"Wind Dir: ")
            lcd.put(lcd:locate(1,0),"Wind Knts: ")
        elseif LCDpageID == 3 then
            lcd.put(lcd:locate(0, 0),"Pressure: ")
            lcd.put(lcd:locate(1, 0),"BMP Temp: ")
        elseif LCDpageID == 4 then
            lcd.put(lcd:locate(0, 0),"DHT Temp: ")
            lcd.put(lcd:locate(1, 0),"Humidity: ")
        else
            lcd.put(lcd:locate(0, 0),"SCREEN ERROR 1")
        end --if
        --updateLCDDisplay()
         collectgarbage()
    end --LCDdisplay

    function updateLCDDisplay()
        if LCDpageID == 1 then
            lcd.put(lcd:locate(0, 11),dataTable[0])
           -- LCDScreenOP.lcdprint(dataTable[0],1,11)
           -- LCDScreenOP.lcdprint(dataTable[7],2,11)
        elseif LCDpageID == 2 then
            --LCDScreenOP.lcdprint(dataTable[2],1,10)
            -- LCDScreenOP.lcdprint(dataTable[3],1,14)
            -- LCDScreenOP.lcdprint(dataTable[1],2,11)
        elseif LCDpageID == 3 then
            -- LCDScreenOP.lcdprint(dataTable[5],1,10)
            -- LCDScreenOP.lcdprint(dataTable[4],2,10)
        elseif LCDpageID == 4 then
            --LCDScreenOP.lcdprint(dataTable[6],1,10)
            -- LCDScreenOP.lcdprint(dataTable[7],2,10)
        else
            --  LCDScreenOP.cls()
            --  LCDScreenOP.cursor(0)
            --  LCDScreenOP.lcdprint("U/D ERROR",1,0)
        end --if
    --   package.loaded.LCDScreenOP = nil
         DbugGarbageCollect()
    end -- updateDisplay

    function LCDScreenChange(level)
        LCDpageID = LCDpageID + 1
        if LCDpageID == LCDmaxScreen + 1 then LCDpageID = 1 end
        LCDdisplay()
    end-- buttonpress

    --============================ CODE ==============================
    i2c.setup(0, 3, 4, i2c.SLOW)
    lcd = dofile("lcd.lua")()
    print ("here")
    gpio.mode(WDPin, gpio.OUTPUT)
    gpio.write(WDPin, gpio.HIGH)
    gpio.mode(LCDscreenPin, gpio.INPUT, gpio.PULLUP)
    DbugGarbageCollect()
    gpio.trig(LCDscreenPin, "down",LCDScreenChange)
    tmr.alarm(2,1500,1,GetData)

and a screen grab of the esplorer window looks like this when loading GetServerData and allowing it to run

<pre>
   abort = true
        startup aborted
        =node.heap()
        >**32616**
        file.remove("GetServerData.lua");
        file.open("GetServerData.lua","w+");
        w = file.writeline

    -- Above code in here !!

        file.close();
        dofile("GetServerData.lua");
        >not enough memory

        dofile('GetServerData.lua')
        not enough memory
        =node.heap()
        >**18800**
        dofile('GetServerData.lua')
        >not enough memory
</pre>

ANY help would be gratefully received and any other info that may help will be happily supplied

Many Thanks Philip

Answer 1

That's quite a lot of code to comb through. It'd be a more helpful if you provided a Minimal, Complete, and Verifiable example which will reproduce the problem (-> Minimal).

A quick glance revealed 2.5 issues but there are likely some more.

Closed upvalues

conn:on("receive",function(conn,payload) leaks because the callback parameter shouldn't be conn but something else instead. See https://github.com/nodemcu/nodemcu-firmware/issues/194#issuecomment-184817667 or https://stackoverflow.com/a/37379426/131929

Sending is asynchronous

conn:send shouldn't be called in quick succession because each of those calls is asynchronous and thus the invocation order is not guaranteed. See the net.socket:send documentation for details and a nice example.

Consider http module

Using http.get may help you reduce the complexity in your code.