How to list all 'romtable' modules built-in into currently running NodeMCU firmware?

Question

This question is as in the title, and pretty much that's it.

The firmware certainly knows what it carries inside. Here's the NodeMCU's "welcome" info:

NodeMCU ESP32 built on nodemcu-build.com provided by frightanic.com
        branch: dev-esp32
        commit: fb12af06e7e01f699d68496b80dae481e355adb7
        SSL: false
        modules: adc,can,dac,file,gpio,i2c,ledc,net,node,ow,sigma_delta,spi,time,tmr,touch,uart,wifi
 build 2021-02-01-23-53-37 powered by Lua 5.1.4 on ESP-IDF v3.3-beta1-1391-g9e70825d1 on SDK IDF
lua: cannot open init.lua
>

I'd be more than happy to just get that comma-separated 'modules' string.

Since built-in modules are typically available under their names, as global variables, it is somewhat-similar to "how to list all global variables" in Lua, as I'd expect, everyone uses this on the normal Lua to inspect everything, including built-ins like math. Yet, it isn't the same here. I've checked.

Listing globals from _G doesn't reveal existence of variables like tmr or uart.

> =tmr
romtable: 0x3f406e30

> =uart
romtable: 0x3f40acc8

> for k,v in pairs(_G) do print(k.." : "..tostring(v)) end
module : function: 0x3ffc2fa0
require : function: 0x3ffc2fd4
pairs : function: 0x3ffba950
newproxy : function: 0x3ffc90b4
package : table: 0x3ffba8fc
_G : table: 0x3ffc29c0
_VERSION : Lua 5.1
ipairs : function: 0x3ffba888

> for k,v in pairs(package.preload) do print(k.." : "..tostring(v)) end
((nothing!))

> for k,v in pairs(package.loaded) do print(k.." : "..tostring(v)) end
package
_G

As you can see, tmr and uart names are recognized, however, they are not listed as the contents of _G.

Interestingly, even the math is was not listed as in _G, even though it's clearly available:

> =math
romtable: 0x3f420ef0
> =_G.math
romtable: 0x3f420ef0
> =math.PI
nil
> =math.pi
3.1415926535898

What made me think, ok, what about those tmr and uart? Yup, the same:

> =_G.tmr
romtable: 0x3f406e30
> =_G.uart
romtable: 0x3f40acc8

and here we have it. They aren't listed in the contents of _G, and yet, they are available.

I'm not an expert in Lua nor NodeMCU, I may have missed something, but digging through the docs, trying out various Lua runtime inspection methods, I just can't figure out if/how to list them from _G.

I have a feeling that the _G has two separate meta-methods, one for listing, one for accessing/getting, and that only the latter one is patched up by the firmware to actually provide the modules-by-name, while the former isn't patched to list them, but that's just a guess.

As a side note, I got stuck on that on NodeMCU on ESP32, the dev-esp32 version (as available on https://nodemcu-build.com/), but as I'm wrinting this I tried the same on ESP8288, and the result is the same: math, tmr, uart are available directly, are available via _G, but aren't listed when inspecting _G's contents.

As a side note#2, after hours of searching I've found this video 'ESP8266 NodeMCU - How to know NodeMCU firmware Module info? ', that presents a "NodeMCU_firmware_info.lua", also called "AEW_NodeMCU_info.lua" (probably branding issue), which effectively simply contains a list of name-variable pairs and iterates over all hardcoded references to various modules, trying out each one for a nil value..

Yeah, I know that will work. And that's exactly what I want to avoid. I'd like to read-out the built-in information without having to hardcode or bruteforce the names!

---

EDIT: as suggested by koyaanisqatsi, I checked _G's metatable and __index, and it turned out to be romtable. Here's a dump:

function printtable(t) for k,v in pairs(t) do print(k.." : "..tostring(v)) end end

> printtable(getmetatable(_G))
__index : romtable: 0x3f4218b8

> printtable(getmetatable(_G)['__index'])
assert : lightfunction: 0x40156de8
collectgarbage : lightfunction: 0x40156be8
dofile : lightfunction: 0x40156bac
error : lightfunction: 0x40156b60
gcinfo : lightfunction: 0x40156b44
getfenv : lightfunction: 0x40156b14
getmetatable : lightfunction: 0x40156e8c
loadfile : lightfunction: 0x401570f4
load : lightfunction: 0x40156da4
loadstring : lightfunction: 0x40157120
next : lightfunction: 0x40156aec
pcall : lightfunction: 0x40156600
print : lightfunction: 0x40156a44
rawequal : lightfunction: 0x40156a1c
rawget : lightfunction: 0x401569f8
rawset : lightfunction: 0x4015694c
select : lightfunction: 0x401568f4
setfenv : lightfunction: 0x40156874
setmetatable : lightfunction: 0x4015677c
tonumber : lightfunction: 0x401566e0
tostring : lightfunction: 0x40156cfc
type : lightfunction: 0x401566bc
unpack : lightfunction: 0x40156638
xpcall : lightfunction: 0x401565bc
__metatable : romtable: 0x3f421b28

So, at least some of the built-in functions showed up finally, but sadly, neither of the modules, not even standard math or debug.

---

EDIT: I also checked how does that look like on on of my ESP8266's.

boot msg:

        branch: release
        commit: 64bbf006898109b936fcc09478cbae9d099885a8
        release: 3.0-master_20200910
        release DTS: 202009090323
        SSL: false
        build type: float
        LFS: 0x40000 bytes total capacity
        modules: adc,bit,cron,encoder,file,gpio,gpio_pulse,i2c,net,node,ow,pwm2,rtctime,sigma_delta,sntp,softuart,spi,tmr,wifi
 build 2020-10-10 21:38 powered by Lua 5.1.4 on SDK 3.0.1-dev(fce080e)

and _G's metatable:

printtable(getmetatable(_G)['__index'])
string : table: 0x402710e0
table : table: 0x402704cc
debug : table: 0x402719b0
coroutine : table: 0x40271694
math : table: 0x40270d04
ROM : table: 0x3ffef580
assert : function: 0x402424cc
......
......
net : table: 0x40274e9c
sntp : table: 0x40275268
bit : table: 0x40275398
adc : table: 0x40275494
gpio : table: 0x402756e4
tmr : table: 0x40275880
ow : table: 0x40275b04
softuart : table: 0x40275cd8
cron : table: 0x40275e2c
gpiopulse : table: 0x40276120

so in fact, the build-in modules DO show up here. Now, it's a puzzle, why they don't on ESP32?

Answer 1

I found it! YAY!

Thank you @koyaanisqatsi.
Your suggestion that __index in _G's metatable was correct, and was 100% perfect for NodeMCU 3.0.0 firmware for ESP8266, and almost precisely half of the solution for ESP32 with dev-esp32 fb12af06e7e01f699d68496b80dae481e355adb7 firmware.

On ESP8266 this retrieves all the keys I wanted to see (mixed up with other entries, but that's a minor point):

function printtable(t) for k,v in pairs(t) do print(k.." : "..tostring(v)) end end

printtable(getmetatable(_G)['__index'])

string : table: 0x402710e0
table : table: 0x402704cc
debug : table: 0x402719b0
coroutine : table: 0x40271694
math : table: 0x40270d04
ROM : table: 0x3ffef580
assert : function: 0x402424cc
collectgarbage : function: 0x40242230
dofile : function: 0x402421d0
error : function: 0x40242180
gcinfo : function: 0x40242158
getfenv : function: 0x4024211c
getmetatable : function: 0x40242598
loadfile : function: 0x40242890
load : function: 0x4024246c
loadstring : function: 0x402428c8
next : function: 0x402420e4
pcall : function: 0x40241ac4
print : function: 0x4024203c
rawequal : function: 0x40241ff8
rawget : function: 0x40241fc4
rawset : function: 0x40241f8c
select : function: 0x40241ea4
setfenv : function: 0x40241e18
setmetatable : function: 0x40241ce4
tonumber : function: 0x40241be0
tostring : function: 0x402423a4
type : function: 0x40241bac
unpack : function: 0x40241b0c
xpcall : function: 0x40241a74
pwm2 : table: 0x402723b4
encoder : table: 0x402724b4
rtctime : table: 0x40272660
i2c : table: 0x4027282c
spi : table: 0x40272a30
sigma_delta : table: 0x40272af8
node : table: 0x40273398
pipe : table: 0x402735f8
file : table: 0x40273a20
wifi : table: 0x4027472c
net : table: 0x40274e9c
sntp : table: 0x40275268
bit : table: 0x40275398
adc : table: 0x40275494
gpio : table: 0x402756e4
tmr : table: 0x40275880
ow : table: 0x40275b04
softuart : table: 0x40275cd8
cron : table: 0x40275e2c
gpiopulse : table: 0x40276120

As you can see, built-ins like string, debug are listed first, then all the optional modules like ow or tmr are at the end. Surely, the order cannot be guaranteed by pairs().

Then, on ESP32's firmware this code only revealed some of the built-in functions like getmetatable or tonumber, but none of the modules. But, it turns out that as __index is a romtable and not a function, it can itself have another metatable!

And this time I get what I wanted to inspect, and it's even cleaned up from all other noise, it's all just modules!

function printtable(t) for k,v in pairs(t) do print(k.." : "..tostring(v)) end end

> printtable( getmetatable(getmetatable(_G)['__index'])['__index'] )
time : romtable: 0x3f403458
ow : romtable: 0x3f403600
net : romtable: 0x3f403c60
touch : romtable: 0x3f404a08
node : romtable: 0x3f405218
spi : romtable: 0x3f405570
sigma_delta : romtable: 0x3f405638
ledc : romtable: 0x3f405978
file : romtable: 0x3f405eb0
gpio : romtable: 0x3f406290
can : romtable: 0x3f406578
i2c : romtable: 0x3f406820
wifi : romtable: 0x3f406b78
tmr : romtable: 0x3f406e30
dac : romtable: 0x3f406fc8
adc : romtable: 0x3f407118
uart : romtable: 0x3f40acc8
string : romtable: 0x3f4212a8
table : romtable: 0x3f421ef0
debug : romtable: 0x3f421cc8
coroutine : romtable: 0x3f4217d0
math : romtable: 0x3f420ef0
ROM : romtable: 0x3f422ac8

Of course, if we have first, then second metatable, I had to check if we can go deeper -- and not, it's nil.

> =getmetatable(getmetatable(getmetatable(_G)['__index'])['__index'])
nil

Anyways, thank you very much koyaanisqatsi and Piglet!

---

EDIT: as Marcel pointed out in a comment, on ESP8266 it's actually as simple as

> = node.info('build_config')['modules']
adc,can,dac,file,gpio,i2c,ledc,net,node,ow,sigma_delta,spi,time,tmr,touch,uart,wifi

However, that's not possible on ESP32 at this moment, as node.info() is not implemented yet. Until that moment, my double-getmetatable workaround is quite useful.