How "self" gets to access the key of a local table?

Question

I am new to Lua, I am studying this video. At 38:56, a code is presented as below:

Sequence = {}
function Sequence:new()
  local new_seq = {last_num = 0}
  self.__index = self
  return setmetatable(new_seq, self)
  end

function Sequence:next()
  print(self.last_num)
  end

My understanding is that self is equivalent to Sequence, and self is set as a metatable of new_seq, and this metatable's __index is also self.

And last_num is one key of table new_seq but not one key of self, how come in the definition of next function you can write self.last_num as treating last_num is one key of self?

Moreover, before calling setmetatable, there is self.__index = self, I thought only metatable has __index as a special key, but before calling setmetatable, Sequence is just a regular table and it is not a metatable yet, how come it has __index?

Answer 1

Short version:

My understanding is that self is equivalent to Sequence…

self is an implicit argument resulting from the method-style function definition. Within the function, it will refer to whatever value gets passed in as the first argument to that function.

(The rest of your questions around self arise from that same confusion.)

I thought only metatable has __index as a special key…

A metatable is just a table. __index is just a key like any other, you can define a field with that name on any table. Only when a lookup on a table fails and Lua notices that this table has an attached metatable, the metatable's field named __index has a special meaning – because that's where Lua will look for a handler.

__index containing a table is just another special case of a handler (because it's so common), __index = some_other_table is roughly equivalent to __index = function( table, key ) return some_other_table[key] end – i.e. "go look over there in some_other_table if table[key] was empty". (It may help to use the long version and print something from there if you have trouble following what happens.)

---

Long version, de-sugaring the code and walking through the details:

A definition function foo:bar( ... ) is the same as function foo.bar( self, ... ) (the name self is automatically chosen, roughly like this in other languages). Additionally, function foo.bar( ... ) is the same as foo.bar = function( ... ). Which means the above code is the same as…

Sequence = {}
Sequence.new = function( self )
  local new_seq = { last_num = 0 }
  self.__index = self
  return setmetatable( new_seq, self )
end

Sequence.next = function( self )
  print( self.last_num )
end

…which is equivalent to…

Sequence = {
  new = function( self )
    local new_seq = { last_num = 0 }
    self.__index = self
    return setmetatable( new_seq, self )
  end,
  next = function( self )
    print( self.last_num )
  end,
}

So, in essence, what this defines is a table that contains two functions, each taking a single parameter. The second of the two functions, next, is pretty simple: it just prints the content of the field last_num of whatever table it's passed (using the name self to refer to it).

Now, just as for definitions, there's some :-syntax sugar for calls. A call foo:bar( ... ) translates to foo.bar( foo, ... ), so when you have some_sequence and say some_sequence:next( ), what happens is a call some_sequence.next( some_sequence ) – the :-syntax for definitions introduces an extra hidden parameter and the :-syntax for calls fills in that extra parameter. In this way, the function that you're treating as a method has access to the table that you're treating as an object and everything works out nicely.

The new function is a bit more involved -- I'll rewrite it into yet another equivalent form to make it easier to read:

function Sequence.new( self )
  self.__index = self
  return setmetatable( { last_num = 0 }, self )
end

So for whatever table gets passed in, it assigns that table to the field __index of that same table and returns a new table with that old table set as the metatable. (Yes, this thing is confusing… don't worry, just keep reading.) To see why and how this works, here's an example:

If you say some_sequence = Sequence:new( ), you'll have the following structure:

some_sequence = { last_num = 0 } -- metatable:-> Sequence
Sequence = { new = (func...), next = (func...), __index = Sequence }

Now, when you say some_sequence:next( ), this translates to the call some_sequence.next( some_sequence ). But some_sequence doesn't have a field next! Because some_sequence has a metatable, Lua goes and looks at that – in this case, the metatable is Sequence. As a lookup (or "index") operation "failed" (it would have returned nil), Lua looks for a handler in the metatable's field __index, finds a table (Sequence again) and re-tries the lookup on that one instead (finding the next function we defined).

Which means in this case we could have equivalently written Sequence.next( some_sequence ) (but in general you don't want to – or can't – manually resolve these references). As described above, next just prints the value of the field last_num of the table it received -- in this case it got some_sequence. Again, everything works out nicely.

---

Some more remarks (and yet another example):

For an introductory example, the code is much more mind-bending and brittle than necessary. Here's yet another version (that's not identical and actually behaves differently, but should be easier to understand):

Sequence = { }
Sequence.__index = Sequence
function Sequence.new( )
    return setmetatable( { last_num = 0 }, Sequence )
end
function Sequence:next( )
    print( self.last_num )
end

Both the version that you have and this version will print 0 when you run the following:

some_sequence = Sequence:new( )
some_sequence:next( )

(I've described above what happens under the hood when you do this for your code, compare and try to figure out what happens with my version before reading on.)

This will also print 0 for both versions:

sequences = { [0] = Sequence }
for i = 1, 10 do
    local current = sequences[#sequences]
    sequences[#sequences+1] = current:new( )
end
local last = sequences[#sequences]
last:next( )

What happens under the hood differs significantly for both versions. This is what sequences will look like for your code:

sequences[0] = Sequence -- with __index = Sequence
sequences[1] = { last_num = 0, __index = sequences[1] } -- metatable:->Sequence
sequences[2] = { last_num = 0, __index = sequences[2] } -- metatable:->sequences[1]
sequences[3] = { last_num = 0, __index = sequences[3] } -- metatable:->sequences[2]
...

and this is what it will look like with my version:

sequences[0] = Sequence -- __index = Sequence, as set at the start
sequences[1] = { last_num = 0 } -- metatable:->Sequence
sequences[2] = { last_num = 0 } -- metatable:->Sequence
sequences[3] = { last_num = 0 } -- metatable:->Sequence
...

(If you'd instead say sequences[#sequences+1] = Sequence:new( ) in the loop above, your code would also produce this.)

With my version, the call last:next( ) fails to find next, looks at the metatable (Sequence), finds an __index field (again, Sequence) and finds next, then proceeds to call it as described above.

With your version, the call last:next( ) fails to find next, looks at the metatable (sequences[9]), finds an __index field (sequences[9]), fails to find next and therefore looks at the metatable (of sequences[9], which is sequences[8]), finds an __index field (sequences[8]), fails to find next and therefore looks at the metatable ... (until we reach sequences[1]) ... fails to find next, looks at the metatable (Sequence), finds an __index field (Sequence) and finally finds next, then proceeds with the call. (This is why I said it's quite hard to follow...)

The code that you have implements prototype-based OOP, with all the pros and cons. As you've seen, the lookup traverses the whole chain, which means that you could define a function sequences[5].next to do something else and subsequently sequences[5] through sequences[10] would find that other function. This can be really useful – no need for all the boilerplate to define a new class to change some functionality, just adjust one object and use it like a class. (This can also be annoying if you accidentally do this.)

My version implements something a bit closer to the class-based OOP seen in many other languages. (You can't accidentally override methods for more than one object at once.) What both of these (and many other approaches to OOP in Lua) have in common is that defining a field of an object with the same name as a method will hide that method and make it inaccessible. (If you define some_sequence.next, saying some_sequence:next( ) or some_sequence.next( some_sequence ) will immediately find the next you defined and Lua won't bother to look at the metatable and so on.)