CODEWITHSUNDEEP
rubyenumerator
Asad Moosvi
Asad Moosvi

Reputation: 490

How are values actually fetched from an Enumerator object in Ruby?

I'm interested in how values are fetched from an Enumerator object. In the following piece of code, I was expecting the first enum.next call to raise an exception as all values had already been received from enum after the call to enum.to_a.

enum = Enumerator.new do |yielder|
  yielder.yield 1
  yielder.yield 2
  yielder.yield 3
end

p enum.to_a # => [1, 2, 3]

puts enum.next # Expected StopIteration here
puts enum.next
puts enum.next
puts enum.next # => StopIteration exception raised

What's the difference between calling next vs an iterator method like to_a on an instance of Enumerator?

Upvotes: 3

Views: 1624

Answers (2)

Philipp Claßen
Philipp Claßen

Reputation: 43970

Short answer: to_a always iterates over all elements and does not advance the position of the iterator. That is why Enumerator#next will start with the first element even if you have called to_a before. Calling to_a does not modify the enumerator object.

Here are the details:

Terms: internal vs external iteration

When discussing iterators in Ruby, two terms come up:

  1. internal iteration (also called implicit iteration)
  2. external iteration

In your question, enum.to_a is an example for enum being used for internal iteration, while enum.next is an example of external iteration.

External iteration provides more control but is a more low-level operation. Internal iteration is often more elegant. The difference is that external iteration makes the state explicit (the current position), while the internal iteration implicitly applies to all elements.

Internal iteration: to_a

to_a will call Enumerator#each, which iterates over the block according to how this Enumerator was constructed.

That is the critical point. As it does not operate on the internal state (the position) of the enumerator object from which it is called, it does not interfere with the calls to next (the external iteration operation).

External iteration: next

When you create the Enumerator object, its state is initialized to point to the first object. You can modify the internal state by calling next, which will advance the position. Once all elements were consumed, it will raise a StopIteration exception.

Note that the state is only relevant when you are using the enumerator object for external iteration. That explains why you can safely call to_a on an enumerator that already consumed all elements, and it will still return a list of all elements. All internal iteration operations (e.g, each, to_a, map`) do not interfere with the external iteration.

Implementation in Rubinius

I looked at the Rubinius source code to understand how it implemented there. Although it is not a language specification, it should be relatively close to the truth. Entry points:

Note that Enumerator includes Enumerable as a mixin.

Upvotes: 7

jimworm
jimworm

Reputation: 2741

Calling #next moves the internal position forward, while #to_a doesn't consider the internal position at all. Try calling next once, then to_a, then next again to experiment.

https://ruby-doc.org/core-2.4.0/Enumerator.html#method-i-next

https://ruby-doc.org/core-2.4.0/Enumerable.html#method-i-to_a

Upvotes: 1

Related Questions