Continuation on: "How do dmapped domains in chapel language get actually mapped onto?"

Question

Cray, sorry for again rising for clarity of answer! I got few more questions about domain-maps after this answer. I will highly appreciate and be very thankful if you clear my doubts about domain-maps.
I hope, I have ordered the questions sequentially.

1.) What are domain-maps? - A domain-map defines a mapping from a global array indexes of domains and arrays to a set of Locales in the cluster.

I have summarized what I have understood from the research paper and other ppts, which may be potentially wrong. Please feel free to correct the answer.

const Domain = {1..8,1..8} dmapped Block( {1..8,1..8} )

Here {1..8,1..8} be the indexspace( domain ), that is distributed to locales using a Block-distribution domain-map with a boundingBox = {1..8,1..8}
From the Block domain-map constructor,

proc block( boundingBox:              domain,
            targetlocales:[] locale = Locales,
            datapartasks            = ...,
            dataparmingranularity   = ...
            )

The Block domain-map only wants to know about the boundingBoX, targetlocales and datapar*'-s and there is no need for a domain, here in this case {1..8,1..8}. I find it difficult to get things correct, due to many interfaces for creating domain-maps within chapel itself, where some interfaces hide some info from user.

So my question is: Did the Block domain-map create instances on targetlocales, which holds the local index sets such as {1..2,3..4} on locale 1, {1..2,1..2} on locale 2 ( where these numbers are just examples, so as to illustrate the mapping process ) ?
In the previous answer, Dr. Brad Chamberlain has mentioned that

"dmapped Block clause() creates instances on the target locales. Each instance of the Block domain map class stores the boundingbox and set of target locales"

I didn't find meaning from it :(
In the whole, please, explain how domain-maps, domains and arrays are working co-operatively. I research some steps, but all misses some kind information I needed to fully understand domain maps.

In this presentation in slide No:34, the local instance domain maps and domains store only the index space, nothing special.

2. In the previous answer, Dr. Brad Chamberlain has also mentioned that

" a given domain map implementation may be very space efficient and minimal, or it can allocate everything on every locale redundantly, as its author thinks best",

in this context what does "allocate everything on every locale redundantly" actually mean? Whether storing the whole array on each locality?

3. In PGAS, the Global instance of a domain-map, domain, array is visible across all locales ?. And I also hope that each query to them takes place through the global instances.

I kindly request you to determine the required interfaces for domain-map, as you mention in the documentation.
I will highly appreciate and be thankful, if I got some explanation on this.
Thank you very much.

Answer 1

1.) What are domain maps ?

A domain map is a Chapel class whose purpose is to describe how domains—and any arrays declared over those domains—are implemented. Practically speaking, they specify:

• how to implement domains declared in terms of that domain map, including how to store them in memory and how to implement methods that the compiler may require (like iteration, membership tests, etc.)
• how to implement arrays declared in terms of that domain, including how to store them in memory and how to implement methods that the compiler may require (like iteration, random access, etc.)

The block domain map only wants to know about the boundingBox, targetlocales, and datapar's, and there is no need for domain, here in this case {1..8,1..8}.

That's correct and this can often be a point of confusion when learning the Block distribution. Let's use a slightly artificial example to make the point:

const D = {1..4, 1..4} dmapped Block({1..8, 1..8});

This declaration says that the plane of 2D indices should be distributed to the locales by blocking the bounding box {1..8, 1..8} between the locales as evenly as possible. If I'm running on 4 locales, locale 0 will own {1..4, 1..4} of the bounding box which implies it will own {-inf..4, -inf..4} in the 2D plane (where, inf is meant to mean "infinity" and practically speaking, -inf will be represented by a small integer). Similarly, locale 3 will own {5..8, 5..8} of the bounding box or {5..inf, 5..inf} from the 2D plane.

As you note, the {1..8, 1..8} above is only an argument to the Block domain map and thus has no bearing on D's value, only its implementation. Because D's value is {1..4, 1..4} it is owned completely by locale 0 since it fits completely within that locale's portion of the 2D plane. If instead it were {2..5, 2..5}, locale 0 would own {2..4, 2..4}, locale 1 would own {2..4, 5..5}, locale 2 would own {5..5, 2..4} and locale 3 would own {5..5, 5..5}.

Because a Block distribution is parameterized by this bounding box domain, this often leads to confusion about why there are two domains used in a declaration like the one above, especially since they are often identical in practice. This has nothing to do with domain maps inherently, and everything to do with the fact that the Block domain map's constructor expects a domain as one of its arguments. In contrast, the Cyclic and BlockCyclic domain maps don't take domain arguments, and are often simpler to understand as a result.

Did the Block domain map creates instances on targetlocales which holds the local index sets...

Yes, in practice, the Block domain map creates an instance of a LocBlock class which stores each locale's local index set—e.g., {-inf..4, inf..4} on locale 0, {-inf..4, 5..inf} on locale 1, etc. for my previous example.

Each instance of the Block domain map class stores the boundingbox and set of target locales

Yes, each locale also stores a copy of the Block domain map class which stores all of the key arguments which parameterize the domain map so that the distribution it defines can be reasoned about by each locale in isolation.

In this presentation in slide no : 34,the local instance domain maps and domains store only the index space,nothing special.

That's correct, the role of the local Block domain map class is to store the portion of the 2D plane owned by that locale (locale #4 or L4 in the slide). Similarly, the role of the local Block domain class is to store the portion of the domain's index set owned by that locale. These are not special, but they are important for defining the distribution and the distributed domain, respectively.

2.In the previous answer,Dr.Brad has also mention that " a given domain map implementation may be very space efficient and minimal, or it can allocate everything on every locale redundantly, as its author thinks best", in this context what "allocate everything on every locale redundantly" actually means ? whether storing the whole array on each locality ?

Yes, by "allocate everything on every locale redundantly" I meant each locale stores the entire array. This is probably not particularly scalable, but the point is that the language and domain map framework say nothing about how arrays are stored as long as they support the expected interface. So the author of a domain map could do this if they chose to.

3. In PGAS, the Global instance of domain map,domain,array is visible across all locales ?. And I also hope that each query to them takes place through the global instances.

You are correct that the global instances of the classes are visible across all locales due to Chapel's PGAS (Partitioned Global Address Space) model. That said, since communication between domains is expensive and these objects tend to change their fields only rarely, in practice we tend to replicate them across locales. On the slide 34 that you referred to the tag "(logically)" is meant to refer to this: There is one conceptual copy of each object, but in practice we tend to replicate them across locales (the author of a domain map may or may not do this, as they wish).

I kindly request you to determine the required interfaces for domain map as you mention in the documentation.

The current documentation on domain map interfaces is available here. The current sources for the Block domain map which implement the 6 descriptors ({global, local} x {domain map, domain, array}) can be found here.