CODEWITHSUNDEEP
hawq
S. K
S. K

Reputation: 505

virtual segment memory/core allocation in Apache Hawq

I am trying to tweak below Hawq configurations at session level for a query-

SET hawq_rm_stmt_nvseg = 40;
SET hawq_rm_stmt_vseg_memory = '4gb';

Hawq is running on Yarn resource manager with

Minumum Hawq queue Used capacity 5%
hawq_rm_nvseg_perquery_perseg_limit = 6 
hawq_rm_min_resource_perseg = 4

When running my query i see only 30 containers being launched. Should it not be 40 containers (1 core per virtual segments)? Please help me understand how virtual segments memory or cores are allocated?

Upvotes: 0

Views: 146

Answers (1)

Jon Roberts
Jon Roberts

Reputation: 2106

hawq_rm_stmt_nvseg is a quota limit. By default, this is 0. So setting this to 40 won't increase the number of vsegs but instead, limit it.

hawq_rm_nvseg_perquery_perseg_limit controls how many vsegs can be created and you are using the default of 6. So the number of vsegs should be 6 * number of nodes. If you see 30, then you probably have 5 nodes.

If you are using randomly distributed tables, you can increase hawq_rm_nvseg_perquery_perseg_limit to get more vsegs to work on your query.

If you are using hash distributed tables, you can recreate the table with a larger bucketnum value which will give you more vsegs when you query it.

Upvotes: 1

Related Questions