CODEWITHSUNDEEP
postgresqlcitus
Franklin Dingemans
Franklin Dingemans

Reputation: 23

All distributed queries fail using Citus task tracker executor

I'm attempting to performance-test distributed joins on Citus 5.0. I have a master and two worker nodes, and a few hash distributed tables that behave as expected with the default config. I need to use the task tracker executor to test queries that require repartitioning.

However, After setting citus.task_executor_type to task-tracker, all queries involving distributed tables fail. For example:

postgres=# SET citus.task_executor_type TO "task-tracker";
SET
postgres=# SELECT 1 FROM distrib_mcuser_car LIMIT 1;

ERROR:  failed to execute job 39
DETAIL:  Too many task tracker failures

Setting citus.task_executor_type in postgresql.conf has the same effect.

Is there some other configuration change I'm missing that's necessary to switch the task executor?

EDIT, more info:

All of the tables so far were distributed like:

SELECT master_create_distributed_table('table_name', 'id', 'hash');
SELECT master_create_worker_shards('table_name', 8, 2);

The schema for distrib_mcuser_car is fairly large, so here's a more simple example:

postgres=# \d+ distrib_test_int
                   Table "public.distrib_test_int"
 Column |  Type   | Modifiers | Storage | Stats target | Description
--------+---------+-----------+---------+--------------+-------------
 num    | integer |           | plain   |              |

postgres=# select * from distrib_test_int;
ERROR:  failed to execute job 76
DETAIL:  Too many task tracker failures

Upvotes: 2

Views: 976

Answers (2)

Marco Slot
Marco Slot

Reputation: 471

The task-tracker executor assigns tasks (queries on shards) to a background worker running on the worker node, which connects to localhost to run the task. If your superuser requires a password when connecting to localhost, then the background worker will be unable to connect. This can be resolved by adding a .pgpass file on the worker nodes for connecting to localhost.

Upvotes: 3

Murat Tuncer
Murat Tuncer

Reputation: 146

You can modify authentication settings and let workers connect to master without password checks by changing pg_hba.conf.

Add following line to master pg_conf.hba:

host    all             all             [worker 1 ip]/32            trust
host    all             all             [worker 2 ip]/32            trust

And following lines to for each worker-1 pg_hba.conf:

host    all             all             [master ip]/32              trust
host    all             all             [worker 2 ip]/32            trust

And following to worker-2 pg_hba.conf:

host    all             all             [master ip]/32              trust
host    all             all             [worker 1 ip]/32            trust

This is only intended for testing, DO NOT USE this for production system without taking necessary security precautions.

Upvotes: 2

Related Questions