CODEWITHSUNDEEP
clinuxlinux-kernelpthreadsscheduling
red0ct
red0ct

Reputation: 5055

CPU isolation from Linux scheduler (isolcpus) doesn't work with pthreads

I have Linux with 6 CPU cores, one of which (0st) is isolated from Linux scheduler through the grub config file(/etc/default/grub) with GRUB_CMDLINE_LINUX_DEFAULT="quiet isolcpus=0 nohz_full=1".
I checked isolation with cat /sys/devices/system/cpu/isolated and it's ok.

Here are the full code snippet (compile with gcc -pthread ...):

#define _GNU_SOURCE
#include <stdio.h>
#include <pthread.h>
#include <sched.h>
#include <unistd.h>

void *thread(void *ptr)
{
    for(;;) {
        show_curr_cpu("pthread");
        sleep(2);
    }
}

void show_curr_cpu(char *str)
{
    unsigned int cpu, node;    
    if (getcpu(&cpu, &node) != 0) {
        perror("getcpu");
        return;
    }
    printf("[%s]Current CPU: %u\n", str, cpu);
}

int main(void)
{
    show_curr_cpu("main-bfr");

    cpu_set_t cpuset;
    CPU_ZERO(&cpuset);
    CPU_SET(0, &cpuset); /* 0st CPU core */
    pthread_t current_thread = pthread_self();
    pthread_setaffinity_np(current_thread, sizeof(cpu_set_t), &cpuset);

    show_curr_cpu("main-aft");

    pthread_t thread1;
    int thr = 1;
    if (pthread_create(&thread1, NULL, *thread, (void *) thr)) {
        perror("pthread_create");
        return -1;
    }

    for(;;) {
        show_curr_cpu("main");
        sleep(2);
    }
}

The result of isolation supposes that it

Remove the specified CPUs, as defined by the cpu_number values, from the general kernel SMP balancing and scheduler algroithms

Also:

The isolcpus kernel parameter designates a set of CPUs where the process scheduler will ignore those CPUs. That is to say, the process scheduler will not include those CPUs as targets to put processes on, migrate processes onto or off of, or take a process off a CPU.

But after running my code I see the following strangeness:

[main-bfr]Current CPU: 5
[main-aft]Current CPU: 0
[main]Current CPU: 0
[pthread]Current CPU: 0
[main]Current CPU: 0
[pthread]Current CPU: 0
^C

I.e. pthread runs on the same CPU core. But how does it possible to have two different processes (pthread is a regular process from the Linux kernel point of view) on the same CPU core working well (core sharing) while Linux scheduler does not participate in this?

Upvotes: 6

Views: 1480

Answers (1)

John Bollinger
John Bollinger

Reputation: 181199

But how does it possible to have two different processes (pthread is a regular process from the Linux kernel point of view) on the same CPU core working well (core sharing) while Linux scheduler does not participate in this?

You seem to be reading too much into what it means to isolate CPUs from the kernel's process scheduler.

No thread of any process runs on any core without the kernel scheduling it there. A core being isolated from the process scheduler really means little more than that userspace processes / threads have to have affinity for that core to be scheduled there. It does not mean that the kernel cannot or does not manage the scheduling of all the threads with affinity for that core. Linux does time-sharing just fine.

Upvotes: 2

Related Questions