CODEWITHSUNDEEP
clinuxfuselibfuse
Rockrid3r
Rockrid3r

Reputation: 191

FUSE deadlocks after 10 threads

Basic description

I stumbled upon very strange behavior of FUSE.

Sources

Here is fuse code: (fuse_minimal.c)

#define FUSE_USE_VERSION 36

#include <fuse3/fuse.h>

//...
static int FUSE_read(const char *path, char *dst, size_t size, off_t offset, struct fuse_file_info *fi){
    char log[50];
    snprintf(log, sizeof(log), "[fuse] Reading file %s, size: %lu", path, size);
    printf("%s\n", log);
    
    // thread 'i' opens file '/tmp/fuse/i'.
    // but here @path="/i"
    // so add +1
    int i = atoi(path + 1); // without leading '/'

    printf("[fuse] Thread %d is blocked\n", i);
    sleep(1000000);
    printf("[fuse] Unblocking thread %d\n", i);
    
    return size;
}

// ...
static struct fuse_operations FUSE_ops = {
    // ...
    .read       = FUSE_read,
    // ...
};

int main(int argc, char** argv) {
    return fuse_main(argc, argv, &FUSE_ops, NULL);
}

Here is the code which starts 100 loops: (test.c)

void* func(void* arg) {
    size_t i = (size_t)arg; 
    void* fuse_page_i = (void*)0x40000 + 2 * i * 0x1000; // the mmaped page
    printf("[*] starting thread %lu, touching %p\n", i, fuse_page_i);
    uint64_t val = *(uint64_t*)fuse_page_i; // read happens => FUSE_read
    printf("val: %lu", val);
}   
    
int main() {
    setbuf(stdout, 0);
    puts("[*] ./fuse");
    int pid = fork();
    if (pid == 0) {
        system("mkdir -p /tmp/fuse && ./fuse_minimal -f /tmp/fuse"); // -f for printf output
        exit(0);
    } 
    sleep(5); // wait until fuse_main is set up
    for (size_t  i = 0; i < 100; ++i) {
        char file_target[20];
        snprintf(file_target, sizeof(file_target), "/tmp/fuse/%lu", i);
        int fd = open(file_target, O_CREAT | O_RDWR, 0666);
        if (mmap((void*)0x40000 + 2 * i * 0x1000, 0x1000,
                PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_FIXED, fd, 0)
            == (void*)-1) {
            perror("mmap");
            exit(1);
        }
    }

    pthread_t thread[100];
    for (size_t i = 0; i < 100; ++i) {
        pthread_create(&thread[i], NULL, func, (void*)i); // see the `func` above
    }

    for (size_t i = 0; i < 100; ++i) { 
        pthread_join(thread[i], NULL); // will not join for lloooong time
    }
}

Compilating with

gcc  fuse_minimal.c -lfuse3 -o fuse_minimal -D_FILE_OFFSET_BITS=64
gcc  test.c -o test

The funniest part. Logs

[*] ./fuse
Ignoring invalid max threads value 4294967295 > max (100000).
[fuse] Opened file /0
[fuse] Opened file /1
[fuse] Opened file /2
[fuse] Opened file /3
[fuse] Opened file /4
[fuse] Opened file /5
[fuse] Opened file /6
[fuse] Opened file /7
[fuse] Opened file /8
[fuse] Opened file /9
[fuse] Opened file /10
[fuse] Opened file /11
[fuse] Opened file /12
[fuse] Opened file /13
[fuse] Opened file /14
[fuse] Opened file /15
[fuse] Opened file /16
[fuse] Opened file /17
[fuse] Opened file /18
[fuse] Opened file /19
[fuse] Opened file /20
[fuse] Opened file /21
[fuse] Opened file /22
[fuse] Opened file /23
[fuse] Opened file /24
[fuse] Opened file /25
[fuse] Opened file /26
[fuse] Opened file /27
[fuse] Opened file /28
[fuse] Opened file /29
[fuse] Opened file /30
[fuse] Opened file /31
[fuse] Opened file /32
[fuse] Opened file /33
[fuse] Opened file /34
[fuse] Opened file /35
[fuse] Opened file /36
[fuse] Opened file /37
[fuse] Opened file /38
[fuse] Opened file /39
[fuse] Opened file /40
[fuse] Opened file /41
[fuse] Opened file /42
[fuse] Opened file /43
[fuse] Opened file /44
[fuse] Opened file /45
[fuse] Opened file /46
[fuse] Opened file /47
[fuse] Opened file /48
[fuse] Opened file /49
[fuse] Opened file /50
[fuse] Opened file /51
[fuse] Opened file /52
[fuse] Opened file /53
[fuse] Opened file /54
[fuse] Opened file /55
[fuse] Opened file /56
[fuse] Opened file /57
[fuse] Opened file /58
[fuse] Opened file /59
[fuse] Opened file /60
[fuse] Opened file /61
[fuse] Opened file /62
[fuse] Opened file /63
[fuse] Opened file /64
[fuse] Opened file /65
[fuse] Opened file /66
[fuse] Opened file /67
[fuse] Opened file /68
[fuse] Opened file /69
[fuse] Opened file /70
[fuse] Opened file /71
[fuse] Opened file /72
[fuse] Opened file /73
[fuse] Opened file /74
[fuse] Opened file /75
[fuse] Opened file /76
[fuse] Opened file /77
[fuse] Opened file /78
[fuse] Opened file /79
[fuse] Opened file /80
[fuse] Opened file /81
[fuse] Opened file /82
[fuse] Opened file /83
[fuse] Opened file /84
[fuse] Opened file /85
[fuse] Opened file /86
[fuse] Opened file /87
[fuse] Opened file /88
[fuse] Opened file /89
[fuse] Opened file /90
[fuse] Opened file /91
[fuse] Opened file /92
[fuse] Opened file /93
[fuse] Opened file /94
[fuse] Opened file /95
[fuse] Opened file /96
[fuse] Opened file /97
[fuse] Opened file /98
[fuse] Opened file /99
[*] starting thread 0, touching 0x40000
[*] starting thread 1, touching 0x42000
[*] starting thread 2, touching 0x44000
[fuse] Reading file /0, size: 4096
[fuse] Thread 0 is blocked
[*] starting thread 3, touching 0x46000
[fuse] Reading file /1, size: 4096
[fuse] Thread 1 is blocked
[*] starting thread 4, touching 0x48000
[*] starting thread 5, touching 0x4a000
[*] starting thread 6, touching 0x4c000
[fuse] Reading file /2, size: 4096
[fuse] Thread 2 is blocked
[*] starting thread 7, touching 0x4e000
[*] starting thread 8, touching 0x50000
[fuse] Reading file /3, size: 4096
[fuse] Thread 3 is blocked
[*] starting thread 9, touching 0x52000
[*] starting thread 10, touching 0x54000
[*] starting thread 11, touching 0x56000
[*] starting thread 12, touching 0x58000
[*] starting thread 13, touching 0x5a000
[*] starting thread 14, touching 0x5c000
[*] starting thread 15, touching 0x5e000
[*] starting thread 16, touching 0x60000
[fuse] Reading file /4, size: 4096
[fuse] Thread 4 is blocked
[*] starting thread 17, touching 0x62000
[*] starting thread 18, touching 0x64000
[*] starting thread 19, touching 0x66000
[fuse] Reading file /5, size: 4096
[fuse] Thread 5 is blocked
[*] starting thread 20, touching 0x68000
[fuse] Reading file /6, size: 4096
[fuse] Thread 6 is blocked
[*] starting thread 21, touching 0x6a000
[*] starting thread 22, touching 0x6c000
[fuse] Reading file /7, size: 4096
[fuse] Thread 7 is blocked
[*] starting thread 23, touching 0x6e000
[*] starting thread 24, touching 0x70000
[fuse] Reading file /8, size: 4096
[fuse] Thread 8 is blocked
[*] starting thread 25, touching 0x72000
[fuse] Reading file /9, size: 4096
[*] starting thread 26, touching 0x74000
[fuse] Thread 9 is blocked
[*] starting thread 27, touching 0x76000
[*] starting thread 28, touching 0x78000
[*] starting thread 29, touching 0x7a000
[*] starting thread 30, touching 0x7c000
[*] starting thread 31, touching 0x7e000
[*] starting thread 32, touching 0x80000
[*] starting thread 33, touching 0x82000
[*] starting thread 34, touching 0x84000
[*] starting thread 35, touching 0x86000
[*] starting thread 36, touching 0x88000
[*] starting thread 37, touching 0x8a000
[*] starting thread 38, touching 0x8c000
[*] starting thread 39, touching 0x8e000
[*] starting thread 40, touching 0x90000
[*] starting thread 41, touching 0x92000
[*] starting thread 42, touching 0x94000
[*] starting thread 43, touching 0x96000
[*] starting thread 44, touching 0x98000
[*] starting thread 45, touching 0x9a000
[*] starting thread 46, touching 0x9c000
[*] starting thread 47, touching 0x9e000
[*] starting thread 48, touching 0xa0000
[*] starting thread 49, touching 0xa2000
[*] starting thread 50, touching 0xa4000
[*] starting thread 51, touching 0xa6000
[*] starting thread 52, touching 0xa8000
[*] starting thread 53, touching 0xaa000
[*] starting thread 54, touching 0xac000
[*] starting thread 55, touching 0xae000
[*] starting thread 56, touching 0xb0000
[*] starting thread 57, touching 0xb2000
[*] starting thread 58, touching 0xb4000
[*] starting thread 59, touching 0xb6000
[*] starting thread 60, touching 0xb8000
[*] starting thread 61, touching 0xba000
[*] starting thread 62, touching 0xbc000
[*] starting thread 63, touching 0xbe000
[*] starting thread 64, touching 0xc0000
[*] starting thread 65, touching 0xc2000
[*] starting thread 66, touching 0xc4000
[*] starting thread 67, touching 0xc6000
[*] starting thread 68, touching 0xc8000
[*] starting thread 69, touching 0xca000
[*] starting thread 70, touching 0xcc000
[*] starting thread 71, touching 0xce000
[*] starting thread 72, touching 0xd0000
[*] starting thread 73, touching 0xd2000
[*] starting thread 74, touching 0xd4000
[*] starting thread 75, touching 0xd6000
[*] starting thread 76, touching 0xd8000
[*] starting thread 77, touching 0xda000
[*] starting thread 78, touching 0xdc000
[*] starting thread 79, touching 0xde000
[*] starting thread 80, touching 0xe0000
[*] starting thread 81, touching 0xe2000
[*] starting thread 82, touching 0xe4000
[*] starting thread 83, touching 0xe6000
[*] starting thread 84, touching 0xe8000
[*] starting thread 85, touching 0xea000
[*] starting thread 86, touching 0xec000
[*] starting thread 87, touching 0xee000
[*] starting thread 88, touching 0xf0000
[*] starting thread 89, touching 0xf2000
[*] starting thread 90, touching 0xf4000
[*] starting thread 91, touching 0xf6000
[*] starting thread 92, touching 0xf8000
[*] starting thread 93, touching 0xfa000
[*] starting thread 94, touching 0xfc000
[*] starting thread 95, touching 0xfe000
[*] starting thread 96, touching 0x100000
[*] starting thread 97, touching 0x102000
[*] starting thread 98, touching 0x104000
[*] starting thread 99, touching 0x106000

As you see in logs, we firstly open 100 files. Then we start 100 threads, every one of them tries to touch his own mmaped file. But! Only 10 of them got inside FUSE_read (and block there: sleep(1000000)). All other threads do touch the file, but can't get into FUSE_read. The program is deadlocked at this point. :(

If we make sleep for smaller time like 20s, threads will continue to FUSE_read after 20s. Tested it.

So, from my understanding of the above, it looks like

I really want to block this 100 threads on fuse operation. What are my options?

Upvotes: 2

Views: 88

Answers (0)

Related Questions