Potential bug in std::filesystem::remove_all with clang++

Question

DO NOT TRY THIS AT HOME

I am having a weird issue with std::filesystem::remove_all. I have written a program that writes N files to disk in a single directory and then deletes all the files afterward (there is a good reason for this). However, when I use std::filesystem::remove_all I get errors like this:

filesystem error: cannot remove all: Structure needs cleaning [./tmp_storage] [./tmp_storage/2197772]

and the folder is not delete (obviously the call failed) and calling ls after shows that the file system is "damaged":

$ ls tmp_storage/
ls: cannot access 'tmp_storage/2197772': Structure needs cleaning
ls: cannot access 'tmp_storage/5493417': Structure needs cleaning
...

and I have to repair the file system. The fully program looks like this:

#include <fmt/core.h>
#include <CLI/CLI.hpp>

#include <filesystem>
#include <fstream>
#include <string>
#include <exception>

int main(int argc, char** argv)
{
  size_t num_files{64000000};

  CLI::App app("Writes N number of files to dir in file system to check the maximum number of files in a directory");
  app.add_option("-c,--count", num_files, fmt::format("How many files generate [Default: {}]", num_files));
  CLI11_PARSE(app, argc, argv);

  std::string base_path = "./tmp_storage";

  if (!std::filesystem::exists(base_path))
  {
    std::filesystem::create_directory(base_path); 
  }

  size_t i;

  for (i = 1; i <= num_files; ++i)
  {
    std::string file_path = fmt::format("{}/{}", base_path, std::to_string(i));
    std::ofstream out(file_path, std::ios::binary);

    if (out.fail())
    {
      break; 
    }

    try
    {
      out << std::to_string(i); 
    }
    catch(const std::exception& e)
    {
      fmt::print("{}\n", e.what());
    }
  }

  fmt::print("Wrote {} out of {} files\n", i, num_files);

  try
  {
    std::filesystem::remove_all(base_path);
  }
  catch(const std::exception& e)
  {
    fmt::print("{}\n", e.what());
  }
  
  fmt::print("Done\n");
  
  return 0; 
}

Compiled with the following Makefile:

CC = clang++
CXX_FLAGS = -std=c++17
LINK_FLAGS = -lfmt

all:
    $(CC) $(CXX_FLAGS) main.cpp -o main $(LINK_FLAGS)

I have been able to replicate the behavior on Fedora Server 33/34 and Ubuntu with Fedora using XFS and Ubuntu using EXT4 and XFS. Is this a bug in std::filesystem::remov_all or am I doing something wrong?

For Fedora the kernel version is: Linux 5.12.12-300.fc34.x86_64 x86_64 with clang version

clang version 12.0.0 (Fedora 12.0.0-2.fc34)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

Answer 1

I tried to reproduce this on Fedora 34 using this modified program (removing the fmt and cli11 dependencies):

#include <filesystem>
#include <fstream>
#include <string>
#include <exception>

int main(int argc, char** argv)
{
  size_t num_files{64000000};

  if (argc > 1)
    num_files = std::stol(argv[1]);

  std::string base_path = "./tmp_storage";

  try
  {
    if (!std::filesystem::exists(base_path))
    {
      std::filesystem::create_directory(base_path); 
    }

    size_t i;

    for (i = 1; i <= num_files; ++i)
    {
      auto si = std::to_string(i);
      std::string file_path = base_path + '/' + si;
      std::ofstream out(file_path, std::ios::binary);

      if (out.fail())
        throw std::system_error(errno, std::generic_category(), "ofstream failed: " + file_path);

      try
      {
        out << si;
      }
      catch(const std::exception& e)
      {
        std::puts(e.what());
      }
    }

    std::printf("Wrote %zu out of %zu files\n", i - 1, num_files);

    std::filesystem::remove_all(base_path);
  }
  catch(const std::exception& e)
  {
    std::puts(e.what());
  }
  
  std::puts("Done");
  
  return 0; 
}

I can't reproduce the errors in F34, using ext4 or xfs or with the default installation choice of btrfs. I also can't reproduce it on another server using xfs, with clang 13.0.0 and libstdc++-11.2.1 and kernel 5.14.0. This means I'm unable to debug where my std::filesystem implementation corrupts the filesystem, and unable to report it to the kernel team.

I'm not sure whether the code is encountering a kernel bug or if you have faulty hardware. Did you check what the system journal said around the time of the filesystem corruption? Where there any errors from the kernel?

Edit: Also, are you using LVM for your disks? I think all my tests were without LVM.

Answer 2

NOTE: This is not a solution to underlying and operating system problems, but a way to avoid it in C++.

The change we need to make to the original code is "minimal". All changes is made to the try block

 try
  {
    std::filesystem::remove_all(base_path);
  }
  catch(const std::exception& e)
  {
    fmt::print("{}\n", e.what());
  }

and replace: std::filesystem::remove_all(base_path); with sequential deletes.

for (auto& path : std::filesystem::directory_iterator(base_path))
{
    std::filesystem::remove(path);
}

Changing the original code to

#include <fmt/core.h>
#include <CLI/CLI.hpp>

#include <filesystem>
#include <fstream>
#include <string>
#include <exception>

int main(int argc, char** argv)
{
    size_t num_files{64000000};
    
    CLI::App app("Writes N number of files to dir in file system to check the maximum number of files in a directory");
    app.add_option("-c,--count", num_files, fmt::format("How many files generate [Default: {}]", num_files));
    CLI11_PARSE(app, argc, argv);

    std::string base_path = "./tmp_storage";

    if (!std::filesystem::exists(base_path))
    {
        std::filesystem::create_directory(base_path); 
    }

    size_t i;

    for (i = 1; i <= num_files; ++i)
    {
        std::string file_path = fmt::format("{}/{}", base_path, std::to_string(i));
        std::ofstream out(file_path, std::ios::binary);

        if (out.fail())
        {
            break; 
        }

        try
        {
            out << std::to_string(i); 
        }
        catch(const std::exception& e)
        {
            fmt::print("{}\n", e.what());
        }
    }

    fmt::print("Wrote {} out of {} files\n", i, num_files);

    try
    {
        for (auto& path : std::filesystem::directory_iterator(base_path))
        {
            std::filesystem::remove(path); 
        }
    }
    catch(const std::exception& e)
    {
        fmt::print("{}\n", e.what());
    }
  
    fmt::print("Done\n");
  
    return 0; 
}