Numerous Hidden Files show on SdFat library ls()

Question

My goal is to be able to navigate through a file system on an SD card and pick out certain file types and display them to the user. When I execute sd.ls(LS_R) on the SdFat library, it shows lots of hidden files. I can deal with most of them fine, but some of them give me headaches. Since the library uses the 8.3 naming convention, it truncates the file/folder names that are too long and replaces it with a "~". This is a problem because then, I cannot distinguish between files/folders that are visible and files/folders that are hidden. Are there any known ways to solve this issue?

Here's my code:

#include <SdFat.h>

const uint8_t chipSelect = 10;

SdFat sd;
SdFile file;

void setup() 
{
    Serial.begin(9600);
    while (!Serial) {} // wait for Leonardo
    delay(1000);

    if (!sd.begin(chipSelect, SPI_HALF_SPEED)) sd.initErrorHalt();

    sd.ls(LS_R);
    while(1);
}

void loop() {}

Here is my "visible" file system:

Folder1/
    test3.txt
    test4.txt
Folder2/
BearsOutside/
test1.txt
test2.txt
LongFilename.txt

And here is the output:

FOLDER1/
  TEST4.TXT
  TEST3.TXT
TEST2.TXT
~1.TRA
TEST1.TXT
TRASHE~1/
SPOTLI~1/
  STORE-V2/
    F8D581~1/
      PSID.DB
      TM~1.SNO
      TM~1.LIO
      LIO~1.CRE
      TMP.CAB
      CA~1.CRE
      INDEXS~1
      ~1.IND
      ~~2.IND
      ~~~3.IND
      ~~~~4.IND
      ~~~~~5.IND
      ~~~~~~34.IND
      ~~~~~~37.IND
      ~~~~~~40.IND
      ~~~~~~43.IND
      ~~~~~~46.IND
      ~~~~~~48.IND
      ~1.DIR
      LIVE~~~4.IND
      LIVE~~2.IND
      LIVE~~~3.IND
      LIVE~~~5.IND
      LIVE~~66.IND
      LIVE~~69.IND
      LIVE~~73.IND
      LIVE~1.SHA
      LIVE~~79.IND
      LIVE~1.DIR
      LIVE0D~1.SHA
      STORE.DB
      STOR~1.DB
      REVERS~1
      TMPSPO~1.STA
      PERMST~1
      STORE_~1
      JOURNA~1.LIV/
      JOURNA~2.LIV/
        RETIRE.3
      JOURNA~3.LIV/
        RETIRE.4
      JOURNA~4.LIV/
      JOURNA~1.ASS/
      JOURNA~2.ASS/
      JOURNA~1.HEA/
      JOURNA~1.MIG/
      JOURNA~2.MIG/
      JOURNA~1
      JOURNA~1.SCA/
        RETIRE.11
      REVERS~1.SHA
      ~1.SHA
      SHUTDO~1
      JOURNA~1.REP/
      CA~1.MOD
      LIVE~155.IND
      LIVE~158.IND
      0DIREC~1.SHA
      ~~~~~166.SHA
      LIVE~169.IND
      LIVE~172.IND
      LIVE~175.IND
      LIVE~178.IND
      LIVE~181.IND
      LIVE~184.IND
      LIVE~1.IND
      LIVE~190.IND
      LIVE~194.SHA
      STOR~1.UPD
      REVERS~1.UPD
      LIVE~202.IND
      TMPSPO~1.LOC
      LIVE~208.IND
      LIVE~211.IND
      LIVE~215.IND
      LIVE~218.SHA
      LIVE~~2.DIR
      LIVE1D~1.SHA
      LIVE~264.SHA
      LIVE~267.IND
      LIVE~270.IND
      LIVE~274.IND
      LIVE~277.IND
      LIVE~~~3.DIR
      LIVE~~2.SHA
      LIVE~~~3.SHA
      LIVE~~~4.SHA
      LIVE~~~5.SHA
      LIVE~296.SHA
      LIVE~300.SHA
      LIVE2D~1.SHA
      LIVE~308.SHA
      LIVE~327.IND
  STORE-V1/
    VOLUME~1.PLI
  VOLUME~1.PLI
FOLDER2/
BEARSO~1/
LONGFI~1.TXT

So my issue is, how do I distinguish between BEARSO~1/ [BearsOutside], which is not hidden, and SPOTLI~1/, which IS hidden?

Answer 1

This is a very old question, but it comes up high in google search so I'll provide some updated information:

As of the time of this writing (Fall 2020) a major upgrade to SdFat exists. SdFat-2.X

It has turned out to be very easy to integrate and very stable.

SdFat-2.0 includes robust support for ExFat volumes, long filenames, and preallocation of file clusters with a mechanism to guarantee non-blocking writes to the SD card. This solves the long-standing problem with occasional blocking long write latency that used to interfere with data-logging on SD cards. For ExFat volumes, the data size calculation for final truncation of the preallocated file space is handled automatically.

SdFat-2.0 supports multiple cards even if they are running on different busses or different bus types. I have a device with three SD cards, one addressed by SDIO and two running on different SPI busses, and I can copy files around without difficulty.

AFAIK there are only two incompatibilities with the older arduino <sd.h> library on which so many other libraries take a dependency:

SDFat-2.0 uses a new syntax for begin(), allowing you to specify how your SD card is wired and how it should be addressed.

SdFat-2.0 doesn't support the older file.name() function because that was set up for 8.3 filenames. Instead you get the long filename by passing a const char* that points to somewhere the filename can be stored. You need the full filename in order to open a file, but if you are OK to stick with 8.3 filenames then you could just create a wrapper that implements a simple parameterless begin() along with the old name() function. That would make everything completely compatible with sd.h and avoid any #ifdef patching of libraries.

That's really the only compatibility issue I've found after multiple years of use.

You can find the github project here: github.com/greiman/SdFat

Answer 2

These files are created automatically by the Mac OSX system... Is it an option for you not to plug the SD card on any Mac? This would solve your problem... ;-)

You could also try to remove them with the Terminal on Mac.

Answer 3

unfortunately some of the primitive functions are not directly available with the

SdFat sd;

object. Hence precluding the use of the "ls" function. The following code should accomplish what you want. It passes the sd objects current file to the pointer a directory entry object of pointer p. which then can have its attributes tested to see if they are hidden or other.

Note there is a method to test if a directory entry is long name. However, I do not believe the entry is caching the long name itself.

The below does compile. And 99.5 of it is from working code that I use. I have added the attribute and long file name detection, so it should work.

void ListFiles2(uint8_t flags) {
  // This code is just copied from SdFile.cpp in the SDFat library
  // and tweaked to print to the serial output in html!
  dir_t p;

  sd.vwd()->rewind();
  Serial.println();
  while (sd.vwd()->readDir(&p) > 0) {
    // done if past last used entry
    if (p.name[0] == DIR_NAME_FREE) break;

    // skip deleted entry and entries for . and  ..
    if (p.name[0] == DIR_NAME_DELETED || p.name[0] == '.') continue;

    // only list subdirectories and files
    if (!DIR_IS_FILE_OR_SUBDIR(&p)) continue;

    if ((p.attributes & DIR_ATT_HIDDEN) != DIR_ATT_HIDDEN) continue;

    // print any indent spaces
    Serial.print(F("  "));
    for (uint8_t i = 0; i < 11; i++) {
      if (p.name[i] == ' ') continue;
      if (i == 8) {
        Serial.print('.');
      }
      Serial.print((char)p.name[i]);
    }
    Serial.print(F(" "));

    // print file name with possible blank fill
    for (uint8_t i = 0; i < 11; i++) {
      if (p.name[i] == ' ') continue;
      if (i == 8) {
        Serial.print('.');
      }
      Serial.print((char)p.name[i]);
      if (DIR_IS_LONG_NAME(&p)) {
        Serial.print(F(" long fn"));
      }
    }

    if (DIR_IS_SUBDIR(&p)) {
      Serial.print('/');
    }

    // print modify date/time if requested
    if (flags & LS_DATE) {
      sd.vwd()->printFatDate(p.lastWriteDate);
      Serial.print(' ');
      sd.vwd()->printFatTime(p.lastWriteTime);
    }
    // print size if requested
    if (!DIR_IS_SUBDIR(&p) && (flags & LS_SIZE)) {
      Serial.print(' ');
      Serial.print(p.fileSize);
    }
    Serial.println();
  }
  Serial.println();
}