Qt loading deleted/renamed file with Windows 10

Question

I'm seeing some weird behavior when running a test install of my Qt program (tried using qt 5.5.1 and 7.0). I haven't noticed the issue when running in a debug/development environment - but see the issue when installed into "Program Files (x86)".

The issue is: I'm using QDirIterator to find database files within the "QStandardPaths::DataLocation" locations and loading them in via sqlite. The phantom files are located in Program Files (x86)//Library/.ndat What I'm seeing is that files from a previous install (which have been deleted) and ones that have been renamed, then deleted, still show up and are readable in the program. These "phantom" files have been blocking loading of the up-to-date file. It's really strange - I wonder if anyone has seen the issue?

I'm running Windows 10 Home on an SSD-based machine (if it matters). Same issue with Qt 5.5.1 and 5.7. I've replicated it on a different machine with similar configuration.

Any ideas?

Here's a summary of my code:

QStringList standardPaths = QStandardPaths::locateAll(QStandardPaths::DataLocation, "Library", QStandardPaths::LocateDirectory);

QStringList fileFilters;
fileFilters << "*.ndat";

foreach (const QString &dir, standardPaths) {
    QDirIterator iterator (dir, fileFilters);
    while (iterator.hasNext()) {
        const QString &filePath = iterator.next();
        QString databaseName = QFileInfo(filePath).baseName();
        database_->open(filePath, baseName); // my function
    }
}

boolDataManager::open (const QString &filePath, const QString &connectionName) {
    QSqlDatabase    db = QSqlDatabase::addDatabase("QSQLITE", connectionName);
    db.setDatabaseName (filePath);
    if (!db.open()) {
        ERROR(QString("Cannot open database %1 with error %2")
                        .arg(QFileInfo(filePath).baseName())
                        .arg(db.lastError().text()));
        printError();
        return false;
    }
    databaseNames_.append(connectionName);
    return true;
}

This code seems to read in files that don't exist anymore - and strangely, reads contents of old files that have been overwritten in the same spot. It only seems to happen when the files are located within the "Program Files" directory; not in a user directory or what-not.

For example, version 1 of my code had a database called "database.dat" with 10 entries. Version 2 of my install overwrote the file with a file of the same name with 20 entries. Version 2 of my code finds the database.dat file but only reads in the older version with 10 entries - Really weird!

Update

It appears that these "phantom" files are stored at: C:\Users/USERNAME/AppData/Local/VirtualStore/Program Files (x86)/PROGRAM NAME/database.dat

My guess is that I'm opening the file in my program not as read-only so Windows creates a working copy in a user-writable location. Will investigate.

Answer 1

After a bunch of poking around, I found the source of the problem (and solution).

Windows (for backwards compatibility) has a VirtualStore function where if the program tries to write to a unwritable file (based on permissions, e.g. Program Files/Progname/test.txt), it'll copy that file into USER/AppData/Local/VirtualStore/Program Files/.... This new file is not deleted when the program is uninstalled, but looks to the QT program as residing at its original location.

The solution is to open the Sqlite database in read only mode:

    QSqlDatabase    db = QSqlDatabase::addDatabase("QSQLITE", connectionName);
    if (!writable_)
        db.setConnectOptions(QLatin1String("QSQLITE_OPEN_READONLY"));
    db.setDatabaseName (filePath);

Now, I'm running into a problem determining whether the file is writable. This:

    writable_ = fInfo.isWritable();

always returns true, even for files in Program Files. Even when enabling NTFS permissions checking:

    extern Q_CORE_EXPORT int qt_ntfs_permission_lookup;
    qt_ntfs_permission_lookup++; // turn permisssions checking on

the permissions check doesn't work. So now I'm simply doing this:

    QString appDir = gApp->applicationDirPath();
    QString relFilepath = QDir(appDir).relativeFilePath(filePath);
    if (!relFilepath.startsWith(".."))
        writable_ = false;

Database is read only (OK for my application) and no longer creates anything within VirtualStore

Answer 2

The issue is Windows caching - I think - One cant really tell with software that doesn't provide any way to debug it - such as Windows.

I've heard that this solution can also be solved (or at least decreased) by turning on the "Application Experience" Service -- I still run into it from time to time, typically when doing too many Filesystem writes in too short of a time.

I dont know exactly what the cause is -- and I'm pretty sure nobody else does or it would have been fixed.. but as far as I know there is no fix for that (as of this answer's date)

--

Here's my solution to problems like this that works 100% of the time:

To avoid this problem, append the version number to the end of your database's filename each time you compile, in fact apppend it to all your files by using a

#define VERSION 4.22.21 

and then just adding .append(QString("%1").arg(VERSION)); or something.

All you have to really do then is write up some quick code to import all the necessary data from an old database or from wherever, which you should have more or less anyways just from using the database.

Better to avoid situations like that than to try and figure them out -- not to mention you now have a perfect revision system without even trying.

UPDATE

Since theres no use case, no code, and no information about the project, I would have to guess at what you were trying to do -

QList<DataDir*> dataDirectories;
DataDir* new_dataDir;

QStringList standardPaths = QStandardPaths::locateAll(QStandardPaths::DataLocation, "Library", QStandardPaths::LocateDirectory);

QStringList fileFilters;
fileFilters << "*.ndat";

foreach (const QString &dir, standardPaths) {

    QDirIterator iterator (dir, fileFilters);
    while (iterator.hasNext()) {
        const QString &filePath = iterator.next();


        QString databaseName = QFileInfo(filePath).baseName();


         database_->open(filePath, baseName); // my function

         /* Do your database reading or writing and save the results 
          *   into a QHash or something then do this: */

          database_->close(); // super important


    }
}