Windows 10 File Cloud / Sync Provider API - TransferData problem

Question

I am building upon the Cloud Mirror Sample and having a similar issue to this one

Here is the test code involved:

        // When the client needs to fetch data from the cloud, this method will be called.
    // The FakeMirrorDataMover class does the actual work of copying files from
    // the "cloud" to the "client" and updating the transfer status along the way.
    void CALLBACK FakeCloudProvider::OnFetchData(
        _In_ CONST CF_CALLBACK_INFO* callbackInfo,
        _In_ CONST CF_CALLBACK_PARAMETERS* callbackParameters)
    {
        //FileCopierWithProgress::CopyFromServerToClient(callbackInfo, callbackParameters, ProviderFolderLocations::GetServerFolder());
    
            const UINT CHUNKSIZE = 48 * 1024 * 1024;
            UINT len;
            LONG64 offset = callbackParameters->FetchData.RequiredFileOffset.QuadPart;
            LONG64 requiredLength = callbackParameters->FetchData.RequiredLength.QuadPart;
            byte *buffer = new byte[CHUNKSIZE];
            FillMemory(buffer, CHUNKSIZE, (byte)0xA5);
    
            while (0 < requiredLength)
            {
                len = requiredLength < CHUNKSIZE ? requiredLength : CHUNKSIZE;
                if (0 != len % 4096)
                    len = 4096 * (len / 4096 + 1);
    
                Placeholders::TransferData(callbackInfo->TransferKey.QuadPart, buffer, offset, len, 0);
    
                requiredLength -= len;
                offset += len;
            }
    
            delete[] buffer;
    }

HRESULT Placeholders::TransferData(
    //_In_ CF_CONNECTION_KEY connectionKey,
    _In_ LONG64 transferKey,
    _In_reads_bytes_opt_(length.QuadPart) LPCVOID transferData,
    _In_ LONG64 startingOffset,
    _In_ LONG64 length,
    _In_ NTSTATUS completionStatus)
{
    CF_OPERATION_INFO opInfo = { 0 };
    CF_OPERATION_PARAMETERS opParams = { 0 };

    opInfo.StructSize = sizeof(opInfo);
    opInfo.Type = CF_OPERATION_TYPE_TRANSFER_DATA;
    opInfo.ConnectionKey = FakeCloudProvider::GetConnectionKey();
    opInfo.TransferKey.QuadPart = transferKey;
    opParams.ParamSize = CF_SIZE_OF_OP_PARAM(TransferData);
    opParams.TransferData.CompletionStatus = completionStatus;
    opParams.TransferData.Buffer = transferData;
    opParams.TransferData.Offset.QuadPart = startingOffset;
    opParams.TransferData.Length.QuadPart = length;

    winrt::check_hresult(CfExecute(&opInfo, &opParams));     
    return S_OK;
}

There is only one placeholder file created inside the sync root folder of ~ 9.3 GB size. After I create this I either double click it or right click and "always keep on this device" - same result.

The result being that after the last call to TransferData the Windows Explorer file progress gets stuck. It will eventually time out, I click on "try again", my breakpoints inside OnFetchData() get hit and file progress remains stuck. If I cancel the transfer then onCancelFetchData() gets called once as it should. Subsequent attempts to download the rest of the file will no longer call onFetchData()

If I comment out the if (0 != len % 4096) len = 4096 * (len / 4096 + 1); part then I am getting the dreaded 0x8007017c the cloud operation is invalid with the rest of the symptoms as above.

I have tried with a smaller ~ 9.3 MB file and it went fine... what else to try?

edit

The ~100MB and ~1GB sizes also working fine

Result of CfExecute calls:

//[...] same output for all previous offsets except for the last call
TransferData method - offset: 1298137088, length: 50331648, syncStatBeforeCall: null, syncStatAfterCall: null, hresult: 0
TransferData method - offset: 1348468736, length: 50331648, syncStatBeforeCall: null, syncStatAfterCall: null, hresult: 0
TransferData method - offset: 1398800384, length: 11265024, syncStatBeforeCall: null, syncStatAfterCall: null, hresult: 8007017c

As far as I can tell null is fine for SyncStatus:

SyncStatus

Note This member is new for Windows 10, version 1803.

The current sync status of the platform.

The platform queries this information upon any failed operations on a cloud file placeholder. If a structure is available, the platform will use the information provided to construct a more meaningful and actionable message to the user. The platform will keep this information on the file until the last handle on it goes away. If null, the platform will clear the previously set sync status, if there is one.

--edit--

HRESULT Placeholders::Create(_In_ PCWSTR destPath,
                          _In_ PCWSTR fileName,
                          _In_ CF_PLACEHOLDER_CREATE_FLAGS flags,
                          _In_ LONG64 fileSize,
                          _In_ DWORD fileAttributes,
                          _In_ LONG64  ftCreationTime,
                          _In_ LONG64  ftLastWriteTime,
                          _In_ LONG64  ftLastAccessTime,
                          _In_ LONG64  ftChangeTime)
{
    CF_PLACEHOLDER_CREATE_INFO cloudEntry;
        
    cloudEntry.FileIdentity = fileName;
    cloudEntry.FileIdentityLength = (DWORD)((wcslen(fileName)+1) * sizeof(WCHAR));
    cloudEntry.RelativeFileName = fileName;
    cloudEntry.Flags = flags;
    cloudEntry.FsMetadata.FileSize.QuadPart = fileSize;
    cloudEntry.FsMetadata.BasicInfo.FileAttributes = fileAttributes;
    cloudEntry.FsMetadata.BasicInfo.CreationTime.QuadPart = ftCreationTime;
    cloudEntry.FsMetadata.BasicInfo.LastWriteTime.QuadPart = ftLastWriteTime;
    cloudEntry.FsMetadata.BasicInfo.LastAccessTime.QuadPart = ftLastAccessTime;
    cloudEntry.FsMetadata.BasicInfo.ChangeTime.QuadPart = ftChangeTime;

    if ((fileAttributes & FILE_ATTRIBUTE_DIRECTORY) != 0)
    {
        cloudEntry.Flags |= CF_PLACEHOLDER_CREATE_FLAG_DISABLE_ON_DEMAND_POPULATION;
        cloudEntry.FsMetadata.FileSize.QuadPart = 0;
        cloudEntry.FileIdentity = nullptr;
    }

    try
    {
        wprintf(L"Creating placeholder for %s\n", fileName);
        winrt::check_hresult(CfCreatePlaceholders(destPath, &cloudEntry, 1, CF_CREATE_FLAG_NONE, NULL));
    }
    catch (...)
    {
        // winrt::to_hresult() will eat the exception if it is a result of winrt::check_hresult,
        // otherwise the exception will get rethrown and this method will crash out as it should
        wprintf(L"Failed to create placeholder for %s with %08x\n", fileName, static_cast<HRESULT>(winrt::to_hresult()));
        // Eating it here lets other files still get a chance. Not worth crashing the sample, but
        // certainly noteworthy for production code
        return static_cast<HRESULT>(winrt::to_hresult());
    }

    return S_OK;
}

Create(_T("C:\\SyncRootDT"), _T("file1"), CF_PLACEHOLDER_CREATE_FLAG_MARK_IN_SYNC,
            10000000000, FILE_ATTRIBUTE_NORMAL, 532657415, 532657415, 532657415, 532657415));

Answer 1

I observe similar behaviour. The hydration of files up to 4Gb works fine, but files over 4Gb always stuck and failed with the 'Cloud operation is invalid' exception. I was able to overcome it with this fix but instead of the CF_CALLBACK_PARAMETERS.FETCHDATA.RequiredLength and OptionalLength, I used a complete file length from CF_CALLBACK_INFO.FileSize. It looks like after a recent Windows update the OptionalLength is always zero. My code is in .NET, but I guess you can easily translate it into C++:

// Use a complete file size in case the file is over 4Gb
if (callbackInfo.FileSize > 0x100000000)
{
    requiredLength = callbackInfo.FileSize;
}