如何像迅雷那样迅速建立大尺寸文件

[失散糖]

Windows系统下，磁盘剩余空间不必考虑的情况，如何迅速建立文件呢？我不需要系统好心帮忙把文件里的各个字节或是扇区清零，只要能够迅速让文件把需要占用的空间占好即可，文件里面是什么乱码都无所谓。迅雷下载大型文件的时候就是这样的，新建一个任务，然后立刻暂停任务，查看任务文件里面的东西，会发现里面全是乱码，应该就是磁盘占用的扇区没有执行清零操作的结果。
什么？SetFilePointer + SetEndOfFile？我试过这个，虽然可以一瞬间建立好文件，而且看起来好像全都清零好了，但实际上这个文件是假的，我往这个文件末尾写东西的时候，还是会有很长时间的磁盘清零操作。有没有什么办法实现上述的事？我打算用这种方式建立大文件，然后在这个大文件上玩FATFS之类的东西。

[Sandrer]

#include "stdafx.h"
#include <tchar.h>
 
int main()
{
    HANDLE hFile;
    HANDLE hMap;
    PDWORD pView;
    ULONGLONG ullOffset;
    int i = 0;
    DWORD dwTimer;
    WCHAR szOutput[128];
 
    dwTimer = GetTickCount();
    hFile = CreateFile(L"E:\\Test.bin", GENERIC_WRITE | GENERIC_READ, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
    if (hFile != INVALID_HANDLE_VALUE)
    {
        hMap = CreateFileMapping(hFile, NULL, PAGE_READWRITE, 2, 0, NULL);
        if (hMap != NULL)
        {
            ullOffset = (2i64 << 32) - 65536;
            pView = (PDWORD)MapViewOfFile(hMap, FILE_MAP_WRITE, ullOffset >> 32, ullOffset & 0xffffffff, 65536);
            for (i = 0; i < 65536 / sizeof(DWORD); i++)
                *(pView + i) = i;

            UnmapViewOfFile(pView);
        }

        CloseHandle(hMap);
        CloseHandle(hFile);
    }
 
    swprintf_s(szOutput, 128, L"Timer: %d\n", GetTickCount() - dwTimer);
    OutputDebugStringW(szOutput);
 
    return 0;
}

这段代码除了创建文件映射外，还在文件末尾 65536 个字节中写入数据 用时 16 毫秒，我想不明白你所说的等待很久是有多久

    hFile = CreateFile(L"E:\\Test.bin", GENERIC_WRITE | GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
    if (hFile != INVALID_HANDLE_VALUE)
    {
        hMap = CreateFileMapping(hFile, NULL, PAGE_READWRITE, 0, 0, NULL);
        if (hMap != NULL)
        {
            ullOffset = (2i64 << 32) - 65536;
            pView = (PDWORD)MapViewOfFile(hMap, FILE_MAP_WRITE, ullOffset >> 32, ullOffset & 0xFFFFFFFF, 65536);

            UnmapViewOfFile(pView);
        }

        CloseHandle(hMap);
        CloseHandle(hFile);
    }

这段代码是用来检查数据是否成功写入的，在 UnmapViewOfFile(pView); 设断点然后查看 pView 的内存，数据也是成功写入了的

[Sandrer]

引用 7 楼 sugar13 的回复:

@zhao4zhong1 我在意的不是文件的大小，哪怕只是1个字节大小的文件，只要系统帮忙把这一个字节占用的扇区给我分配出来就好了，哪怕这文件的这一个字节的内容是未定义的内容，乱码什么的都无所谓。 提到大文件，是说这种想法通常都是用来建立大文件来的，小文件也就不在意系统好心帮忙的清零操作耗时了。 提到稀疏文件，是说用SetFilePointer + SetEndOfFile或者CreateFileMapping的方式创建的文件应该属于稀疏文件。

正常读写大文件，还是用文件映射来操作的 普通的文件读写操作在系统内核是需要复制两次，才能从文件到用户内存，或者从用户内存到文件 而文件映射则省了一次复制过程，所以比正常文件读写api要快 而且系统内部的粘贴板、本机命名管道、本机网络通讯、窗口消息的传递等等这些，在内核也是以文件映射的方式来操作

[失散糖]

@zhao4zhong1 迅雷是用 SetFilePointer + SetEndOfFile + SetFileValidData的方式建立大文件的，其中SetFileValidData又是通过调用GetModuleHandleW + GetProcAddress动态加载的，但是我调用SetFileValidData的时候，函数总是返回FALSE，错误码1314，意为客户端没有所需的特权。。 重新监视，发现迅雷打开文件之前，通过LookupPrivilegeValue + AdjustTokenPrivileges的方式，大概是在设置权限或是令牌什么的。请问这俩函数要怎么用呢？

[失散糖]

@Sandrer 创建文件之后，我希望能够用fopen系列函数，普普通通的操作文件。至于耗时操作，基本上都是几十秒到几分钟，这个耗时并不是把要写的内容复制给系统什么的耗时，而是说系统把稀疏文件转化成普通文件时，给文件占用的扇区进行清零的耗时。因为测试程序卡在了fwrite里面，所以任务管理器都没有办法把那个耗时的测试程序强制终止。

[赵4老师]

大文件是大文件 稀疏文件是稀疏文件 楼主怎么死不改悔呢？！ 5楼我贴的内容楼主不看是吧！

[赵4老师]

引用 10 楼 sugar13 的回复:

@zhao4zhong1 迅雷是用 SetFilePointer + SetEndOfFile + SetFileValidData的方式建立大文件的，其中SetFileValidData又是通过调用GetModuleHandleW + GetProcAddress动态加载的，但是我调用SetFileValidData的时候，函数总是返回FALSE，错误码1314，意为客户端没有所需的特权。。 重新监视，发现迅雷打开文件之前，通过LookupPrivilegeValue + AdjustTokenPrivileges的方式，大概是在设置权限或是令牌什么的。请问这俩函数要怎么用呢？

谦虚使人进步！

bool AdjustPrivileges() {
    HANDLE hToken;
    TOKEN_PRIVILEGES tp;
    TOKEN_PRIVILEGES oldtp;
    DWORD dwSize=sizeof(TOKEN_PRIVILEGES);
    LUID luid;

    if (!OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, &hToken)) {
        if (GetLastError()==ERROR_CALL_NOT_IMPLEMENTED) return true;
        else return false;
    }
    if (!LookupPrivilegeValue(NULL, SE_DEBUG_NAME, &luid)) {
        CloseHandle(hToken);
        return false;
    }
    ZeroMemory(&tp, sizeof(tp));
    tp.PrivilegeCount=1;
    tp.Privileges[0].Luid=luid;
    tp.Privileges[0].Attributes=SE_PRIVILEGE_ENABLED;
    /* Adjust Token Privileges */
    if (!AdjustTokenPrivileges(hToken, FALSE, &tp, sizeof(TOKEN_PRIVILEGES), &oldtp, &dwSize)) {
        CloseHandle(hToken);
        return false;
    }
    // close handles
    CloseHandle(hToken);
    return true;
}

[赵4老师]

SetFileValidData Function Sets the valid data length of the specified file. Important The file cannot be on a network drive, or be compressed, sparse, or transacted. The SetFileValidData function sets the logical end of a file. To set the size of a file, use the SetEndOfFile function. The physical file size is also referred to as the end of the file. BOOL WINAPI SetFileValidData( __in HANDLE hFile, __in LONGLONG ValidDataLength ); Parameters hFile A handle to the file. The file must have been opened with the GENERIC_WRITE access right, and the SE_MANAGE_VOLUME_NAME privilege enabled. The file cannot be a network file, or be compressed, sparse, or transacted. For more information, see File Security and Access Rights. ValidDataLength The new valid data length. This parameter must be a positive value that is greater than the current valid data length, but less than the current file size. Return Value If the function succeeds, the return value is nonzero. If the function fails, the return value is 0 (zero). To get extended error information, call GetLastError. Remarks Typically, the SetFileValidData function is used by system-level applications on their own private data. Not all file systems use valid data length. Some file systems can track multiple valid data ranges. In general, most applications will never need to call this function. A caller must have the SE_MANAGE_VOLUME_NAME privilege enabled when opening a file initially. Applications should only call SetFileValidData on files that restrict access to those entities that have SE_MANAGE_VOLUME_NAME access. The application must ensure that the unwritten ranges of the file are never exposed, or security issues can result. For more information, see Running with Special Privileges. Each file stream has the following: File size: the size of the data in a file, to the byte. Allocation size: the size of the space that is allocated for a file on a disk, which is always an even multiple of the cluster size. Valid data length: the length of the data in a file that is actually written, to the byte. This value is always less than or equal to the file size. The SetFileValidData function allows you to avoid zero-ing data when writing non-sequentially to a file. The function makes the data in the file valid without writing to the file. You can use the SetFileValidData function to create large files where performance is important. Call this function to skip the time that it takes to fill a file with 0 (zeroes) when the file is created or extended. Simply set the valid data length. Requirements Client Requires Windows Vista or Windows XP. Server Requires Windows Server 2008 or Windows Server 2003. Header Declared in WinBase.h; include Windows.h. Library Use Kernel32.lib. DLL Requires Kernel32.dll. See Also File Management Functions SetEndOfFile Send comments about this topic to Microsoft Build date: 8/15/2007

[失散糖]

@zhao4zhong1 以前我在FAT32文件系统，用手动破坏文件分配表的方式，实现过上述的功能，能够快速建立3.99G的大文件，文件内容是乱码，但是破坏力太大，得修复文件系统才行…… 我现在希望的是通过“合法”的系统调用完成这个功能，不必考虑文件系统类型，不必考虑文件路径中可能含有的用例如mklink建立的目录软链接。 AdjustPrivileges()直接用好像不行，仍然是错误码1314，令牌的句柄和文件的句柄是如何关联起来的呢？

[jiht594]

引用 15 楼 sugar13 的回复:

@zhao4zhong1 以前我在FAT32文件系统，用手动破坏文件分配表的方式，实现过上述的功能，能够快速建立3.99G的大文件，文件内容是乱码，但是破坏力太大，得修复文件系统才行…… 我现在希望的是通过“合法”的系统调用完成这个功能，不必考虑文件系统类型，不必考虑文件路径中可能含有的用例如mklink建立的目录软链接。 AdjustPrivileges()直接用好像不行，仍然是错误码1314，令牌的句柄和文件的句柄是如何关联起来的呢？

你用的是SE_MANAGE_VOLUME_NAME权限吗

[赵4老师]

安装比如MSDN2008 循ms-help://MS.VSCC.v90/MS.MSDNQTR.v90.chs/fileio/fs/setfilevaliddata.htm中的链接看且搜相关关键词。

[赵4老师]

http://www.baigoogledu.com/s.php?hl=zh-CN&q=SetFileValidData

[Sandrer]

用内存映射, 它也会帮你把文件里面的数据清零, 但速度飞快

#include "stdafx.h"
#include <tchar.h>

int main()
{
    HANDLE hFile;
    HANDLE hMap;
    DWORD dwTimer;
    WCHAR szOutput[128];

    dwTimer = GetTickCount();
    hFile = CreateFile(L"E:\\Test.bin", GENERIC_WRITE | GENERIC_READ, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
    if (hFile != INVALID_HANDLE_VALUE)
    {
        hMap = CreateFileMapping(hFile, NULL, PAGE_READWRITE, 2, 0, NULL);
        CloseHandle(hMap);
        CloseHandle(hFile);
    }

    swprintf_s(szOutput, 128, L"Timer: %d\n", GetTickCount() - dwTimer);
    OutputDebugStringW(szOutput);

	return 0;
}

上面的代码, 建立一个 8GB 的文件, 用时为 0

[失散糖]

完成了，测试代码如下： （姑且算是测试，所以并没有加严格的错误处理） （当然也没有注释，因为这是抄迅雷的思路，各个API的调用顺序都一样的，其中的原理完全不知道）

#include <stdio.h>
#include <windows.h>

static void open_undefined( const char *file_name, __int64 file_size )
{
    HMODULE module = ::GetModuleHandle ("kernel32");

    typedef BOOL (WINAPI *sfvd) ( HANDLE hFile, LONGLONG ValidDataLength);
    sfvd SetFileValidData = (sfvd) ::GetProcAddress (module, "SetFileValidData");

    HANDLE token = 0;
    ::OpenProcessToken (INVALID_HANDLE_VALUE, TOKEN_ALL_ACCESS, &token);

    // SE_MANAGE_VOLUME_NAME = "SeManageVolumePrivilege"
    TOKEN_PRIVILEGES tp = { 0 };
    ::LookupPrivilegeValue (0, "SeManageVolumePrivilege", &tp.Privileges->Luid);

    tp.PrivilegeCount = 1;
    tp.Privileges->Attributes = SE_PRIVILEGE_ENABLED;
    ::AdjustTokenPrivileges (token, 0, &tp, sizeof (tp), 0, 0);

    HANDLE file = ::CreateFile (file_name, GENERIC_WRITE, 0, 0, CREATE_NEW, 0, 0);

    if (file != INVALID_HANDLE_VALUE)
    {
        LONG size_lo = (LONG) file_size;
        LONG size_hi = (LONG) (file_size >> 32);
        ::SetFilePointer (file, size_lo, &size_hi, FILE_BEGIN);
        ::SetEndOfFile (file);
        SetFileValidData (file, file_size);
        ::CloseHandle (file);
    }

    ::CloseHandle (token);

    return;
}

extern "C" int __cdecl _fseeki64( FILE *stream, __int64 offset, int origin);

int main()
{
    const char *file_name = "undefined.flat";
    const __int64 file_size = 0x123456789;
    ::remove (file_name);
    ::open_undefined (file_name, file_size);

    if (FILE *stream = ::fopen (file_name, "rb+"))
    {
        ::fseek (stream, 0x1FE, SEEK_SET);
        ::fwrite ("\xAA\x55", 2, 1, stream);
        ::_fseeki64 (stream, -1, SEEK_END);
        ::fwrite ("\r", 1, 1, stream);
        ::fclose (stream);
    }

    return 0;
}

在Win7下NTFS分区中的运行效果，秒间完成，创建的文件，内容全是乱码，估计是之前删除过的文件的残渣。 在FAT32分区中，把文件尺寸改成小于4G，同样可以秒间完成，同样内容全是乱码。

[失散糖]

引用 2 楼 Sandrer 的回复:

用内存映射, 它也会帮你把文件里面的数据清零, 但速度飞快

#include "stdafx.h"
#include <tchar.h>

int main()
{
    HANDLE hFile;
    HANDLE hMap;
    DWORD dwTimer;
    WCHAR szOutput[128];

    dwTimer = GetTickCount();
    hFile = CreateFile(L"E:\\Test.bin", GENERIC_WRITE | GENERIC_READ, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
    if (hFile != INVALID_HANDLE_VALUE)
    {
        hMap = CreateFileMapping(hFile, NULL, PAGE_READWRITE, 2, 0, NULL);
        CloseHandle(hMap);
        CloseHandle(hFile);
    }

    swprintf_s(szOutput, 128, L"Timer: %d\n", GetTickCount() - dwTimer);
    OutputDebugStringW(szOutput);

	return 0;
}

上面的代码, 建立一个 8GB 的文件, 用时为 0

一样的

extern "C" int __cdecl _fseeki64(FILE *, __int64, int);

我在的时候，仍然很耗时

    if (FILE *stream = ::fopen ("E:/Test.bin", "rb+"))
    {
        ::_fseeki64 (stream, 0x1FFFFFFFF, SEEK_SET);
        ::fwrite ("!", 1, 1, stream);
        ::fclose (stream);
    }

经历过一次耗时的更新之后，再次尝试文件结尾处修改文件内容的时候，就是瞬间完成了。 也就是说，这个方法建立的文件仍然是假的，经过耗时的更新之后，文件才变成真的。

[失散糖]

这种文件，是叫稀疏文件来着？

[赵4老师]

大文件是大文件 稀疏文件是稀疏文件

Sparse File Operations
[This is preliminary documentation and subject to change.] 

You can determine whether a file system supports sparse files by calling the GetVolumeInformation function and examining the FILE_SUPPORTS_SPARSE_FILES bit flag.

Most applications are not aware of sparse-files and will not create sparse files. The fact that an application is reading a sparse file is transparent to the application. An application that is aware of sparse-files should determine whether its data set is suitable to be kept in a sparse file. After that determination is made, the application must explicitly declare a file as sparse, using the FSCTL_SET_SPARSE operation.

After an application has set a file to be sparse, the application can use the FSCTL_SET_ZERO_DATA operation to set a region of the file to zero. In addition, the application can use the FSCTL_QUERY_ALLOCATED_RANGES operation to speed searches for nonzero data in the sparse file.

When you perform a write operation (with a function or operation other than FSCTL_SET_ZERO_DATA) whose data consists nothing but zeros, zeros will be written to the disk for the entire length of the write. To zero out a range of the file and maintain sparseness, use FSCTL_SET_ZERO_DATA.

A sparse aware application may also set an existing file to be sparse. If an application sets an existing file to be sparse, it should then scan the file for regions which contain zeros, and use FSCTL_SET_ZERO_DATA to reset those regions, and possibly deallocate some physical disk storage. An application upgraded to sparse file awareness should perform this conversion.

When you perform a read operation from a zeroed-out portion of a sparse file may not read from the hard drive. Instead, the system recognizes that the portion of the file to be read contains zeros, and it returns a buffer full of zeros without actually reading from the disk.

As with any other file, the system can write data to or read data from any position in a sparse file. Nonzero data being written to a previously zeroed portion of the file may result in allocation of disk space. Zeros being written over nonzero data (with FSCTL_SET_ZERO_DATA) may result in a deallocation of disk space.

Note  It is up to the application to maintain sparseness by writing zeros with FSCTL_SET_ZERO_DATA.

Note  Defragmenting tools which handle compressed files on NTFS file systems will correctly handle sparse files on NTFS 5.0 volumes.

楼主何不使用 Process Monitor 实时监视文件系统、注册表、进程、线程和 DLL 活动。 http://www.microsoft.com/china/technet/sysinternals/utilities/processmonitor.mspx 或 WinAPIOverridehttp://jacquelin.potier.free.fr/winapioverride32/ 监视迅雷建立大文件时都调用了哪些Win API呢？

[赵4老师]

SetEndOfFile The SetEndOfFile function moves the end-of-file (EOF) position for the specified file to the current position of the file pointer. BOOL SetEndOfFile( HANDLE hFile // handle of file whose EOF is to be set ); Parameters hFile Handle to the file to have its EOF position moved. The file handle must have been created with GENERIC_WRITE access to the file. Return Values If the function succeeds, the return value is nonzero. If the function fails, the return value is zero. To get extended error information, call GetLastError. Remarks This function can be used to truncate or extend a file. If the file is extended, the contents of the file between the old EOF position and the new position are not defined. If you called CreateFileMapping to create a file-mapping object for hFile, you must first call UnmapViewOfFile to unmap all views and call CloseHandle to close the file-mapping object before you can call SetEndOfFile. QuickInfo Windows NT: Requires version 3.1 or later. Windows: Requires Windows 95 or later. Windows CE: Requires version 1.0 or later. Header: Declared in winbase.h. Import Library: Use kernel32.lib. See Also File I/O Overview, File Functions, CloseHandle, CreateFile, CreateFileMapping, UnmapViewOfFile

[jiht594]

是不是 内存文件映射啊

[失散糖]

顺带一提，我在测试的时候都是关掉用户账户控制的，所以默认就有管理员权限，可以成功；当我启用用户账户控制后，楼上的代码还是失败的，这时需要以管理员身份运行才能成功。我记得使用迅雷的时候不需要什么用管理员身份的，我才想到，我监视的不是迅雷的主程序，因为监视过发现里面没有创建文件之类的调用；我监视的是迅雷的服务，名叫XLServicePlatform，大概服务默认就有管理员权限，或者更高的权限吧，所以迅雷能够成功建立那种文件。

[赵4老师]

查MSDN是Windows程序员必须掌握的技能之一。