16,551
社区成员
发帖
与我相关
我的任务
分享关于文件读写与虚拟缓存!
现有一个项目,要频繁的读取3个百兆的文件(每个都一百多兆).
为了提高效率
我创建了索引,记录数据文件里数据的内容.
然后在程序启动时,打开3个数据文件的句柄.
读取时根据索引使用seek来位移然后读取.
退出时关闭
文件操作使用的是FILE相关的fopen fclose fseek fread
现在的问题是.
在长时间运行后，两三个小时以后,程序的虚拟内存就会疯狂上升不下降.
想问一下有什么好的解决办法.
先谢谢大家了
为了提高效率
我创建了索引,记录数据文件里数据的内容.
然后在程序启动时,打开3个数据文件的句柄.
读取时根据索引使用seek来位移然后读取.
退出时关闭
文件操作使用的是FILE相关的fopen fclose fseek fread
现在的问题是.
在长时间运行后，两三个小时以后,程序的虚拟内存就会疯狂上升不下降.
想问一下有什么好的解决办法.
先谢谢大家了
...全文
请发表友善的回复…
发表回复
double2 2005-06-08
- 打赏
- 举报
to qrMs:现在已经用了CreateFile CreateFileMapping MapViewOfFile来做文件的读取了
qrlvls 2005-06-08
- 打赏
- 举报
用 CreateFile 的方式会吗？
double2 2005-06-07
- 打赏
- 举报
to keiy:谢谢了.已经用过，还用相关api手工查找过了
都没有什么帮助
都没有什么帮助
柯本 2005-06-07
- 打赏
- 举报
用BoundsChecker查内存泄漏.下载地址
http://soft.ttdown.com/SoftView/SoftView_25686.html
http://soft.ttdown.com/SoftView/SoftView_25686.html
double2 2005-06-06
- 打赏
- 举报
谢谢诸位,我用映像后,情况有所改善,但是时间再长一些又出问题,而且现在在开始时,内存变的会增加比较快
yuanxiaojin 2005-06-06
- 打赏
- 举报
学习中！^_^！
double2 2005-06-06
- 打赏
- 举报
刚刚吧vector都替换成*,把resize替换成new,把clear替换成delete都没有什么改善.
大家再帮忙想想吧
大家再帮忙想想吧
double2 2005-06-06
- 打赏
- 举报
to zkxz:
这个程序里面很少使用句柄.
到是会经常用int的vector,不过可以确认每次使用完都clear了
不知道koko1998说的是否正确我先去试一下.
先谢谢诸位的回帖了
这个程序里面很少使用句柄.
到是会经常用int的vector,不过可以确认每次使用完都clear了
不知道koko1998说的是否正确我先去试一下.
先谢谢诸位的回帖了
zkxz 2005-06-06
- 打赏
- 举报
to 楼主，
那还有其它隐性的没有释放？比如不用的Handle需要CloseHandle。
那还有其它隐性的没有释放？比如不用的Handle需要CloseHandle。
koko1998 2005-06-06
- 打赏
- 举报
现在系统中，好像如果你内存将要被耗尽，系统会自动缓存作为内存给应用程序使用。
koko1998 2005-06-06
- 打赏
- 举报
很可能是你使用了大量的数组，Array等等
这些内存系统没有及时回收。
这些内存系统没有及时回收。
double2 2005-06-06
- 打赏
- 举报
to zkxz:
这个可以肯定,new都delete了
这个可以肯定,new都delete了
zkxz 2005-06-06
- 打赏
- 举报
有可能是申请的内存没有及时释放
kingzai 2005-06-05
- 打赏
- 举报
Platform SDK: File Storage
MapViewOfFile
The MapViewOfFile function maps a view of a file into the address space of the calling process.
To specify a suggested base address, use the MapViewOfFileEx function.
LPVOID MapViewOfFile(
HANDLE hFileMappingObject, // handle to file-mapping object
DWORD dwDesiredAccess, // access mode
DWORD dwFileOffsetHigh, // high-order DWORD of offset
DWORD dwFileOffsetLow, // low-order DWORD of offset
SIZE_T dwNumberOfBytesToMap // number of bytes to map
);
Parameters
hFileMappingObject
[in] Handle to an open handle of a file-mapping object. The CreateFileMapping and OpenFileMapping functions return this handle.
dwDesiredAccess
[in] Specifies the type of access to the file view and, therefore, the protection of the pages mapped by the file. This parameter can be one of the following values. Value Meaning
FILE_MAP_WRITE Read/write access. The hFileMappingObject parameter must have been created with PAGE_READWRITE protection. A read/write view of the file is mapped.
FILE_MAP_READ Read-only access. The hFileMappingObject parameter must have been created with PAGE_READWRITE or PAGE_READONLY protection. A read-only view of the file is mapped.
FILE_MAP_ALL_ACCESS Same as FILE_MAP_WRITE.
FILE_MAP_COPY Copy on write access. If you create the map with PAGE_WRITECOPY and the view with FILE_MAP_COPY, you will receive a view to file. If you write to it, the pages are automatically swappable and the modifications you make will not go to the original data file.
Windows 95/98/Me: You must pass PAGE_WRITECOPY to CreateFileMapping; otherwise, an error will be returned.
If you share the mapping between multiple processes using DuplicateHandle or OpenFileMapping and one process writes to a view, the modification is propagated to the other process. The original file does not change.
Windows NT/2000/XP: There is no restriction as to how the hFileMappingObject parameter must be created. Copy on write is valid for any type of view.
If you share the mapping between multiple processes using DuplicateHandle or OpenFileMapping and one process writes to a view, the modification is not propagated to the other process. The original file does not change.
dwFileOffsetHigh
[in] Specifies the high-order DWORD of the file offset where mapping is to begin.
dwFileOffsetLow
[in] Specifies the low-order DWORD of the file offset where mapping is to begin. The combination of the high and low offsets must specify an offset within the file that matches the system's memory allocation granularity, or the function fails. That is, the offset must be a multiple of the allocation granularity. Use the GetSystemInfo function, which fills in the members of a SYSTEM_INFO structure, to obtain the system's memory allocation granularity.
dwNumberOfBytesToMap
[in] Specifies the number of bytes of the file to map. If dwNumberOfBytesToMap is zero, the entire file is mapped.
Return Values
If the function succeeds, the return value is the starting address of the mapped view.
If the function fails, the return value is NULL. To get extended error information, call GetLastError.
Remarks
Mapping a file makes the specified portion of the file visible in the address space of the calling process.
Multiple views of a file (or a file-mapping object and its mapped file) are said to be "coherent" if they contain identical data at a specified time. This occurs if the file views are derived from the same file-mapping object. A process can duplicate a file-mapping object handle into another process by using the DuplicateHandle function, or another process can open a file-mapping object by name by using the OpenFileMapping function.
A mapped view of a file is not guaranteed to be coherent with a file being accessed by the ReadFile or WriteFile function:
Windows 95/98/Me: MapViewOfFile may require the swapfile to grow. If the swapfile cannot grow, the function fails.
Windows NT/2000/XP: If the file-mapping object is backed by the paging file (hFile is INVALID_HANDLE_VALUE), the paging file must be large enough to hold the entire mapping. If it is not, MapViewOfFile fails.
Note To guard against an access violation, use structured exception handling to protect any code that writes to or reads from a memory mapped view. For more information, see Reading and Writing.
Example Code
For an example, see Creating Named Shared Memory.
Requirements
Windows NT/2000/XP: Included in Windows NT 3.1 and later.
Windows 95/98/Me: Included in Windows 95 and later.
Header: Declared in Winbase.h; include Windows.h.
Library: Use Kernel32.lib.
See Also
File Mapping Overview, File Mapping Functions, CreateFileMapping, DuplicateHandle, GetSystemInfo, MapViewOfFileEx, OpenFileMapping, UnmapViewOfFile, SYSTEM_INFO
Platform SDK Release: August 2001 What did you think of this topic?
Let us know. Order a Platform SDK CD Online
(U.S/Canada) (International)
Requirements
Windows NT/2000/XP: Included in Windows NT 3.1 and later.
Windows 95/98/Me: Included in Windows 95 and later.
Header: Declared in Winbase.h; include Windows.h.
Library: Use Kernel32.lib.
See Also
File Mapping Overview, File Mapping Functions, CreateFileMapping, DuplicateHandle, GetSystemInfo, MapViewOfFileEx, OpenFileMapping, UnmapViewOfFile, SYSTEM_INFO
MapViewOfFile
The MapViewOfFile function maps a view of a file into the address space of the calling process.
To specify a suggested base address, use the MapViewOfFileEx function.
LPVOID MapViewOfFile(
HANDLE hFileMappingObject, // handle to file-mapping object
DWORD dwDesiredAccess, // access mode
DWORD dwFileOffsetHigh, // high-order DWORD of offset
DWORD dwFileOffsetLow, // low-order DWORD of offset
SIZE_T dwNumberOfBytesToMap // number of bytes to map
);
Parameters
hFileMappingObject
[in] Handle to an open handle of a file-mapping object. The CreateFileMapping and OpenFileMapping functions return this handle.
dwDesiredAccess
[in] Specifies the type of access to the file view and, therefore, the protection of the pages mapped by the file. This parameter can be one of the following values. Value Meaning
FILE_MAP_WRITE Read/write access. The hFileMappingObject parameter must have been created with PAGE_READWRITE protection. A read/write view of the file is mapped.
FILE_MAP_READ Read-only access. The hFileMappingObject parameter must have been created with PAGE_READWRITE or PAGE_READONLY protection. A read-only view of the file is mapped.
FILE_MAP_ALL_ACCESS Same as FILE_MAP_WRITE.
FILE_MAP_COPY Copy on write access. If you create the map with PAGE_WRITECOPY and the view with FILE_MAP_COPY, you will receive a view to file. If you write to it, the pages are automatically swappable and the modifications you make will not go to the original data file.
Windows 95/98/Me: You must pass PAGE_WRITECOPY to CreateFileMapping; otherwise, an error will be returned.
If you share the mapping between multiple processes using DuplicateHandle or OpenFileMapping and one process writes to a view, the modification is propagated to the other process. The original file does not change.
Windows NT/2000/XP: There is no restriction as to how the hFileMappingObject parameter must be created. Copy on write is valid for any type of view.
If you share the mapping between multiple processes using DuplicateHandle or OpenFileMapping and one process writes to a view, the modification is not propagated to the other process. The original file does not change.
dwFileOffsetHigh
[in] Specifies the high-order DWORD of the file offset where mapping is to begin.
dwFileOffsetLow
[in] Specifies the low-order DWORD of the file offset where mapping is to begin. The combination of the high and low offsets must specify an offset within the file that matches the system's memory allocation granularity, or the function fails. That is, the offset must be a multiple of the allocation granularity. Use the GetSystemInfo function, which fills in the members of a SYSTEM_INFO structure, to obtain the system's memory allocation granularity.
dwNumberOfBytesToMap
[in] Specifies the number of bytes of the file to map. If dwNumberOfBytesToMap is zero, the entire file is mapped.
Return Values
If the function succeeds, the return value is the starting address of the mapped view.
If the function fails, the return value is NULL. To get extended error information, call GetLastError.
Remarks
Mapping a file makes the specified portion of the file visible in the address space of the calling process.
Multiple views of a file (or a file-mapping object and its mapped file) are said to be "coherent" if they contain identical data at a specified time. This occurs if the file views are derived from the same file-mapping object. A process can duplicate a file-mapping object handle into another process by using the DuplicateHandle function, or another process can open a file-mapping object by name by using the OpenFileMapping function.
A mapped view of a file is not guaranteed to be coherent with a file being accessed by the ReadFile or WriteFile function:
Windows 95/98/Me: MapViewOfFile may require the swapfile to grow. If the swapfile cannot grow, the function fails.
Windows NT/2000/XP: If the file-mapping object is backed by the paging file (hFile is INVALID_HANDLE_VALUE), the paging file must be large enough to hold the entire mapping. If it is not, MapViewOfFile fails.
Note To guard against an access violation, use structured exception handling to protect any code that writes to or reads from a memory mapped view. For more information, see Reading and Writing.
Example Code
For an example, see Creating Named Shared Memory.
Requirements
Windows NT/2000/XP: Included in Windows NT 3.1 and later.
Windows 95/98/Me: Included in Windows 95 and later.
Header: Declared in Winbase.h; include Windows.h.
Library: Use Kernel32.lib.
See Also
File Mapping Overview, File Mapping Functions, CreateFileMapping, DuplicateHandle, GetSystemInfo, MapViewOfFileEx, OpenFileMapping, UnmapViewOfFile, SYSTEM_INFO
Platform SDK Release: August 2001 What did you think of this topic?
Let us know. Order a Platform SDK CD Online
(U.S/Canada) (International)
Requirements
Windows NT/2000/XP: Included in Windows NT 3.1 and later.
Windows 95/98/Me: Included in Windows 95 and later.
Header: Declared in Winbase.h; include Windows.h.
Library: Use Kernel32.lib.
See Also
File Mapping Overview, File Mapping Functions, CreateFileMapping, DuplicateHandle, GetSystemInfo, MapViewOfFileEx, OpenFileMapping, UnmapViewOfFile, SYSTEM_INFO
oyljerry 2005-06-05
- 打赏
- 举报
这么大的文件用内存文件映射比较好
truewill 2005-06-05
- 打赏
- 举报
fopen？我记得有文章说过打开这种大文件用内存影射的方法比较好
ps：系统的虚拟内存应该是不会下降的吧……
ps：系统的虚拟内存应该是不会下降的吧……
jxnczxl 2005-06-05
- 打赏
- 举报
是不是申请的虚拟内存没有正确回收。
