69,370
社区成员
发帖
与我相关
我的任务
分享请问：我想把一个几百兆上G的文件调入内存…………
我想把一个上百兆上吉的数据文件调入内存，通过fopen()打开，然后读取纪录，通过链表（一个节点一个记录）存储在内存中，中间不调用windown API函数，不知可不可以，不知还需要对内存进行特殊的控制吗？
...全文
请发表友善的回复…
发表回复
jj520 2003-09-05
- 打赏
- 举报
不现实
sakurar 2003-09-05
- 打赏
- 举报
回复人： lyr311(老刘) ( ) 信誉：100 2003-08-20 21:46:00 得分:0
我认为不一定能行，如果超过2G的话，现在除了大型机外，工作站上的软件好多也只能操作2G的文件，大型的数学软件Matlab也只能操作2G的文件，并且是其6.0以后的版本才可以的。这是因为好多机器都是32位的，大型机、并行机有64位甚至更多的，32位机理论上能操作的文件大小也就是2^(32-1)≈2G啊！
不能同意这种看法。32位计算机不是说就一定只能处理32数据。呵呵，不然我们常用的__int64是啥。就说身边的软件吧，你试试WinRAR，我常用它压几个G的文件（都超过2G），速度并不比小的慢（按时间/体积的比例）。还有一些十六进制编辑器，1G多的文件，它只读它让你看到的那几十K字节。
感觉只有分块处理，一下子载入不现实，也没必要。
我认为不一定能行，如果超过2G的话，现在除了大型机外，工作站上的软件好多也只能操作2G的文件，大型的数学软件Matlab也只能操作2G的文件，并且是其6.0以后的版本才可以的。这是因为好多机器都是32位的，大型机、并行机有64位甚至更多的，32位机理论上能操作的文件大小也就是2^(32-1)≈2G啊！
不能同意这种看法。32位计算机不是说就一定只能处理32数据。呵呵，不然我们常用的__int64是啥。就说身边的软件吧，你试试WinRAR，我常用它压几个G的文件（都超过2G），速度并不比小的慢（按时间/体积的比例）。还有一些十六进制编辑器，1G多的文件，它只读它让你看到的那几十K字节。
感觉只有分块处理，一下子载入不现实，也没必要。
cndragons 2003-09-05
- 打赏
- 举报
foen()打开文件没有限制，完全可以，不用特殊处理,但是读数据慢，没有必要都放到内存里面啊；其次，链表存储这些记录有什么用，如果是顺序读取，直接从文件读就行了，如果是查找，就应该采用树，象平衡二叉树查找非常快的！
jrhjrh 2003-08-25
- 打赏
- 举报
把虚拟内存改成10G 不行了吗，哈哈
huanghelang 2003-08-25
- 打赏
- 举报
我对内存映射的概念不是很清楚，那位大哥可以给我解释一下，同时告诉我如何内存映射？鄙人不胜感激！
huanghelang 2003-08-24
- 打赏
- 举报
将数据文件读入内存再处理，其速度当然快了。问题是如何做？
liuty2006 2003-08-24
- 打赏
- 举报
mk
dwenj 2003-08-24
- 打赏
- 举报
同意。
映射会比较好。
映射会比较好。
huylghost 2003-08-24
- 打赏
- 举报
MARK
nonocast 2003-08-24
- 打赏
- 举报
内存映射
以100M为单位
以100M为单位
caomuyong1 2003-08-24
- 打赏
- 举报
兄弟，用文件映象吧。
ironic 2003-08-23
- 打赏
- 举报
文件映射只是将文件映射到虚存，I/O性能...
努力学习的小白小白小白 2003-08-21
- 打赏
- 举报
不太现实吧。
你的内存那么大？？
等一会，你就CMemoryException * 了
况且读那么大有什么用处阿
你的内存那么大？？
等一会，你就CMemoryException * 了
况且读那么大有什么用处阿
fireseed 2003-08-21
- 打赏
- 举报
用文件映射不可以吗？
Platform SDK: File Storage
MapViewOfFile
The MapViewOfFile function maps a view of a file into the address space of the calling process.
To specify a suggested base address, use the MapViewOfFileEx function.
LPVOID MapViewOfFile(
HANDLE hFileMappingObject, // handle to file-mapping object
DWORD dwDesiredAccess, // access mode
DWORD dwFileOffsetHigh, // high-order DWORD of offset
DWORD dwFileOffsetLow, // low-order DWORD of offset
SIZE_T dwNumberOfBytesToMap // number of bytes to map
);
Parameters
hFileMappingObject
[in] Handle to an open handle of a file-mapping object. The CreateFileMapping and OpenFileMapping functions return this handle.
dwDesiredAccess
[in] Specifies the type of access to the file view and, therefore, the protection of the pages mapped by the file. This parameter can be one of the following values. Value Meaning
FILE_MAP_WRITE Read/write access. The hFileMappingObject parameter must have been created with PAGE_READWRITE protection. A read/write view of the file is mapped.
FILE_MAP_READ Read-only access. The hFileMappingObject parameter must have been created with PAGE_READWRITE or PAGE_READONLY protection. A read-only view of the file is mapped.
FILE_MAP_ALL_ACCESS Same as FILE_MAP_WRITE.
FILE_MAP_COPY Copy on write access. If you create the map with PAGE_WRITECOPY and the view with FILE_MAP_COPY, you will receive a view to file. If you write to it, the pages are automatically swappable and the modifications you make will not go to the original data file.
Windows 95/98/Me: You must pass PAGE_WRITECOPY to CreateFileMapping; otherwise, an error will be returned.
If you share the mapping between multiple processes using DuplicateHandle or OpenFileMapping and one process writes to a view, the modification is propagated to the other process. The original file does not change.
Windows NT/2000/XP: There is no restriction as to how the hFileMappingObject parameter must be created. Copy on write is valid for any type of view.
If you share the mapping between multiple processes using DuplicateHandle or OpenFileMapping and one process writes to a view, the modification is not propagated to the other process. The original file does not change.
dwFileOffsetHigh
[in] Specifies the high-order DWORD of the file offset where mapping is to begin.
dwFileOffsetLow
[in] Specifies the low-order DWORD of the file offset where mapping is to begin. The combination of the high and low offsets must specify an offset within the file that matches the system's memory allocation granularity, or the function fails. That is, the offset must be a multiple of the allocation granularity. Use the GetSystemInfo function, which fills in the members of a SYSTEM_INFO structure, to obtain the system's memory allocation granularity.
dwNumberOfBytesToMap
[in] Specifies the number of bytes of the file to map. If dwNumberOfBytesToMap is zero, the entire file is mapped.
Return Values
If the function succeeds, the return value is the starting address of the mapped view.
If the function fails, the return value is NULL. To get extended error information, call GetLastError.
Remarks
Mapping a file makes the specified portion of the file visible in the address space of the calling process.
Multiple views of a file (or a file-mapping object and its mapped file) are said to be "coherent" if they contain identical data at a specified time. This occurs if the file views are derived from the same file-mapping object. A process can duplicate a file-mapping object handle into another process by using the DuplicateHandle function, or another process can open a file-mapping object by name by using the OpenFileMapping function.
A mapped view of a file is not guaranteed to be coherent with a file being accessed by the ReadFile or WriteFile function:
Windows 95/98/Me: MapViewOfFile may require the swapfile to grow. If the swapfile cannot grow, the function fails.
Windows NT/2000/XP: If the file-mapping object is backed by the paging file (hFile is INVALID_HANDLE_VALUE), the paging file must be large enough to hold the entire mapping. If it is not, MapViewOfFile fails.
Note To guard against an access violation, use structured exception handling to protect any code that writes to or reads from a memory mapped view. For more information, see Reading and Writing.
Example Code
For an example, see Creating Named Shared Memory.
Requirements
Windows NT/2000/XP: Included in Windows NT 3.1 and later.
Windows 95/98/Me: Included in Windows 95 and later.
Header: Declared in Winbase.h; include Windows.h.
Library: Use Kernel32.lib.
See Also
File Mapping Overview, File Mapping Functions, CreateFileMapping, DuplicateHandle, GetSystemInfo, MapViewOfFileEx, OpenFileMapping, UnmapViewOfFile, SYSTEM_INFO
Platform SDK Release: August 2001 What did you think of this topic?
Let us know. Order a Platform SDK CD Online
(U.S/Canada) (International)
Requirements
Windows NT/2000/XP: Included in Windows NT 3.1 and later.
Windows 95/98/Me: Included in Windows 95 and later.
Header: Declared in Winbase.h; include Windows.h.
Library: Use Kernel32.lib.
See Also
File Mapping Overview, File Mapping Functions, CreateFileMapping, DuplicateHandle, GetSystemInfo, MapViewOfFileEx, OpenFileMapping, UnmapViewOfFile, SYSTEM_INFO
Platform SDK: File Storage
MapViewOfFile
The MapViewOfFile function maps a view of a file into the address space of the calling process.
To specify a suggested base address, use the MapViewOfFileEx function.
LPVOID MapViewOfFile(
HANDLE hFileMappingObject, // handle to file-mapping object
DWORD dwDesiredAccess, // access mode
DWORD dwFileOffsetHigh, // high-order DWORD of offset
DWORD dwFileOffsetLow, // low-order DWORD of offset
SIZE_T dwNumberOfBytesToMap // number of bytes to map
);
Parameters
hFileMappingObject
[in] Handle to an open handle of a file-mapping object. The CreateFileMapping and OpenFileMapping functions return this handle.
dwDesiredAccess
[in] Specifies the type of access to the file view and, therefore, the protection of the pages mapped by the file. This parameter can be one of the following values. Value Meaning
FILE_MAP_WRITE Read/write access. The hFileMappingObject parameter must have been created with PAGE_READWRITE protection. A read/write view of the file is mapped.
FILE_MAP_READ Read-only access. The hFileMappingObject parameter must have been created with PAGE_READWRITE or PAGE_READONLY protection. A read-only view of the file is mapped.
FILE_MAP_ALL_ACCESS Same as FILE_MAP_WRITE.
FILE_MAP_COPY Copy on write access. If you create the map with PAGE_WRITECOPY and the view with FILE_MAP_COPY, you will receive a view to file. If you write to it, the pages are automatically swappable and the modifications you make will not go to the original data file.
Windows 95/98/Me: You must pass PAGE_WRITECOPY to CreateFileMapping; otherwise, an error will be returned.
If you share the mapping between multiple processes using DuplicateHandle or OpenFileMapping and one process writes to a view, the modification is propagated to the other process. The original file does not change.
Windows NT/2000/XP: There is no restriction as to how the hFileMappingObject parameter must be created. Copy on write is valid for any type of view.
If you share the mapping between multiple processes using DuplicateHandle or OpenFileMapping and one process writes to a view, the modification is not propagated to the other process. The original file does not change.
dwFileOffsetHigh
[in] Specifies the high-order DWORD of the file offset where mapping is to begin.
dwFileOffsetLow
[in] Specifies the low-order DWORD of the file offset where mapping is to begin. The combination of the high and low offsets must specify an offset within the file that matches the system's memory allocation granularity, or the function fails. That is, the offset must be a multiple of the allocation granularity. Use the GetSystemInfo function, which fills in the members of a SYSTEM_INFO structure, to obtain the system's memory allocation granularity.
dwNumberOfBytesToMap
[in] Specifies the number of bytes of the file to map. If dwNumberOfBytesToMap is zero, the entire file is mapped.
Return Values
If the function succeeds, the return value is the starting address of the mapped view.
If the function fails, the return value is NULL. To get extended error information, call GetLastError.
Remarks
Mapping a file makes the specified portion of the file visible in the address space of the calling process.
Multiple views of a file (or a file-mapping object and its mapped file) are said to be "coherent" if they contain identical data at a specified time. This occurs if the file views are derived from the same file-mapping object. A process can duplicate a file-mapping object handle into another process by using the DuplicateHandle function, or another process can open a file-mapping object by name by using the OpenFileMapping function.
A mapped view of a file is not guaranteed to be coherent with a file being accessed by the ReadFile or WriteFile function:
Windows 95/98/Me: MapViewOfFile may require the swapfile to grow. If the swapfile cannot grow, the function fails.
Windows NT/2000/XP: If the file-mapping object is backed by the paging file (hFile is INVALID_HANDLE_VALUE), the paging file must be large enough to hold the entire mapping. If it is not, MapViewOfFile fails.
Note To guard against an access violation, use structured exception handling to protect any code that writes to or reads from a memory mapped view. For more information, see Reading and Writing.
Example Code
For an example, see Creating Named Shared Memory.
Requirements
Windows NT/2000/XP: Included in Windows NT 3.1 and later.
Windows 95/98/Me: Included in Windows 95 and later.
Header: Declared in Winbase.h; include Windows.h.
Library: Use Kernel32.lib.
See Also
File Mapping Overview, File Mapping Functions, CreateFileMapping, DuplicateHandle, GetSystemInfo, MapViewOfFileEx, OpenFileMapping, UnmapViewOfFile, SYSTEM_INFO
Platform SDK Release: August 2001 What did you think of this topic?
Let us know. Order a Platform SDK CD Online
(U.S/Canada) (International)
Requirements
Windows NT/2000/XP: Included in Windows NT 3.1 and later.
Windows 95/98/Me: Included in Windows 95 and later.
Header: Declared in Winbase.h; include Windows.h.
Library: Use Kernel32.lib.
See Also
File Mapping Overview, File Mapping Functions, CreateFileMapping, DuplicateHandle, GetSystemInfo, MapViewOfFileEx, OpenFileMapping, UnmapViewOfFile, SYSTEM_INFO
huanghelang 2003-08-21
- 打赏
- 举报
各位高手说得都有道理，但使我想知道如果用得是UNIX，是否他的最大可用内存也是4G？
如果内存足够大，系统不繁忙时操作系统是否也会将其调入虚拟内存，可有什么办法让其不进入虚拟内存或者说少进行I/O操作？
如果内存足够大，系统不繁忙时操作系统是否也会将其调入虚拟内存，可有什么办法让其不进入虚拟内存或者说少进行I/O操作？
minghui000 2003-08-20
- 打赏
- 举报
买一条1G的内存
ironic 2003-08-20
- 打赏
- 举报
我们先来看看，你能不能得到上G的内存：
在win2k中，一般的程序可用的内存空间为2G - 64k -64k
(你不知道你用的是不是虚存，因为虚存对程序是透明的）
通过修改启动时的一个选项，可以将程序的可用内存空间增加到3G
（win2k的文档说明，这一项是用来支持需要内存非常大的程序，如数据挖掘)
所以说，你可以用到的内存空间可以达到1G
但是，当你用的内存增大至一定值时(尤其是当你申请的内存长时不用时），
系统会启用大量的虚存(这本身就是一项代价很大的I/O操作)
这样，原来你以为能减少I/O的操作却适得其反
你可以考虑别的方法
lyr311 2003-08-20
- 打赏
- 举报
我认为不一定能行，如果超过2G的话，现在除了大型机外，工作站上的软件好多也只能操作2G的文件，大型的数学软件Matlab也只能操作2G的文件，并且是其6.0以后的版本才可以的。这是因为好多机器都是32位的，大型机、并行机有64位甚至更多的，32位机理论上能操作的文件大小也就是2^(32-1)≈2G啊！
mme 2003-08-20
- 打赏
- 举报
不现实啊,这么大,等它读完了,你也睡着了,谁用这样的程序啊,呵呵
Jinhao 2003-08-20
- 打赏
- 举报
说句实话，我也知道这个方法的速度不够高，我这样说是因为我只看了题目。
加载更多回复（10）
