多程序写一个文件/文件的并行写入如何处理？

gong_xufei 2014-01-19 04:35:58

如题

1、问题背景：
（1）多个程序同时运行（不是MPI也不是OpenMP，是多个独立程序）
（2）每个程序都要以append方式写同一个文件

2、请问：
（1）如果就按照常规的不考虑同时操作的情况，两个或多个程序同时对文件进行操作时会出现什么情况？排队写？竞争？……？
（2）如有问题，应该如何应对？

3、我的猜测：
需要加特殊处理：因为某个程序中的文件指针不会因为别的程序中对文件写入了内容而后移，所以当它继续写的时候，会覆盖别的程序已经写入的内容。

想过编程验证，但是估计了一下不大好验证。

求指导。
该问题应该不是依赖于编程语言的一个问题，为方便，如涉及编程语言，且以C语言讨论。

...全文

667 11 打赏收藏转发到动态举报

写回复

用AI写文章

11 条回复

切换为时间正序

请发表友善的回复…

发表回复

lm_whales 2014-01-22

打赏
举报

文件可以加锁(Windows,unix,linux?)，文件可以共享读，写方式打开，共享的文件，可以锁住某一部分(文件锁)。如果你使用DOS早期版本，另说。。。不过早期版本的DOS，只能同时运行一个程序，多个程序同时运行很困难。

赵4老师 2014-01-22

打赏
举报

_locking Locks or unlocks bytes of a file. int _locking( int handle, int mode, long nbytes ); Routine Required Header Optional Headers Compatibility _locking <io.h> and <sys/locking.h> <errno.h> Win 95, Win NT For additional compatibility information, see Compatibility in the Introduction. Libraries LIBC.LIB Single thread static library, retail version LIBCMT.LIB Multithread static library, retail version MSVCRT.LIB Import library for MSVCRT.DLL, retail version Return Value _locking returns 0 if successful. A return value of –1 indicates failure, in which case errno is set to one of the following values: EACCES Locking violation (file already locked or unlocked). EBADF Invalid file handle. EDEADLOCK Locking violation. Returned when the _LK_LOCK or _LK_RLCK flag is specified and the file cannot be locked after 10 attempts. EINVAL An invalid argument was given to _locking. Parameters handle File handle mode Locking action to perform nbytes Number of bytes to lock Remarks The _locking function locks or unlocks nbytes bytes of the file specified by handle. Locking bytes in a file prevents access to those bytes by other processes. All locking or unlocking begins at the current position of the file pointer and proceeds for the next nbytes bytes. It is possible to lock bytes past end of file. mode must be one of the following manifest constants, which are defined in LOCKING.H: _LK_LOCK Locks the specified bytes. If the bytes cannot be locked, the program immediately tries again after 1 second. If, after 10 attempts, the bytes cannot be locked, the constant returns an error. _LK_NBLCK Locks the specified bytes. If the bytes cannot be locked, the constant returns an error. _LK_NBRLCK Same as _LK_NBLCK. _LK_RLCK Same as _LK_LOCK. _LK_UNLCK Unlocks the specified bytes, which must have been previously locked. Multiple regions of a file that do not overlap can be locked. A region being unlocked must have been previously locked. _locking does not merge adjacent regions; if two locked regions are adjacent, each region must be unlocked separately. Regions should be locked only briefly and should be unlocked before closing a file or exiting the program. Example /* LOCKING.C: This program opens a file with sharing. It locks * some bytes before reading them, then unlocks them. Note that the * program works correctly only if the file exists. */ #include <io.h> #include <sys/types.h> #include <sys/stat.h> #include <sys/locking.h> #include <share.h> #include <fcntl.h> #include <stdio.h> #include <stdlib.h> void main( void ) { int fh, numread; char buffer[40]; /* Quit if can't open file or system doesn't * support sharing. */ fh = _sopen( "locking.c", _O_RDWR, _SH_DENYNO, _S_IREAD | _S_IWRITE ); if( fh == -1 ) exit( 1 ); /* Lock some bytes and read them. Then unlock. */ if( _locking( fh, LK_NBLCK, 30L ) != -1 ) { printf( "No one can change these bytes while I'm reading them\n" ); numread = _read( fh, buffer, 30 ); printf( "%d bytes read: %.30s\n", numread, buffer ); lseek( fh, 0L, SEEK_SET ); _locking( fh, LK_UNLCK, 30L ); printf( "Now I'm done. Do what you will with them\n" ); } else perror( "Locking failed\n" ); _close( fh ); } Output No one can change these bytes while I'm reading them 30 bytes read: /* LOCKING.C: This program ope Now I'm done. Do what you will with them File Handling Routines See Also _creat, _open

赵4老师 2014-01-21

打赏
举报

改为多程序通过多个数据库连接写数据库表。

jiewinli 2014-01-20

打赏
举报

学习下 ……

dcw0402 2014-01-20

打赏
举报

这些并行的程序如果是代码可控的话,可以把文件操作封装成一个动态库,通过内核级全局锁操作文件,可以保证数据无误,这样一来,就把并行变成同步了

gong_xufei 2014-01-19

打赏
举报

引用 3 楼 u011704121 的回复:

其实可以把所有要写的东西排队，只有一个程序可以写，别的程序想写，就发送给他就行

可否再描述详细一点？我一会在做一般的C编程，排队、发送什么的概念有点陌生

gong_xufei 2014-01-19

打赏
举报

引用 4 楼 mujiok2003 的回复:

[quote=引用 2 楼 gong_xufei 的回复:] [quote=引用 1 楼 mujiok2003 的回复:] 1. 打开文件的时候可以指明是否是独占 2. 在共享模式下,由操作系统保证同步.

多谢解答。但实现上还是不很明白，别处也没搜索到：（1）如何指明是否独占？（2）共享模式是否需要在程序中进行实现？如何实现？这两个问题貌似是一个。。。 PS：我做的是linux上的C编程谢谢！[/quote] http://linux.die.net/man/2/open[/quote] 貌似很强大的样子，先研究一下。