iocp 乱序的问题...

[downmooner]

我搜索了很多，发现网上的观点分2种，会乱序和不会乱序。
我觉得会乱序，有些疑问想请教大家。

windows网络编程2上原话：
All overlapped operations are guaranteed to be executed in the order that the application issued them. However, the completion notifications returned from a completion port are not guaranteed to be in that same order.

这样的话，问题来了，先比如wsarecv;
假设我接收文件, 协议消息长度是9k，有包头4字节表示长度什么的。
我连续投递2次wsarecv。第1次wsarecv (4),第2次wsarecv (9k - 4).

虽然都成功得到了。但get通知的顺序 第1次得到9k - 4,第2次得到4字节。这样逻辑就乱了。

再如wsasend
如果投递了2次, 第1次的返回通知，说发了一半. 这样第2个肯定还没完成.
那我继续接着第1次完成的一半投递剩下的一半。那我第3次的会接着第2次发？这也乱了。

还有因为是多个工作线程, 如果分别有2个线程得到了通知。那岂不是还要做线程同步

所以我觉得对于单个socket，同一时间只能存在一个wsasend 和 wsarecv.

请问我的想法对不对。如果是对的，该怎么解决呢

[kyotrue]

你先后向192.168.1.2和192.168.1.3发送了两个数据包，发往192.168.1.3的可能先完成，这样就是所谓的乱序了。

但是如果你在同一个TCP连接上，先后提交了几个发送操作，那么完成时通知顺序肯定是提交时的顺序，这就是所谓的不乱序。

[downmooner]

Before continuing, there is one other important aspect about Windows completion ports that needs to be stressed. All overlapped operations are guaranteed to be executed in the order that the application issued them. However, the completion notifications returned from a completion port are not guaranteed to be in that same order. That is, if an application posts two overlapped WSARecv operations, one with a 10 KB buffer and the next with a 12 KB buffer, the 10 KB buffer is filled first, followed by the 12 KB buffer. The application's worker thread may receive notification from GetQueuedCompletionStatus for the 12 KB WSARecv before the completion event for the 10 KB operation. Of course, this is only an issue when multiple operations are posted on a socket.

书上的原话，没有讲不同连接上的通知可能是乱序，相同连接的确定有序呀。

[downmooner]

Before continuing, there is one other important aspect about Windows completion ports that needs to be stressed. All overlapped operations are guaranteed to be executed in the order that the application issued them. However, the completion notifications returned from a completion port are not guaranteed to be in that same order. That is, if an application posts two overlapped WSARecv operations, one with a 10 KB buffer and the next with a 12 KB buffer, the 10 KB buffer is filled first, followed by the 12 KB buffer. The application's worker thread may receive notification from GetQueuedCompletionStatus for the 12 KB WSARecv before the completion event for the 10 KB operation. Of course, this is only an issue when multiple operations are posted on a socket.

[kyotrue]

As you would expect, entries are removed from the I/O completion queue in a first-in first-out fashion. However, as you might not expect, threads that call GetQueuedCompletionStatus are awakened in a last-in first-out (LIFO) fashion. The reason for this is again to improve performance. For example, say that four threads are waiting in the waiting thread queue. If a single completed I/O entry appears, the last thread to call GetQueuedCompletionStatus wakes up to process the entry. When this last thread is finished processing the entry, the thread again calls GetQueuedCompletionStatus to enter the waiting thread queue. Now if another I/O completion entry appears, the same thread that processed the first entry is awakened to process the new entry.

《Windows核心编程》上原话，先完成肯定先被取出。TCP是面向数据流的，不可能乱序传输啊

[downmooner]

[Quote=引用 1 楼 kyotrue 的回复:]

你先后向192.168.1.2和192.168.1.3发送了两个数据包，发往192.168.1.3的可能先完成，这样就是所谓的乱序了。

但是如果你在同一个TCP连接上，先后提交了几个发送操作，那么完成时通知顺序肯定是提交时的顺序，这就是所谓的不乱序。
[/Quote]

还有按照你的观点,如我举例的wsasend也还是乱序的。

[smwhotjay]

虽然tcp是顺序的.但你的io操作却乱序了。 比如你同时投递两个异步Recv. 那么可能getiocp状态时分别两个Recv都完成了,但字节可能不足。

[downmooner]

[Quote=引用 4 楼 kyotrue 的回复:]

Assembly code
As you would expect, entries are removed from the I/O completion queue in a first-in first-out fashion. However, as you might not expect, threads that call GetQueuedCompletionStatus ar……
[/Quote]

那是在这个线程已经在执行的状态,如果这个线程被cpu切换了呢。肯定有另外的工作线程来获取通知。

[kyotrue]

把帖子连接发出来吧，看看你是怎么检测是否乱序的

[downmooner]

就是看着书上想的。帖子都是csdn论坛上的

[kyotrue]

先完成先被处理这个应该是没问题的，主要问题就在于先提交的是否先完成。

总之TCP本身协议栈就决定了它是有序的，如果你实在不放心，可以在OVERLAPPED结构里面保存个序号。

[downmooner]

tcp栈当然是有序的。我的意思是获取完成通知是无序的

[hurryboylqs]

不是获取完成通知无序，而是因为IOCP一般会在多个工作线程中工作，导致所谓的乱序，如果仅仅有一个工作线程来获取完成通知，那就不存在上面乱序问题了，但单个线程处理体现不出IOCP的优势
关于实现细节可以看下面这篇文章：
http://www.codeproject.com/KB/IP/iocp_server_client.aspx

[xiayadong]

发送端是不会乱序的，接收端实际上也不会乱序，之所以接收端乱序，是完成封包时通知顺序不同而已，比如：

你在发送端发送:111,然后再发送222

你在接收端先后投递了两个Wsarecv:wsarecv1,wsarecv2;
那么wsarecv1接收到111,wsarecv2接收到222,
因为线程切换的原因，系统可能会先通知你wsarecv2接收到了封包通知，再通知你swarecv1接收到了封包通知，这样就会引起乱序了，解决方法是：可以在投递的封包中加上投递的序号。
可以参考《windows网络与通信程序设计》

[jwybobo2007]

没细看你说的内容。

本身来说不存在乱序和不乱序的说话。
看你怎么操作了。

如果你多次连续投地 wsasend,那不管是tcp还是udp都可能顺序不一致。
同理wsarecv也一样

[yynetsdk]

对completekey进行顺序标记，并加缓存队列

[shenyi0106]

如果同时存在多个一样的提交动作(wsarecv或者wsasend),是有乱序的可能，所以你要额外处理，或者同一个socket的同一种操作之保留一个提交动作

[downmooner]

我比较赞成一个socket同种操作只能存在一个提交。
但上面的链接作者说
This can be solved by only using one worker thread, and committing only one I/O call and waiting for it to finish, but if we do this, we lose all the benefits of IOCP.

我感觉是危言耸听。相对单个socket来，这样应答式的是慢了。但大批量的时候总体会慢吗？还有他说只有一个工作线程应该是错的。 后面的and应该是or
他的解决方法是添加序列号，那他这样不要考虑线程同步吗？这样肯定也有效率问题.还有我记得windows网络编程2的iocp源代码就是多线程单socket send和recv保持1个。

A simple practical solution to this problem is to add a sequence number to our buffer class, and process the data in the buffer if the buffer sequence number is in order。。。

[shenyi0106]

worker thread和logic thread分开的话，不见得慢，而且还增强了系统的健壮性和可扩展性

[smwhotjay]

iocp要发挥强大能力,就必须overlapped. 重叠io.意思就是对socket异步提交多个io. 实现起来效率最高,也最复杂. 这就是重叠io的原意

[lijianli9]

简单点处理就是同一时刻只有一个recv或者send，
复杂点的话就是你发送和接收需要发送包ID，然后组包。