记一次axsi2调用 web services 提示 java.net.SocketException: Connection reset 的解决

boboy1234 2022-11-02 10:44:20

最终问题的解决与源代码没有关系，在此就不贴代码了。

一、问题发生

在普通电脑上通过axsi2调用远端服务器的web services上传一直正常，但将程序部署到我们服务器上，调用上级单位web services一直提示Connection reset，具体报错如下：

[EHRLOG] 2022-10-23 16:54:06,113 - INFO - Unable to sendViaPost to url[http://xxx.xxx.xxx.xxx:xxxx/pha/services/phaService] - org.apache.axis2.transport.http.HTTPSenderorg.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:202) -140823 [http-nio-8077-exec-9]
java.net.SocketException: Connection reset
   at java.net.SocketInputStream.read(Unknown Source)
   at java.net.SocketInputStream.read(Unknown Source)
   at java.io.BufferedInputStream.fill(Unknown Source)
   at java.io.BufferedInputStream.read(Unknown Source)
   at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
   at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
   at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
   at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
   at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
   at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
   at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:557)
   at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)
   at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
   at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
   at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
   at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:438)
   at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
   at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
   at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
   at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:540)
   at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:521)
   at org.apache.axis2.rpc.client.RPCServiceClient.invokeBlocking(RPCServiceClient.java:102)
   at org.sx.ehr.webservice.client.pha.ServiceFuncHelper.functionMain(ServiceFuncHelper.java:36)
   at org.sx.ehr.webservice.client.pha.action.UpLoadAction.upLoadRpt(UpLoadAction.java:185)
   at org.sx.ehr.webservice.client.pha.action.UpLoadAction.uploadAreaRptByArea(UpLoadAction.java:130)
   at org.apache.jsp.sdc.pt.EHR_005fCAPITAL_005fRPT.uploadCapitalRpt_jsp._jspService(uploadCapitalRpt_jsp.java:540)
   at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:71)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
   at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:476)
   at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386)
   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.sx.ehr.common.SqlFilter.doFilter(SqlFilter.java:46)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
   at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
   at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:615)
   at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
   at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
   at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1627)
   at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
   at java.lang.Thread.run(Unknown Source)

二、尝试解决

在网上找了很多种解决办法，一般都是增加配置项，比如协议版本、超时等：

1、options.setProperty(HTTPConstants.CHUNKED, "FALSE");
2、options.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, "FALSE");
3、options.setProperty(HTTPConstants.AUTO_RELEASE_CONNECTION , Boolean.TRUE);
4、options.setProperty(HTTPConstants.HTTP_PROTOCOL_VERSION,HTTPConstants.HEADER_PROTOCOL_10); 或者11。
5、options.setTimeOutInMilliSeconds(600000);
6、options.setProperty(HTTPConstants.SO_TIMEOUT,60000);
7、options.setProperty(HTTPConstants.CONNECTION_TIMEOUT,60000);

但是测试后还是提示Connection reset。

三、抓包分析

实在找不到原因，只能抓包看一下，175是我本地服务器，36是远端服务器：

1、发现是本地175服务器主动发送了[RST,ACK]，远端服务器收到后关闭了连接，所以日志提示Connection reset。
2、又研究了一下TCP抓包最终发现：
No46:本地服务器向远端发送了Seq=185 Ack=1 Len=2048的包；
No53、54等：远端提示[TCP dup ack]重新请求Seq=1 Ack=185的包，证明远端没有收到No46发送的包；
No58的快速重发及No65、68、72等：本地服务器又重新发送了Seq=185 Ack=1的包，但Len=1460,与第一次No46发的包长度不一致，第一次是2048，这次是1460。
3、根据TCP协议，每次的包长度会影响下一个包的Seq，所以经请求后重新发的包长度与第一次不一致，当然不会被远端服务器接收，就是接收了，与前面No44、45、46、47、48都无法对应，也应该是无效包。
4、这里就有一个问题，在TCP协议开始协商时，No42、43两方的MSS都为1460.为何No46会发送长度2048的包，经远端请求后重发，长度又变为1460，当时就觉得此处肯定有问题。

四、最终解决

目前问题就是TCP协议开始协商时，No42、43两方的MSS都为1460，但No46发送了长度2048的包。经过查询，发现这跟网卡的TSO（TCP Segment Offload）功能有关，具体也搞不太懂，好像就是一种使用网卡替代CPU，将超过MSS的包进行分段，所以可以发送超过MSS长度的包。

我们的服务器是Windows Server，查询了一下关闭网卡TSO功能的方式，将TSO关闭后，调用接口就正常了。原来测试电脑的网卡没这个功能，所以上传正常，部署到服务器后，网卡默认开启了这个功能，就有问题了。

PS 在虚拟机上是关闭这个Checksum Offload。