Netty大并发时出现解码错误

meoujb 2013-11-05 10:41:22
用netty写的一个网关服务器的验证，在并发数比较大的时候，就会出现解码的问题，各位帮忙看看，好几天了，一直没解决。

(1)每隔50ms生成一个客户端进行压测，客户端每隔1s发送200bytes的数据，服务器收到后，将其加入到队列中。

(2)另开一个线程处理队列，目前是简单的将其发给客户端。

(3)客户端收到服务器的消息后，也是将其加入到队列中，由另外一个线程进行处理，将消息发送时间和接收时间记录到文件中。

协议：
包体长度+包体
包体使用的是一个map，数据格式定义：
* 字段1键名长度 字段1键名 字段1值长度 字段1值
* 字段2键名长度 字段2键名 字段2值长度 字段2值
* 字段3键名长度 字段3键名 字段3值长度 字段3值 
public class XLRequest {

   private int length;	// 数据包长

  

   private Map<String,String> datas=new HashMap<String, String>(); //参数

   。。。。。。。

}


编码：
public class MessageEncoder extends OneToOneEncoder  {

	

	public void encodeDataBuffer(Map<String, String> values, ChannelBuffer totalBuffer) throws Exception {

		if (values != null && values.size() > 0) {

			for (Entry<String, String> entry : values.entrySet()) {

				String key = entry.getKey();

				String value = entry.getValue();

				totalBuffer.writeInt(key.getBytes().length);

				totalBuffer.writeBytes(key.getBytes());

				totalBuffer.writeInt(value.getBytes().length);

				totalBuffer.writeBytes(value.getBytes());

			}

		}

	}

	

	public int getDataLength(Map<String, String> values){

		int dataLength = 0;

		for (Entry<String, String> entry : values.entrySet()) {

			String key = entry.getKey();

			String value = entry.getValue();

			dataLength = dataLength + 4 + key.getBytes().length + 4

					+ value.getBytes().length;

		}

		return dataLength;

	}



	@Override

	protected Object encode(ChannelHandlerContext ctx, Channel channel,

			Object msg) throws Exception {

		XLRequest request = (XLRequest) msg;

		ChannelBuffer totalBuffer = ChannelBuffers.dynamicBuffer();

		

		Map<String, String> datas = request.getDatas();

		if(datas==null || datas.isEmpty()){

			throw new Exception("内容为空!");

		}

		totalBuffer.writeInt(getDataLength(datas));

		

		encodeDataBuffer(datas, totalBuffer);

		return totalBuffer;

	}

}


解码：
public class MessageDecoder extends FrameDecoder {

	public static final Logger logger = LoggerFactory.getLogger(MessageDecoder.class);

	

	@Override

	protected Object decode(ChannelHandlerContext ctx, Channel channel,

			ChannelBuffer buffer) throws Exception {

		

		if (buffer.readableBytes() < 4) { 

			return null;

		}

		buffer.markReaderIndex();

		int totalLength = buffer.readInt(); // 数据包长

		

		if (buffer.readableBytes() < totalLength) {

			buffer.resetReaderIndex();

			return null;

		}

		ChannelBuffer dataBuffer = ChannelBuffers.buffer(totalLength);

		buffer.readBytes(dataBuffer, totalLength);

		

		XLRequest request = new XLRequest();

		request.setLength(totalLength);

		request.setDatas(decodeDataBuffer(dataBuffer));

		return request;

	}

	

	public Map<String, String> decodeDataBuffer(ChannelBuffer dataBuffer ) throws IOException, Exception {

		Map<String, String> dataMap = new HashMap<String, String>();

		

		if (dataBuffer != null && dataBuffer.readableBytes() > 0) {

			int processIndex = 0;

			int totalLength = dataBuffer.readableBytes();

			while (processIndex < totalLength) {

				/**

				 * 获取Key

				 */

				int size = dataBuffer.readInt();

				byte[] contents = new byte[size];

				dataBuffer.readBytes(contents);

				String key = new String(contents);

				processIndex = processIndex + size + 4;

				

				/**

				 * 获取Value

				 */

			

				size = dataBuffer.readInt();

				contents = new byte[size];

				dataBuffer.readBytes(contents);

				String value = new String(contents);

				

				dataMap.put(key, value);

				processIndex = processIndex + size + 4;

			}

		}

		return dataMap;

	}

}


ClientHandler：
public XLRequest createSendData(Integer channelId) {

		XLRequest request = new XLRequest();

		if(firstCreated){

			msg2_1024bytes = formResponseMessage(2);

			msg5_1024bytes = formResponseMessage(5);

			msg10_1024bytes = formResponseMessage(10);

			msg50_1024bytes = formResponseMessage(50);

			firstCreated = false;

		}

		String msg = getMessage(ConfigUtil.SEND_BYTES);

		request.setValue("msg", msg);

		request.setValue("sid", lastSid.getAndIncrement()+"");

		request.setValue("time", System.currentTimeMillis()+"");

		return request;

	}

	

	@Override

	public void messageReceived(ChannelHandlerContext ctx, MessageEvent messageEvent)

			throws Exception {

		XLRequest request = (XLRequest) messageEvent.getMessage();

		

		request.setValue("recvTime", System.currentTimeMillis()+"");

		messageManager.addSocketMessage(messageManager.handleMessage(CommandType.CLIENT_REQUEST, 

				ctx.getChannel(), request));

	}

	



	@Override

	public void channelConnected(ChannelHandlerContext ctx, ChannelStateEvent e)

			throws Exception {

		Channel channel = ctx.getChannel();

		if (channel != null) {

			// send data

			XLRequest clientRequest = createSendData(channel.getId());

			ClientSenderThread senderThread = new ClientSenderThread(channel, clientRequest);

			Thread thread = new Thread(senderThread);

			thread.start();

		}

		super.channelConnected(ctx, e);

	}


ClientFactory：
public class ClientFactory implements ChannelPipelineFactory {



	private MessageDecoder messageDecoder = new MessageDecoder();

	private MessageEncoder messageEncoder = new MessageEncoder();

	private ClientHandler clientHandler = new ClientHandler();	

	

	ExecutionHandler executionHandler = new ExecutionHandler( 

			new OrderedMemoryAwareThreadPoolExecutor(

					SysInfo.getProcessors()*ConfigUtil.GATEWAY_PROCESS, 

					ConfigUtil.MAX_CHANNEL_MEMORY_SIZE, 

					0)

			);

	public ChannelPipeline getPipeline() throws Exception {



		ChannelPipeline cp = Channels.pipeline();

		cp.addLast("messageClientDecoder", messageDecoder);

		cp.addLast("messageClientEncoder", messageEncoder);

		cp.addLast("executionHandler", executionHandler);

		cp.addLast("clientHandler", clientHandler);

		return cp;

	}

}

...全文
549 7 打赏 收藏 转发到动态 举报
AI 作业
写回复
用AI写文章
7 条回复
切换为时间正序
请发表友善的回复…
发表回复
meoujb 2013-12-14
  • 打赏
  • 举报
 回复
public class GatewayFactory implements ChannelPipelineFactory { private MessageDecoder messageDecoder = new MessageDecoder(); private MessageEncoder messageEncoder = new MessageEncoder(); private GatewayMessageHandler gatewayMessageHandler = new GatewayMessageHandler(); ExecutionHandler executionHandler = new ExecutionHandler( new OrderedMemoryAwareThreadPoolExecutor( SysInfo.getProcessors()*ConfigUtil.GATEWAY_PROCESS, ConfigUtil.MAX_CHANNEL_MEMORY_SIZE, 0) ); public ChannelPipeline getPipeline() throws Exception { ChannelPipeline cp = Channels.pipeline(); cp.addLast("messageServerDecoder", messageDecoder); cp.addLast("messageServerEncoder", messageEncoder); cp.addLast("executionHandler", executionHandler); cp.addLast("gatewayMessageHandler", gatewayMessageHandler); return cp; } } 错误在这里,应该是每次都要NEW一个的，cp.addLast("messageServerDecoder", new messageDecoder())
SimonDW 2013-11-12
  • 打赏
  • 举报
 回复
MessageDecoder 的decodeDataBuffer里readInt前没对readableBytes检查。 考虑到Tcp分段问题，可能等整个消息到了之后再解码更好 另外 netty本身即有基于长度字段的decoder LengthFieldBasedFrameDecoder
meoujb 2013-11-05
  • 打赏
  • 举报
 回复
哎，眼看帖子就要沉下去了，来人啦。。。。
meoujb 2013-11-05
  • 打赏
  • 举报
 回复
引用 2 楼 shadowsick 的回复:
帮顶下吧...
小丑哥_V5 2013-11-05
  • 打赏
  • 举报
 回复
帮顶下吧...
meoujb 2013-11-05
  • 打赏
  • 举报
 回复
ServerHandler: 
	@Override
	public void messageReceived(ChannelHandlerContext ctx, MessageEvent messageEvent)
			throws Exception {
		Channel sendChannel = ctx.getChannel();
		doProcessFromClient(ctx, messageEvent, sendChannel);
	}
	private void doProcessFromClient(ChannelHandlerContext ctx,
			MessageEvent messageEvent, Channel sendChannel) {
		XLRequest request = (XLRequest)messageEvent.getMessage();
		byte command = CommandType.GATE_TO_CLIENT_NORMAL;
		messageManager.addSocketMessage(messageManager.handleMessage(command, ctx.getChannel(), request));
	}
GateWayFactory: 
public class GatewayFactory implements ChannelPipelineFactory {

	private MessageDecoder messageDecoder = new MessageDecoder();
	private MessageEncoder messageEncoder = new MessageEncoder();
	private GatewayMessageHandler gatewayMessageHandler = new GatewayMessageHandler();		
	ExecutionHandler executionHandler = new ExecutionHandler( 
			new OrderedMemoryAwareThreadPoolExecutor(
					SysInfo.getProcessors()*ConfigUtil.GATEWAY_PROCESS, 
					ConfigUtil.MAX_CHANNEL_MEMORY_SIZE, 
					0)
			);
	
	public ChannelPipeline getPipeline() throws Exception {
		ChannelPipeline cp = Channels.pipeline();
		cp.addLast("messageServerDecoder", messageDecoder);
		cp.addLast("messageServerEncoder", messageEncoder);
		cp.addLast("executionHandler", executionHandler);
		cp.addLast("gatewayMessageHandler", gatewayMessageHandler);
		return cp;
	}
	
}
消息队列： 
public class MessageManager {
	
	public static final Logger logger = LoggerFactory.getLogger(MessageManager.class);

	/** MessageManager的消息队列，下面的addSocketMessage方法就是向这个队列塞MessagePack的*/
    private LinkedBlockingQueue<MessagePack> receivedQueen = new LinkedBlockingQueue<MessagePack>();
    private ExecutorService pool;
    
    
    /** 登记式单例类 */
    private static AtomicInteger instanceIndex = new AtomicInteger(0);
    private static Map<String, MessageManager> instanceMap = new HashMap<String, MessageManager>();
    private String instanceName = "";
    public static final String INSTANCE_CLIENT_TO_GATE = "client_to_gate";
    public static final String INSTANCE_GATE_TO_CLIENT = "gate_to_client";
    private MessageManager(String name){
    	this.instanceName = name;
    }
    public String getInstanceName() {
		return instanceName;
	}
	public static MessageManager getInstance(String name){
    	if(StringUtils.isBlank(name)){
    		name = MessageManager.class.getName().concat(instanceIndex.get()+"");
    		instanceIndex.getAndIncrement();
    	}
    	
    	synchronized (instanceMap) {
    		MessageManager instance = instanceMap.get(name);
        	if(instance==null){
        		instance = new MessageManager(name);
        		instanceMap.put(name, instance);
        	}
        	return instance;
		}
    }
	

    public MessagePack handleMessage(byte command, Channel sendChannel, XLRequest request) {
        MessagePack messagePack = null;

        switch (command) {
        case CommandType.GATE_TO_CLIENT_NORMAL: 
        	messagePack = new GateMessagePack(command, sendChannel, request);
        	break;
        case CommandType.CLIENT_REQUEST:
        	messagePack = new ClientMessagePack(command, sendChannel, request);
        }
        return messagePack;
    }

    public void start() {
    	pool = Executors.newFixedThreadPool(SysInfo.getProcessors()*2);
        for(int i=0;i<SysInfo.getProcessors();i++){
        	 pool.execute(new PushRecvThread());
        }
    }

	private class PushRecvThread implements Runnable {

		public void run() {
			while (true) {
				// 从队列中取继承MessagePack的实例
				MessagePack message;
				try {
					message = receivedQueen.poll(1, TimeUnit.MILLISECONDS);
					if (message != null) {
						message.onHandler();
					}
				} catch (InterruptedException e) {
					logger.error(e.getMessage(), e);
				}
			}
		}
	}
	public void addSocketMessage(MessagePack message) {
		if (message != null) {
			try {
				boolean success = receivedQueen.offer(message, 1,TimeUnit.MILLISECONDS);
			} catch (InterruptedException e) {
				logger.error(e.getMessage(), e);
			}
		}
	}
}
代码比较多，源代码我上传到百度网盘了，地址： http://pan.baidu.com/s/17Vbyg 7k用户每隔1s发送200byte数据的时候，就会出现如下的错误，而且还有个很奇怪的问题，在这7k用户建立连接的过程中，不会出现解码问题，全部建立连接后不久，就会出现解码问题。不知道是不是我写的代码有问题，各位大牛帮忙看看吧。 
java.lang.IndexOutOfBoundsException: Not enough readable bytes - Need 4, maximum is 3
	at org.jboss.netty.buffer.AbstractChannelBuffer.checkReadableBytes(AbstractChannelBuffer.java:661)
	at org.jboss.netty.buffer.AbstractChannelBuffer.readInt(AbstractChannelBuffer.java:273)
	at com.zltx.game.sky.net.verify.MessageDecoder.decodeDataBuffer(MessageDecoder.java:51)
	at com.zltx.game.sky.net.verify.MessageDecoder.decode(MessageDecoder.java:37)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
java.lang.IndexOutOfBoundsException: Not enough readable bytes - Need 825570103, maximum is 0
	at org.jboss.netty.buffer.AbstractChannelBuffer.checkReadableBytes(AbstractChannelBuffer.java:661)
	at org.jboss.netty.buffer.AbstractChannelBuffer.readBytes(AbstractChannelBuffer.java:338)
	at org.jboss.netty.buffer.AbstractChannelBuffer.readBytes(AbstractChannelBuffer.java:344)
	at com.zltx.game.sky.net.verify.MessageDecoder.decodeDataBuffer(MessageDecoder.java:53)
	at com.zltx.game.sky.net.verify.MessageDecoder.decode(MessageDecoder.java:37)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
MiceRice 2013-11-05
  • 打赏
  • 举报
 回复
java.lang.IndexOutOfBoundsException: Not enough readable bytes - Need 4, maximum is 3 第一个错误看起来不像是什么错误，你要求读取4个字节，但实际上可读取的只有3个字节，也许是网络还没发送过来。 java.lang.IndexOutOfBoundsException: Not enough readable bytes - Need 825570103, maximum is 0 第二个错误比较离谱点，你要接受的数据居然有这么大？825MB？！是不是前面识别错误了？ 建议用工具截取网络数据包，看看是不是发送跟接收处理跟预期的不一致。
93个netty高并发教学视频下载.txt
93个netty高并发全面的教学视频下载，每个视频在400-700M，一到两个小时时长的视频，无机器码和解压密码，下载下来的就是MP4格式视频。点击即可观看学习。下载txt文档，里面有永久分享的连接。包括01_学习的要义；02_Netty宏观理解；03_Netty课程大纲深度解读；04_项目环境搭建与Gradle配置；05_Netty执行流程分析与重要组件介绍；06_Netty回调与Channel执行流程分析；07_Netty的Socket编程详解；08_Netty多客户端连接与通信，09_Netty读写检测机制与长连接要素，10_Netty对WebSocket的支援；11_Netty实现服务器端与客户端的长连接通信；12_Google Protobuf详解；13_定义Protobuf文件及消息详解；14_Protobuf完整实例详解；15_Protobuf集成Netty与多协议消息传递；16_Protobuf多协议消息支援与工程最佳实践；17_Protobuf使用最佳实践与Apache Thrift介绍；18_Apache Thrift应用详解与实例剖析；19_Apache Thrift原理与架构解析；20_通过Apache Thrift实现Java与Python的RPC调用；21_gRPC深入详解 ；22_gRPC实践 ；23_Gradle Wrapper在Gradle项目构建中的最佳实践；24_gRPC整合Gradle与代码生成············82_Netty引用计数原子更新揭秘与AtomicIntegerFieldUpdater深度剖析；83_AtomicIntegerFieldUpdater实例演练与volatile关键字分析；84_Netty引用计数注意事项与内存泄露检测方式；85_Netty编解码器剖析与入站出站处理器详解；86_Netty自定义编解码器与TCP粘包拆包问题；87_Netty编解码器执行流程深入分析；88_ReplayingDecoder源码分析与特性解读；89_Netty常见且重要编解码器详解；90_TCP粘包与拆包实例演示及分析；91_Netty自定义协议与TCP粘包拆包问题解决之道；92_精通并发与Netty课程总结与展望
浅析Netty,让高并发来的更猛烈些吧
netty简单介绍，希望对你有所帮助
netty-高并发编程-视频教程张龙-90讲完整视频
netty-高并发编程-视频教程张龙
Netty性能测试
从测试结果看，Netty性能是非常高的，在所有使用默认配置的情况下，单台服务器能够达到4万次请求解析，作为RPC框架是足够用的。
netty-redis-zookeeper高并发实战学习-netty-redis-zookeeper.zip
netty-redis-zookeeper高并发实战学习-netty-redis-zookeeper
Java EE

67,549

社区成员

225,860

社区内容

发帖
与我相关
我的任务
社区描述
J2EE只是Java企业应用。我们需要一个跨J2SE/WEB/EJB的微容器，保护我们的业务核心组件（中间件），以延续它的生命力，而不是依赖J2SE/J2EE版本。
社区管理员
  • Java EE
加入社区
  • 近7日
  • 近30日
  • 至今

加载中

查看更多榜单
社区公告
暂无公告

试试用AI创作助手写篇文章吧

+ 用AI写文章