Hadoop HDFS FSDataOutputStream append问题 提示文件已创建

[chen820655096]

我在对文件进行追加操作时出现如下异常（源代码在后面），求各位大神帮忙看看，万分感谢
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException): Failed to create file [/fileserver/230/230_201406190911170002.rar] for [DFSClient_NONMAPREDUCE_1390018422_23] on client [192.168.95.5], because this file is already being created by [DFSClient_NONMAPREDUCE_-1981255109_33] on [192.168.95.5]
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2320)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2153)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:2386)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2347)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:508)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:320)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59572)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)

at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy18.append(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy18.append(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.append(ClientNamenodeProtocolTranslatorPB.java:245)
at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1480)
at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1520)
at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1508)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:310)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:306)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:306)
at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1160)
at com.cmcciot.platform.filemanager.serviceimpl.HdfsStorgeFileServiceImpl.Storge(HdfsStorgeFileServiceImpl.java:41)
at com.cmcciot.platform.filemanager.service.FileService.UploadImage(FileService.java:376)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:855)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)

FileSystem fileSystem=getFileSystem();
			try {
				FSDataOutputStream out=fileSystem.append(new Path(outStreampath));//这句出错
				IOUtils.copyBytes(inStream, out, 1024,true);
		        inStream.close(); // InputStream对象
		        out.close();
			} catch (Exception e) {
				e.printStackTrace();
			}
			finally
			{
				
		        fileSystem.close();
			}

[chen820655096]

自己顶一下。。。求助

[人生偌只如初见]

删除这个 /fileserver/230/230_201406190911170002.rar 也不行吗？

[chen820655096]

引用 2 楼 wulinshishen 的回复:

删除这个 /fileserver/230/230_201406190911170002.rar 也不行吗？

删除可以，但是我想做断点续传

[人生偌只如初见]

引用 3 楼 chen820655096 的回复:

[quote=引用 2 楼 wulinshishen 的回复:] 删除这个 /fileserver/230/230_201406190911170002.rar 也不行吗？

删除可以，但是我想做断点续传[/quote] /fileserver/230/230_201406190911170002.rar 这个文件是干什么的？你的执行过程是什么？

[chen820655096]

引用 4 楼 wulinshishen 的回复:

[quote=引用 3 楼 chen820655096 的回复:] [quote=引用 2 楼 wulinshishen 的回复:] 删除这个 /fileserver/230/230_201406190911170002.rar 也不行吗？

删除可以，但是我想做断点续传[/quote] /fileserver/230/230_201406190911170002.rar 这个文件是干什么的？你的执行过程是什么？[/quote] 问题已经解决，但不知道原因是什么，我加了如下配置就没有报上面这个错误 String url = "hdfs://hadoop:8020" ; Configuration conf = new Configuration(); conf .set("dfs.client.block.write.replace-datanode-on-failure.policy" ,"NEVER" ); conf .set("dfs.client.block.write.replace-datanode-on-failure.enable" ,"true" ); return FileSystem. get( new URI( url), conf);

[qingyuan18]

dfs.client.block.write.replace-datanode-on-failure.enable" ,"true" --这种配置是可以在hdfs-core.xml中配置的，不用写在代码里面 从配置项文字上看似乎是每次重启job的时候如果发现dfs上有相同ID的文件的block，不会重新创建该文件

[tchqiq]

mark~还未尝试过断点续传

[jeffwinner01]

原因： 多线程写入同一个文件，文件被别的线程占用，或者没有被关闭 解决办法：用文件共享锁锁住文件