yarn wordcount计算报错：

wzwdev1980 2018-07-10 07:38:41
nodemanager报错：
couldn't find app application_1531219399016_0002 while processing FINISH_CONTAINERS event

然后控制台报错:
[hadoop@webf hadoop]$ /alidata/server/hadoop-3.0.0/bin/hadoop jar /alidata/server/hadoop-3.0.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0.jar wordcount /test/words.txt /output
2018-07-10 18:16:44,872 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1531216699747_0001
2018-07-10 18:16:46,714 INFO input.FileInputFormat: Total input files to process : 1
2018-07-10 18:16:47,451 INFO mapreduce.JobSubmitter: number of splits:1
2018-07-10 18:16:47,560 INFO Configuration.deprecation: yarn.resourcemanager.zk-address is deprecated. Instead, use hadoop.zk.address
2018-07-10 18:16:47,561 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2018-07-10 18:16:49,494 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1531216699747_0001
2018-07-10 18:16:49,497 INFO mapreduce.JobSubmitter: Executing with tokens: []
2018-07-10 18:16:50,483 INFO conf.Configuration: resource-types.xml not found
2018-07-10 18:16:50,484 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2018-07-10 18:16:50,826 INFO impl.YarnClientImpl: Submitted application application_1531216699747_0001
2018-07-10 18:16:50,963 INFO mapreduce.Job: The url to track the job: http://hd4.wzw.com:8088/proxy/application_1531216699747_0001/
2018-07-10 18:16:50,964 INFO mapreduce.Job: Running job: job_1531216699747_0001
2018-07-10 18:22:52,899 INFO mapreduce.Job: Job job_1531216699747_0001 running in uber mode : false
2018-07-10 18:22:52,902 INFO mapreduce.Job: map 0% reduce 0%
2018-07-10 18:22:52,934 INFO mapreduce.Job: Job job_1531216699747_0001 failed with state FAILED due to: Application application_1531216699747_0001 failed 2 times due to Error launching appattempt_1531216699747_0001_000002. Got exception: java.net.ConnectException: Call From webf.com/127.0.0.1 to webf.com:39839 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor43.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495)
at org.apache.hadoop.ipc.Client.call(Client.java:1437)
at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy88.startContainers(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:128)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy89.startContainers(Unknown Source)
at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:123)
at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:304)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:685)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:788)
at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:409)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1552)
at org.apache.hadoop.ipc.Client.call(Client.java:1383)
... 19 more
. Failing the application.
2018-07-10 18:22:52,997 INFO mapreduce.Job: Counters: 0
...全文
781 2 打赏 收藏 转发到动态 举报
写回复
用AI写文章
2 条回复
切换为时间正序
请发表友善的回复…
发表回复
迷途1503 2018-07-13
  • 打赏
  • 举报
 回复
Call From webf.com/127.0.0.1 to webf.com:39839 failed on connection exception: java.net.ConnectException:

webf.com是什么？39839端口是做什么的
wzwdev1980 2018-07-11
  • 打赏
  • 举报
 回复
配置文件：
yarn-site.xml
<?xml version="1.0"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at



    http://www.apache.org/licenses/LICENSE-2.0



  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<configuration>

	<property>

		<name>yarn.resourcemanager.connect.retry-interval.ms</name>

		<value>2000</value>

		<description>RM失联后重新连接的时间</description>

	</property>

	<property>

		<name>yarn.resourcemanager.ha.enabled</name>

		<value>true</value>

		<description>启用RM高可用</description>			

	</property>

	<property>

		<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>

		<value>true</value>

		<description>启用自动故障转移</description>

	</property>

	<property>

		<name>yarn.resourcemanager.ha.automatic-failover.embedded</name>

		<value>true</value>

		<description>启用内置的自动故障转移</description>		

	</property>

	<property>

		<name>yarn.resourcemanager.cluster-id</name>

		<value>rmcluster</value>

		<description>RM集群标识符</description>

	</property>

	<property>

		<name>yarn.resourcemanager.ha.rm-ids</name>

		<value>rm1,rm2,rm3</value>

		<description>两台ResourceManager主机</description>

	</property>

	<property>

		<name>yarn.resourcemanager.hostname.rm1</name>

		<value>hd4.wzw.com</value>

		<description>RM主机地址</description>

	</property>	

	<property>

		<name>yarn.resourcemanager.scheduler.address.rm1</name>

		<value>hd4.wzw.com:8030</value>

		<description>RM对ApplicationMaster暴露的访问地址。ApplicationMaster通过该地址向RM申请资源、释放资源等</description>	

	</property>

	<property>

		<name>yarn.resourcemanager.resource-tracker.address.rm1</name>

		<value>hd4.wzw.com:8031</value>

		<description>RM对NodeManager暴露的地址。NodeManager通过该地址向RM汇报心跳，领取任务等</description>	

	</property>

	<property>

		<name>yarn.resourcemanager.address.rm1</name>

		<value>hd4.wzw.com:8032</value>

		<description>RM对客户端暴露的地址。客户端通过该地址向RM提交应用程序、杀死应用程序等</description>	

	</property>

	<property>

		<name>yarn.resourcemanager.admin.address.rm1</name>

		<value>hd4.wzw.com:8033</value>

		<description>对管理员暴露的访问地址。管理员通过该地址向RM发送管理命令等</description>	

	</property>

	<property>

		<name>yarn.resourcemanager.webapp.address.rm1</name>

		<value>hd4.wzw.com:8088</value>

		<description>rm1对外的web服务地址。用户可通过该地址在浏览器中查看集群各类信息</description>

	</property>

	<property>

		<name>yarn.resourcemanager.hostname.rm2</name>

		<value>hd5.wzw.com</value>

		<description>RM主机地址</description>

	</property>	

	<property>

		<name>yarn.resourcemanager.scheduler.address.rm2</name>

		<value>hd5.wzw.com:8030</value>

		<description>RM对ApplicationMaster暴露的访问地址。ApplicationMaster通过该地址向RM申请资源、释放资源等</description>	

	</property>

	<property>

		<name>yarn.resourcemanager.resource-tracker.address.rm2</name>

		<value>hd5.wzw.com:8031</value>

		<description>RM对NodeManager暴露的地址。NodeManager通过该地址向RM汇报心跳，领取任务等</description>	

	</property>

	<property>

		<name>yarn.resourcemanager.address.rm2</name>

		<value>hd5.wzw.com:8032</value>

		<description>RM对客户端暴露的地址。客户端通过该地址向RM提交应用程序、杀死应用程序等</description>	

	</property>

	<property>

		<name>yarn.resourcemanager.admin.address.rm2</name>

		<value>hd5.wzw.com:8033</value>

		<description>对管理员暴露的访问地址。管理员通过该地址向RM发送管理命令等</description>	

	</property>

	<property>

		<name>yarn.resourcemanager.webapp.address.rm2</name>

		<value>hd5.wzw.com:8088</value>

		<description>rm2对外的HTTP访问地址。用户可通过该地址在浏览器中查看集群各类信息</description>

	</property>

	<property>

		<name>yarn.resourcemanager.hostname.rm3</name>

		<value>hd6.wzw.com</value>

		<description>RM主机地址</description>

	</property>	

	<property>

		<name>yarn.resourcemanager.scheduler.address.rm3</name>

		<value>hd6.wzw.com:8030</value>

		<description>RM对ApplicationMaster暴露的访问地址。ApplicationMaster通过该地址向RM申请资源、释放资源等</description>	

	</property>

	<property>

		<name>yarn.resourcemanager.resource-tracker.address.rm3</name>

		<value>hd6.wzw.com:8031</value>

		<description>RM对NodeManager暴露的地址。NodeManager通过该地址向RM汇报心跳，领取任务等</description>	

	</property>

	<property>

		<name>yarn.resourcemanager.address.rm3</name>

		<value>hd6.wzw.com:8032</value>

		<description>RM对客户端暴露的地址。客户端通过该地址向RM提交应用程序、杀死应用程序等</description>	

	</property>

	<property>

		<name>yarn.resourcemanager.admin.address.rm3</name>

		<value>hd6.wzw.com:8033</value>

		<description>对管理员暴露的访问地址。管理员通过该地址向RM发送管理命令等</description>	

	</property>

	<property>

		<name>yarn.resourcemanager.webapp.address.rm3</name>

		<value>hd6.wzw.com:8088</value>

		<description>rm3对外的HTTP访问地址。用户可通过该地址在浏览器中查看集群各类信息</description>

	</property>

	<property>

		<name>yarn.resourcemanager.recovery.enabled</name>

		<value>true</value>

		<description>启用RM重启的功能，默认为false</description>

	</property>

	<property>	   

	   <name>yarn.resourcemanager.store.class</name>

	   <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>

	   <description>RM状态信息存储方式，一种基于内存(MemStore)，另一种基于ZK(ZKStore)</description>

	 </property>

	 <property>

		<name>yarn.resourcemanager.zk-address</name>

		<value>hd4.wzw.com:2181,hd5.wzw.com:2181,hd6.wzw.com:2181</value>

		<description>ZK集群地址</description>

	</property>

	<property>

		<name>yarn.resourcemanager.zk.state-store.address</name>

		<value>hd4.wzw.com:2181,hd5.wzw.com:2181,hd6.wzw.com:2181</value>

		<description>ZK集群保存状态信息</description>

	</property>

	<property>

		<name>yarn.nodemanager.aux-services</name>

		<value>mapreduce_shuffle</value>

		<description>NodeManager上运行的附属服务。需配置成mapreduce_shuffle，才可运行MapReduce程序</description>

	</property>

	<property>

		<name>yarn.resourcemanager.resource-tracker.client.thread-count</name>

		<value>50</value>

		<description>处理来自NodeManager的RPC请求的Handler数目，默认值50</description>

	</property>

	<property>

		<name>yarn.resourcemanager.scheduler.client.thread-count</name>

		<value>50</value>

		<description>处理来自ApplicationMaster的RPC请求的Handler数目，默认值50</description>

	</property>	

	<property>  

		<name>yarn.nodemanager.resource.cpu-vcores</name>  

		<value>2</value>  

		<description>NodeManager总的可用虚拟CPU个数，默认值8</description>

	</property> 

	<property>  

		<name>yarn.nodemanager.vmem-pmem-ratio</name>  

		<value>2.1</value>  

		<description>每使用1MB物理内存，最多可用的虚拟内存数，默认值2.1</description>

	</property> 

	<property>  

		<name>yarn.nodemanager.resource.memory-mb</name>  

		<value>1536</value>  

		<description>NodeManager总的可用物理内存。该参数一旦设置，整个运行过程中不可动态修改。默认值是8192MB，即使你的机器内存不够8192MB，YARN也会按照这些内存来使用（傻不傻），因此这个值通常一定要配置。不过Apache已经正在尝试将该参数做成可动态修改的</description>

	</property>  

	<property>  

		<name>yarn.scheduler.minimum-allocation-mb</name>  

		<value>512</value>  

		<description>运行MapReduce作业每个Task可申请的最小内存资源量，默认值1024</description>

	</property> 

	<property>  

		<name>yarn.scheduler.maximum-allocation-mb</name>  

		<value>1536</value>  

		<description>运行MapReduce作业每个Task可申请的最大内存资源量，默认值8192</description>

	</property> 

	<property>  

		<name>yarn.scheduler.minimum-allocation-vcores</name>  

		<value>1</value>  

		<description>运行MapReduce作业每个Task可申请的虚拟CPU资源量，默认值1</description>

	</property> 

	<property>  

		<name>yarn.scheduler.maximum-allocation-vcores</name>  

		<value>2</value>  

		<description>运行MapReduce作业每个Task可申请的虚拟CPU资源量，默认值32</description>

	</property>

	<property>  

		<name>yarn.app.mapreduce.am.resource.mb</name>  

		<value>512</value>  

		<description>MapReduce ApplicationMaster使用的内存</description>

	</property>

	<property>  

		<name>yarn.app.mapreduce.am.command-opts</name>  

		<value>-Xmx409m</value>  

		<description>MapReduce ApplicationMaster传递给虚拟机的启动参数</description>

	</property>	

</confi


mapred-site.xml
<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at



    http://www.apache.org/licenses/LICENSE-2.0



  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->



<!-- Put site-specific property overrides in this file. -->



<configuration>

	<property>

		<name>mapreduce.framework.name</name>

		<value>yarn</value>

		<description>分布式计算框架</description>  

	</property>	

	<property>

		<name>mapreduce.jobhistory.address</name>

		<value>0.0.0.0:10020</value>

		<description>MapReduce JobHistory Server地址</description>  

	</property>

	<property>

		<name>mapreduce.jobhistory.webapp.address</name>

		<value>0.0.0.0:19888</value>

		<description>MapReduce JobHistory Server Web UI地址</description>  

	</property>

	<property>

		<name>mapred.job.tracker.http.address</name>

		<value>0.0.0.0:50030</value>

		<description>JobTracker的HTTP服务器和端口</description>  

	</property>

	<property>

		<name>mapred.task.tracker.http.address</name>

		<value>0.0.0.0:50060</value>

		<description>TaskTracker的HTTP服务器和端口</description>  

	</property>

	<property>

		<name>mapreduce.map.memory.mb</name>

		<value>512</value>

		<description>每个Map Task需要的内存量</description>  

	</property>

	<property>

		<name>mapreduce.map.java.opts</name>

		<value>-Xmx409m</value>

		<description>每个Map Task传递给虚拟机的启动参数</description>  

	</property>

	<property>

		<name>mapreduce.reduce.memory.mb</name>

		<value>1024</value>

		<description>每个Map Task需要的内存量</description>  

	</property>

	<property>

		<name>mapreduce.reduce.java.opts</name>

		<value>-Xmx819m</value>

		<description>每个Map Task传递给虚拟机的启动参数</description>  

	</property>

	<property>

		<name>mapreduce.task.io.sort.mb</name>

		<value>204</value>

		<description>Mapper中的Kvbuffer的大小</description>  

	</property>

</configuration>
YARN & Docker在Hulu的实战
Docker实战视频教程，YARN是Hadoop 2.0中的分布式资源管理系统，能够对集群中的各类计算框架进行统一的管理和调度，同时使用轻量级隔离方案cgroups对应用程序进行隔离，避免彼此之间的干扰。然而，目前YARN对环境...
Flink: Wordcount应用程序报错：ClassNotFoundException: yarn
Flink: Wordcount应用程序报错：ClassNotFoundException: yarn.exceptions.YarnException 大数据近年来，随着大数据技术的快速发展，人们对于处理海量数据的需求也日益增长。Apache Flink作为一种流式计算框架，在大数据处理领域得到了广泛应用。然而，有时候在使用Flink时会遇到一些错误和异常情况，例如Wordcount应用程序报错：ClassNotFoundException: yarn.exceptions.YarnException。本
hadoop：IDEA本地编写mapreducer的wordcount并测试，并上传到hadoop的linux服务器进行测试
直接上代码： package main.java; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io...
IDEA报错：org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory file:/D:/output already
D:\develop\Java\jdk1.8.0_202\bin\java.exe “-javaagent:F:\IDEA\IntelliJ IDEA 2019.3.3\lib\idea_rt.jar=7676:F:\IDEA\IntelliJ IDEA 2019.3.3\bin” -Dfile.encoding=UTF-8 -classpath D:\develop\Java\jdk1.8.0_...
在Hadoop下运行wordcount一直报错系统找不到指定路径
在Hadoop下运行MapReduce任务欢迎使用Markdown编辑器新的改变功能快捷键合理的创建标题，有助于目录的生成如何改变文本的样式插入链接与图片如何插入一段漂亮的代码片生成一个适合你的列表创建一个表格设定内容居中、居左、居右SmartyPants创建一个自定义列表如何创建一个注脚注释也是必不可少的KaTeX数学公式新的甘特图功能，丰富你的文章UML 图表FLowchart流程图导出与导入导出导入 在分布式计算课上老师要求在windows10上安装hadoop并运行wordcount任务，以下是运行
Spark

1,261

社区成员

1,169

社区内容

发帖
与我相关
我的任务
社区描述
Spark由Scala写成，是UC Berkeley AMP lab所开源的类Hadoop MapReduce的通用的并行计算框架，Spark基于MapReduce算法实现的分布式计算。
社区管理员
  • Spark
  • shiter
加入社区
  • 近7日
  • 近30日
  • 至今

加载中

查看更多榜单
社区公告
暂无公告

试试用AI创作助手写篇文章吧

+ 用AI写文章