hadoop单词计数器案例遇到的问题

[李花怒放一树白]

在linux使用eclipse运行的话
控制台打印如下,而且居然还能得到结果
由于输出太多我只好分多个帖子贴出来了
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hbase-0.96.0-hadoop2/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/mahout-distribution-0.6/lib/slf4j-jcl-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/mahout-distribution-0.6/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/mahout-distribution-0.6/mahout-examples-0.6-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/zookeeper-3.4.6/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2014-04-05 12:24:52,057 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-04-05 12:24:53,052 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - session.id is deprecated. Instead, use dfs.metrics.session-id
2014-04-05 12:24:53,055 INFO jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId=
2014-04-05 12:24:53,628 WARN mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(258)) - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2014-04-05 12:24:53,671 INFO input.FileInputFormat (FileInputFormat.java:listStatus(287)) - Total input paths to process : 1
2014-04-05 12:24:53,829 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(394)) - number of splits:1
2014-04-05 12:24:53,861 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - user.name is deprecated. Instead, use mapreduce.job.user.name
2014-04-05 12:24:53,864 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
2014-04-05 12:24:53,866 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2014-04-05 12:24:53,867 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
2014-04-05 12:24:53,867 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.job.name is deprecated. Instead, use mapreduce.job.name
2014-04-05 12:24:53,868 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
2014-04-05 12:24:53,869 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
2014-04-05 12:24:53,869 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
2014-04-05 12:24:53,869 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2014-04-05 12:24:53,871 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
2014-04-05 12:24:53,879 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
2014-04-05 12:24:54,330 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(477)) - Submitting tokens for job: job_local1283073663_0001
2014-04-05 12:24:54,543 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/usr/local/hadoop_workspace/HADOOP_STUDY/build/test/mapred/staging/hadoop1283073663/.staging/job_local1283073663_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2014-04-05 12:24:54,544 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/usr/local/hadoop_workspace/HADOOP_STUDY/build/test/mapred/staging/hadoop1283073663/.staging/job_local1283073663_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2014-04-05 12:24:54,558 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/usr/local/hadoop_workspace/HADOOP_STUDY/build/test/mapred/staging/hadoop1283073663/.staging/job_local1283073663_0001/job.xml:an attempt to override final parameter: hadoop.tmp.dir; Ignoring.
2014-04-05 12:24:55,121 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/usr/local/hadoop_workspace/HADOOP_STUDY/build/test/mapred/local/localRunner/hadoop/job_local1283073663_0001/job_local1283073663_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2014-04-05 12:24:55,122 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/usr/local/hadoop_workspace/HADOOP_STUDY/build/test/mapred/local/localRunner/hadoop/job_local1283073663_0001/job_local1283073663_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2014-04-05 12:24:55,134 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/usr/local/hadoop_workspace/HADOOP_STUDY/build/test/mapred/local/localRunner/hadoop/job_local1283073663_0001/job_local1283073663_0001.xml:an attempt to override final parameter: hadoop.tmp.dir; Ignoring.
2014-04-05 12:24:55,155 INFO mapreduce.Job (Job.java:submit(1272)) - The url to track the job: http://localhost:8080/
2014-04-05 12:24:55,156 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1317)) - Running job: job_local1283073663_0001
2014-04-05 12:24:55,157 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(323)) - OutputCommitter set in config null
2014-04-05 12:24:55,174 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(341)) - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2014-04-05 12:24:55,332 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(389)) - Waiting for map tasks
2014-04-05 12:24:55,335 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(216)) - Starting task: attempt_local1283073663_0001_m_000000_0
2014-04-05 12:24:55,395 INFO mapred.Task (Task.java:initialize(581)) - Using ResourceCalculatorProcessTree : [ ]
2014-04-05 12:24:55,402 INFO mapred.MapTask (MapTask.java:runNewMapper(732)) - Processing split: hdfs://192.168.3.77:8020/chapter1/wordcount/word.txt:0+88
2014-04-05 12:24:55,417 INFO mapred.MapTask (MapTask.java:createSortingCollector(387)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2014-04-05 12:24:55,509 INFO mapred.MapTask (MapTask.java:setEquator(1183)) - (EQUATOR) 0 kvi 26214396(104857584)
2014-04-05 12:24:55,510 INFO mapred.MapTask (MapTask.java:init(975)) - mapreduce.task.io.sort.mb: 100
2014-04-05 12:24:55,510 INFO mapred.MapTask (MapTask.java:init(976)) - soft limit at 83886080
2014-04-05 12:24:55,511 INFO mapred.MapTask (MapTask.java:init(977)) - bufstart = 0; bufvoid = 104857600
2014-04-05 12:24:55,511 INFO mapred.MapTask (MapTask.java:init(978)) - kvstart = 26214396; length = 6553600
2014-04-05 12:24:55,616 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(513)) -
2014-04-05 12:24:55,623 INFO mapred.MapTask (MapTask.java:flush(1440)) - Starting flush of map output
2014-04-05 12:24:55,623 INFO mapred.MapTask (MapTask.java:flush(1459)) - Spilling map output
2014-04-05 12:24:55,624 INFO mapred.MapTask (MapTask.java:flush(1460)) - bufstart = 0; bufend = 136; bufvoid = 104857600
2014-04-05 12:24:55,624 INFO mapred.MapTask (MapTask.java:flush(1462)) - kvstart = 26214396(104857584); kvend = 26214348(104857392); length = 49/6553600
2014-04-05 12:24:55,646 INFO mapred.MapTask (MapTask.java:sortAndSpill(1648)) - Finished spill 0
2014-04-05 12:24:55,653 INFO mapred.Task (Task.java:done(995)) - Task:attempt_local1283073663_0001_m_000000_0 is done. And is in the process of committing
2014-04-05 12:24:55,677 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(513)) - map
2014-04-05 12:24:55,678 INFO mapred.Task (Task.java:sendDone(1115)) - Task 'attempt_local1283073663_0001_m_000000_0' done.
2014-04-05 12:24:55,678 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(241)) - Finishing task: attempt_local1283073663_0001_m_000000_0
2014-04-05 12:24:55,678 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(397)) - Map task executor complete.
2014-04-05 12:24:55,708 INFO mapred.Task (Task.java:initialize(581)) - Using ResourceCalculatorProcessTree : [ ]
2014-04-05 12:24:55,716 INFO mapred.Merger (Merger.java:merge(568)) - Merging 1 sorted segments
2014-04-05 12:24:55,721 INFO mapred.Merger (Merger.java:merge(667)) - Down to the last merge-pass, with 1 segments left of total size: 160 bytes
2014-04-05 12:24:55,723 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(513)) -
2014-04-05 12:24:55,755 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords

[有情男]

我在linux上按照cookbook书上的步骤 编译书上给的代码 在命令行上运行 输出看上去就比较正常 没有出现eclipse上那么多警告和忽略配置 感觉楼主应该是eclipse配置的classpath有问题造成的

[李花怒放一树白]

2014-04-05 12:24:55,930 INFO mapred.Task (Task.java:done(995)) - Task:attempt_local1283073663_0001_r_000000_0 is done. And is in the process of committing 2014-04-05 12:24:55,936 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(513)) - 2014-04-05 12:24:55,936 INFO mapred.Task (Task.java:commit(1156)) - Task attempt_local1283073663_0001_r_000000_0 is allowed to commit now 2014-04-05 12:24:55,951 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1283073663_0001_r_000000_0' to hdfs://192.168.3.77/chapter1/wordcount/out/_temporary/0/task_local1283073663_0001_r_000000 2014-04-05 12:24:55,953 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(513)) - reduce > reduce 2014-04-05 12:24:55,954 INFO mapred.Task (Task.java:sendDone(1115)) - Task 'attempt_local1283073663_0001_r_000000_0' done. 2014-04-05 12:24:56,159 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1338)) - Job job_local1283073663_0001 running in uber mode : false 2014-04-05 12:24:56,161 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1345)) - map 100% reduce 100% 2014-04-05 12:24:56,164 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1356)) - Job job_local1283073663_0001 completed successfully 2014-04-05 12:24:56,191 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1363)) - Counters: 32 File System Counters FILE: Number of bytes read=546 FILE: Number of bytes written=409518 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=176 HDFS: Number of bytes written=81 HDFS: Number of read operations=13 HDFS: Number of large read operations=0 HDFS: Number of write operations=4 Map-Reduce Framework Map input records=4 Map output records=13 Map output bytes=136 Map output materialized bytes=168 Input split bytes=117 Combine input records=0 Combine output records=0 Reduce input groups=9 Reduce shuffle bytes=0 Reduce input records=13 Reduce output records=9 Spilled Records=26 Shuffled Maps =0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=404750336 File Input Format Counters Bytes Read=88 File Output Format Counters Bytes Written=81 ======================以上是linux的eclipse运行后控制台的输出======================== ======================接下来是windows的eclipse运行后控制台输出,前后相差挺大====================== SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/D:/hadoop2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/D:/hbase-0.96.0-hadoop2/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/D:/mahout-distribution-0.6/mahout-examples-0.6-job.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/D:/mahout-distribution-0.6/lib/slf4j-jcl-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/D:/mahout-distribution-0.6/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/D:/zookeeper-3.4.6/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2014-04-09 11:45:34,291 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-04-09 11:45:44,279 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - session.id is deprecated. Instead, use dfs.metrics.session-id 2014-04-09 11:45:44,281 INFO jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId= 2014-04-09 11:45:44,716 WARN mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(258)) - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2014-04-09 11:45:44,748 INFO input.FileInputFormat (FileInputFormat.java:listStatus(287)) - Total input paths to process : 1 2014-04-09 11:45:45,043 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(394)) - number of splits:1 2014-04-09 11:45:45,168 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - user.name is deprecated. Instead, use mapreduce.job.user.name 2014-04-09 11:45:45,170 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 2014-04-09 11:45:45,171 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used 2014-04-09 11:45:45,171 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 2014-04-09 11:45:45,171 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.job.name is deprecated. Instead, use mapreduce.job.name 2014-04-09 11:45:45,172 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class 2014-04-09 11:45:45,172 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 2014-04-09 11:45:45,172 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 2014-04-09 11:45:45,172 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 2014-04-09 11:45:45,173 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 2014-04-09 11:45:45,176 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 2014-04-09 11:45:45,389 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(477)) - Submitting tokens for job: job_local1495588904_0001 2014-04-09 11:45:45,511 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/E:/hadoop_workspace/HADOOP_STU/build/test/mapred/staging/vip1495588904/.staging/job_local1495588904_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2014-04-09 11:45:45,512 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/E:/hadoop_workspace/HADOOP_STU/build/test/mapred/staging/vip1495588904/.staging/job_local1495588904_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2014-04-09 11:45:45,521 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/E:/hadoop_workspace/HADOOP_STU/build/test/mapred/staging/vip1495588904/.staging/job_local1495588904_0001/job.xml:an attempt to override final parameter: hadoop.tmp.dir; Ignoring. 2014-04-09 11:45:45,633 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(439)) - Cleaning up the staging area file:/E:/hadoop_workspace/HADOOP_STU/build/test/mapred/staging/vip1495588904/.staging/job_local1495588904_0001 Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:435) at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:977) at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:177) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:164) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:98) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:285) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:131) at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:157) at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:636) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286) at chapter1.src.chapter1.WordCount.main(WordCount.java:103) 由此可见,这其中还是有很多问题,那我应该如何一步一步的去排查和修改,求大神指点!!!