DistributedLzoIndexer使用的问题，求助！

[昔_日_的_光]

在使用lzo压缩的时候，调用DistributedLzoIndexer，运行过程中存在很多的问题。求助使用过lzo压缩的大牛们。
先贴出调用的代码：

	int result = job.waitForCompletion(true) ? 0 : 1;
	System.out.println(new String[] { outputPath });
	DistributedLzoIndexer.main(new String[] { outputPath });
	System.exit(result);
	return 0;

相应的pom文件依赖：
<dependency>
<groupId>org.anarres.lzo</groupId>
<artifactId>lzo-hadoop</artifactId>
<version>1.0.1</version>
</dependency>
执行时候报的错误日志：

14/11/08 15:44:48 INFO lzo.DistributedLzoIndexer: Adding LZO file hdfs://hdpnn:9000/group/dataproducts/zhubao/resys/proj/resys_log_2_test_temp/step_0_merge_data/part-r-00499.lzo to indexing list (no index currently exists)
CONSOLE# 14/11/08 15:44:49 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
CONSOLE# 14/11/08 15:45:01 INFO input.FileInputFormat: Total input paths to process : 500
CONSOLE# 14/11/08 15:45:01 INFO mapred.JobClient: Running job: job_201401091245_100017381
CONSOLE# 14/11/08 15:45:03 INFO mapred.JobClient: map 0% reduce 0%
CONSOLE# 14/11/08 15:45:28 INFO mapred.JobClient: Task Id : attempt_201401091245_100017381_m_000209_0, Status : FAILED
CONSOLE# java.lang.NullPointerException
CONSOLE# at com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:49)
CONSOLE# at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:464)
CONSOLE# at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:669)
CONSOLE# at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
CONSOLE# at org.apache.hadoop.mapred.Child.main(Child.java:165)
CONSOLE# 
CONSOLE# 14/11/08 15:45:28 INFO mapred.JobClient: Task Id : attempt_201401091245_100017381_m_000387_0, Status : FAILED
CONSOLE# java.lang.NullPointerException
CONSOLE# at com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:49)
CONSOLE# at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:464)
CONSOLE# at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:669)
CONSOLE# at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
CONSOLE# at org.apache.hadoop.mapred.Child.main(Child.java:165)

相应的出错地方的源码：

  
public void initialize(InputSplit genericSplit, TaskAttemptContext taskAttemptContext) throws IOException {
    context = taskAttemptContext;
    FileSplit fileSplit = (FileSplit)genericSplit;
    lzoFile = fileSplit.getPath();
    // The LzoSplitInputFormat is not splittable, so the split length is the whole file.
    totalFileSize = fileSplit.getLength();

    // Jump through some hoops to create the lzo codec.
    Configuration conf = context.getConfiguration();
    CompressionCodecFactory factory = new CompressionCodecFactory(conf);
    CompressionCodec codec = factory.getCodec(lzoFile);
    ((Configurable)codec).setConf(conf);

    LzopDecompressor lzopDecompressor = (LzopDecompressor)codec.createDecompressor();
    FileSystem fs = lzoFile.getFileSystem(conf);
    rawInputStream = fs.open(lzoFile);

    // Creating the LzopInputStream here just reads the lzo header for us, nothing more.
    // We do the rest of our input off of the raw stream is.
    codec.createInputStream(rawInputStream, lzopDecompressor);

    // This must be called AFTER createInputStream is called, because createInputStream
    // is what reads the header, which has the checksum information.  Otherwise getChecksumsCount
    // erroneously returns zero, and all block offsets will be wrong.
    numChecksums = lzopDecompressor.getChecksumsCount();
  }

其中报错的应该是这两行：

CompressionCodec codec = factory.getCodec(lzoFile);
((Configurable)codec).setConf(conf);

另外顺便告诉大家，reduce的outputpath里面的文件超过900多的时候，可能会导致index失败。是因为源码中，将path放在数组里面，导致数组内存不足，运行失败。

盼望大牛帮助。拜谢了。

[人生偌只如初见]

看下configuration中有没有io.compression.codecs这个参数，也可以在程序中添加如下内容试试 conf.set("io.compression.codecs", "org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,com.hadoop.compression.lzo.LzopCodec"); conf.set("io.compression.codec.lzo.class", "com.hadoop.compression.lzo.LzoCodec");

[skyWalker_ONLY]

参考官方的说法http://grokbase.com/p/hadoop/common-user/107wn90ere/error-caused-by-java-lang-classnotfoundexception-com-hadoop-compression-lzo-lzopcodec，应该是配置文件有问题

[昔_日_的_光]

引用 1 楼 wulinshishen 的回复:

看下configuration中有没有io.compression.codecs这个参数，也可以在程序中添加如下内容试试 conf.set("io.compression.codecs", "org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,com.hadoop.compression.lzo.LzopCodec"); conf.set("io.compression.codec.lzo.class", "com.hadoop.compression.lzo.LzoCodec");

谢谢，这个肯定有配置的。

[昔_日_的_光]

引用 1 楼 wulinshishen 的回复:

看下configuration中有没有io.compression.codecs这个参数，也可以在程序中添加如下内容试试 conf.set("io.compression.codecs", "org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,com.hadoop.compression.lzo.LzopCodec"); conf.set("io.compression.codec.lzo.class", "com.hadoop.compression.lzo.LzoCodec");

谢谢，我看看，并试试。

[昔_日_的_光]

找到错误了。是因为我直接将压缩包打入源码，导致的设置错误。改成下面的就好了。

	int result = job.waitForCompletion(true) ? 0 : 1;
	DistributedLzoIndexer lzoIndexer = new DistributedLzoIndexer();
	Configuration indexConf = new Configuration();
	indexConf.set("io.compression.codecs",
		"com.hadoop.compression.lzo.LzopCodec");
	lzoIndexer.setConf(indexConf);
	lzoIndexer.run(new String[] { outputPath });
	System.exit(result);
	return 0;