Hadoop调用JNI出错

电灯泡 2012-07-30 04:48:38
本人在用hadoop集群调用JNI(C++)的时候出错,看样子应该是由于slave的环境问题而导致找不到相应的.so文件,但各种环境配置如下都已经试过,还是不能解决,有什么思路去调试或者解决吗?谢谢。
conf.set("mapred.map.child.java.opts", "-Djava.library.path=/home/hadoop/");
// conf.set("mapreduce.map.env", "LD_LIBRARY_PATH=/home/user/c_workspace/aurixminer/Debug/");
// conf.set("mapred.child.env", "LD_LIBRARY_PATH=/home/hadoop/:/home/hadoop/:/opt/Aurix_audio_miner/lib");

syslog logs

2012-07-30 12:31:06,541 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2012-07-30 12:31:06,609 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /home/hadoop/temp/mapred/local/taskTracker/distcache/-5775899438045898939_-1554142946_756173787/172.16.17.102/libs/Aurix_audio_miner.zip <- /home/hadoop/temp/mapred/local/taskTracker/hadoop/jobcache/job_201207271309_0024/attempt_201207271309_0024_m_000000_0/work/aurixminer
2012-07-30 12:31:06,616 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /home/hadoop/temp/mapred/local/taskTracker/distcache/-5382690226256483786_-469324545_1488550505/172.16.17.102/libs/libaurixminer.so <- /home/hadoop/temp/mapred/local/taskTracker/hadoop/jobcache/job_201207271309_0024/attempt_201207271309_0024_m_000000_0/work/libaurixminer.so
2012-07-30 12:31:06,675 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /home/hadoop/temp/mapred/local/taskTracker/hadoop/jobcache/job_201207271309_0024/jars/.job.jar.crc <- /home/hadoop/temp/mapred/local/taskTracker/hadoop/jobcache/job_201207271309_0024/attempt_201207271309_0024_m_000000_0/work/.job.jar.crc
2012-07-30 12:31:06,679 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /home/hadoop/temp/mapred/local/taskTracker/hadoop/jobcache/job_201207271309_0024/jars/META-INF <- /home/hadoop/temp/mapred/local/taskTracker/hadoop/jobcache/job_201207271309_0024/attempt_201207271309_0024_m_000000_0/work/META-INF
2012-07-30 12:31:06,684 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /home/hadoop/temp/mapred/local/taskTracker/hadoop/jobcache/job_201207271309_0024/jars/job.jar <- /home/hadoop/temp/mapred/local/taskTracker/hadoop/jobcache/job_201207271309_0024/attempt_201207271309_0024_m_000000_0/work/job.jar
2012-07-30 12:31:06,688 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /home/hadoop/temp/mapred/local/taskTracker/hadoop/jobcache/job_201207271309_0024/jars/pbi <- /home/hadoop/temp/mapred/local/taskTracker/hadoop/jobcache/job_201207271309_0024/attempt_201207271309_0024_m_000000_0/work/pbi
2012-07-30 12:31:06,818 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2012-07-30 12:31:06,901 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2012-07-30 12:31:06,947 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@e41bc3
2012-07-30 12:31:07,022 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
2012-07-30 12:31:07,199 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2012-07-30 12:31:07,251 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
2012-07-30 12:31:07,252 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName hadoop for UID 1001 from the native implementation
2012-07-30 12:31:07,256 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.UnsatisfiedLinkError: pbi.audioproc.audioengine.AurixFileBasedEngine.ntIndexAudioFiles([Ljava/lang/String;Ljava/lang/String;II)[Ljava/lang/String;
at pbi.audioproc.audioengine.AurixFileBasedEngine.ntIndexAudioFiles(Native Method)
at pbi.audioproc.audioengine.AurixFileBasedEngine.indexAudioFiles(Unknown Source)
at pbi.audioproc.file.AudioIndexMapper.map(Unknown Source)
at pbi.audioproc.file.AudioIndexMapper.map(Unknown Source)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

...全文
182 1 打赏 收藏 转发到动态 举报
写回复
用AI写文章
1 条回复
切换为时间正序
请发表友善的回复…
发表回复
ancanglee 2012-09-11
  • 打赏
  • 举报
回复
注意每个分布式SLAVE点都应该有相应的SO包。

20,807

社区成员

发帖
与我相关
我的任务
社区描述
Hadoop生态大数据交流社区,致力于有Hadoop,hive,Spark,Hbase,Flink,ClickHouse,Kafka,数据仓库,大数据集群运维技术分享和交流等。致力于收集优质的博客
社区管理员
  • 分布式计算/Hadoop社区
  • 涤生大数据
加入社区
  • 近7日
  • 近30日
  • 至今
社区公告
暂无公告

试试用AI创作助手写篇文章吧