求助Hadoop集群模式下，编写java程序出问题。Wrong FS: hdfs://localhost:, expected: hdfs://Master:9

冰河是你 2017-08-11 09:28:16

已将core-site.xml和hdfs-site.xml两个文件放在eclipse的bin目录下，如何解决。具体报错如下
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:, expected: hdfs://Master:9000
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:648)
at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)
at Chapter.main(Chapter.java:25)

...全文

541 1 打赏收藏转发到动态举报

写回复

1 条回复

切换为时间正序

请发表友善的回复…

发表回复

冰河是你 2017-08-12

打赏
举报

回复

已解决，是代码問题。

Spark streaming应用运行7天之后，自动退出，日志显示token for xxx(用户名): HDFS_DELEGATION_TOKEN owner=xxxx@xxxx.com, renewer=yarn, realUser=, issueDate=1581323654722, maxDate=1581928454722, sequenceNumber=6445344, masterKeyId=1583) is expired, current time: 2020-02-17 16:37:40,567+0800 expected renewal time: 2020-02-17

Title: Learning Hadoop 2 Author: Gabriele Modena, Garry Turkington Length: 316 pages Edition: 1 Language: English Publisher: Packt Publishing Publication Date: 2014-12-29 ISBN-10: 1783285516 ISBN-13: 9781783285518 Design and implement data processing, lifecycle management, and analytic workflows with the cutting-edge toolbox of Hadoop 2 About This Book Construct state-of-the-art applications using higher-level interfaces and tools beyond the traditional MapReduce approach Use the unique features of Hadoop 2 to model and analyze Twitter's global stream of user generated data Develop a prototype on a local cluster and deploy to the cloud (Amazon Web Services) Who This Book Is For If you are a system or application developer interested in learning how to solve practical problems using the Hadoop framework, then this book is ideal for you. You are expected to be familiar with the Unix/Linux command-line interface and have some experience with the Java programming language. Familiarity with Hadoop would be a plus. In Detail This book introduces you to the world of building data-processing applications with the wide variety of tools supported by Hadoop 2. Starting with the core components of the framework?HDFS and YARN?this book will guide you through how to build applications using a variety of approaches. You will learn how YARN completely changes the relationship between MapReduce and Hadoop and allows the latter to support more varied processing approaches and a broader array of applications. These include real-time processing with Apache Samza and iterative computation with Apache Spark. Next up, we discuss Apache Pig and the dataflow data model it provides. You will discover how to use Pig to analyze a Twitter dataset. With this book, you will be able to make your life easier by using tools such as Apache Hive, Apache Oozie, Hadoop Streaming, Apache Crunch, and Kite SDK. The last part of this book discusses the likely future direction of major Hadoop components and how to get involved with the Hadoop community. Table of Contents Chapter 1. Introduction Chapter 2. Storage Chapter 3. Processing – Mapreduce And Beyond Chapter 4. Real-Time Computation With Samza Chapter 5. Iterative Computation With Spark Chapter 6. Data Analysis With Apache Pig Chapter 7. Hadoop And Sql Chapter 8. Data Lifecycle Management Chapter 9. Making Development Easier Chapter 10. Running A Hadoop Cluster Chapter 11. Where To Go Next

Doug Cutting, Hadoop’s creator, likes to call Hadoop the kernel for big data, and I’d tend to agree. With its distributed storage and compute capabilities, Hadoop is fundamentally an enabling technology for working with huge datasets. Hadoop, to me, provides a bridge between structured (RDBMS) and unstructured (log files, XML, text) data, and allows these datasets to be easily joined together. This has evolved from traditional use cases, such as combining OLTP and log files, to more sophisticated uses, such as using Hadoop for data warehousing (exemplified by Facebook) and the field of data science, which studies and makes new discoveries about data. This book collects a number of intermediary and advanced Hadoop examples and presents them in a problem/solution format. Each of the 85 techniques addresses a specific task you’ll face, like using Flume to move log files into Hadoop or using Mahout for predictive analysis. Each problem is explored step by step and, as you work through them, you’ll find yourself growing more comfortable with Hadoop and at home in the world of big data. This hands-on book targets users who have some practical experience with Hadoop and understand the basic concepts of MapReduce and HDFS. Manning’s Hadoop in Action by Chuck Lam contains the necessary prerequisites to understand and apply the techniques covered in this book. Many techniques in this book are Java-based, which means readers are expected to possess an intermediate-level knowledge of Java. An excellent text for all levels of Java users is Effective Java, Second Edition, by Joshua Bloch (Addison-Wesley, 2008).

About This Book Explore the integration of Apache Spark with third party applications such as H20, Databricks and Titan Evaluate how Cassandra and Hbase can be used for storage An advanced guide with a combination of instructions and practical examples to extend the most up-to date Spark functionalities Who This Book Is For If you are a developer with some experience with Spark and want to strengthen your knowledge of how to get around in the world of Spark, then this book is ideal for you. Basic knowledge of Linux, Hadoop and Spark is assumed. Reasonable knowledge of Scala is expected. What You Will Learn Extend the tools available for processing and storage Examine clustering and classification using MLlib Discover Spark stream processing via Flume, HDFS Create a schema in Spark SQL, and learn how a Spark schema can be populated with data Study Spark based graph processing using Spark GraphX Combine Spark with H20 and deep learning and learn why it is useful Evaluate how graph storage works with Apache Spark, Titan, HBase and Cassandra Use Apache Spark in the cloud with Databricks and AWS In Detail Apache Spark is an in-memory cluster based parallel processing system that provides a wide range of functionality like graph processing, machine learning, stream processing and SQL. It operates at unprecedented speeds, is easy to use and offers a rich set of data transformations. This book aims to take your limited knowledge of Spark to the next level by teaching you how to expand Spark functionality. The book commences with an overview of the Spark eco-system. You will learn how to use MLlib to create a fully working neural net for handwriting recognition. You will then discover how stream processing can be tuned for optimal performance and to ensure parallel processing. The book extends to show how to incorporate H20 for machine learning, Titan for graph based storage, Databricks for cloud-based Spark. Intermediate Scala based code examples are provided for Apache Spark module processing in a CentOS Linux and Databricks cloud environment. Table of Contents Chapter 1: Apache Spark Chapter 2: Apache Spark Mllib Chapter 3: Apache Spark Streaming Chapter 4: Apache Spark Sql Chapter 5: Apache Spark Graphx Chapter 6: Graph-Based Storage Chapter 7: Extending Spark With H2O Chapter 8: Spark Databricks Chapter 9: Databricks Visualization

Wrong FS: hdfs expected: file: 场景：错误java.lang.IllegalArgumentException: Wrong FS: hdfs:/, expected: file:/// 异常java.lang.IllegalArgumentException: Wrong FS: hdfs:/, expected: file:///

Hadoop生态社区

20,809

社区成员

4,691

社区内容

发帖

与我相关

我的任务

社区管理员

加入社区

近7日
近30日
至今

加载中

查看更多榜单

社区公告

暂无公告

试试用AI创作助手写篇文章吧

+ 用AI写文章