Hadoop第08天-02.hadoop sequencefile

十八掌教育 2023-01-12 22:38:34

课时名称	课时知识点
Hadoop第08天-02.hadoop sequencefile

...全文

9 回复打赏收藏转发到动态举报

写回复

回复

切换为时间正序

请发表友善的回复…

发表回复

序列文件示例使用序列文件的示例集合设置：克隆项目 cd /tmp && git clone https://github.com/sakserv/sequencefile-examples.git 构建项目 cd /tmp/sequencefile-examples && bash -x bin/build.sh 将序列文件写入 HDFS hadoop jar target/sequencefile-examples-0.0.1-SNAPSHOT.jar \ com.github.sakserv.sequencefile.SequenceFileWriter \ /tmp/seqfile_ex/seqfile.seq 从 HDFS 读取序列文件 hadoop jar target/sequencefile-examples-0.0.1-SNAPSHOT.jar

hadoop权威指南第三版（英文版）。 Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv 1. Meet Hadoop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Data! Data Storage and Analysis Comparison with Other Systems RDBMS Grid Computing Volunteer Computing A Brief History of Hadoop Apache Hadoop and the Hadoop Ecosystem Hadoop Releases What’s Covered in this Book Compatibility 2. MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 A Weather Dataset Data Format Analyzing the Data with Unix Tools Analyzing the Data with Hadoop Map and Reduce Java MapReduce Scaling Out Data Flow Combiner Functions Running a Distributed MapReduce Job Hadoop Streaming Ruby Python iii Hadoop Pipes Compiling and Running 3. The Hadoop Distributed Filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 The Design of HDFS HDFS Concepts Blocks Namenodes and Datanodes HDFS Federation HDFS High-Availability The Command-Line Interface Basic Filesystem Operations Hadoop Filesystems Interfaces The Java Interface Reading Data from a Hadoop URL Reading Data Using the FileSystem API Writing Data Directories Querying the Filesystem Deleting Data Data Flow Anatomy of a File Read Anatomy of a File Write Coherency Model Parallel Copying with distcp Keeping an HDFS Cluster Balanced Hadoop Archives Using Hadoop Archives Limitations 4. Hadoop I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 Data Integrity Data Integrity in HDFS LocalFileSystem ChecksumFileSystem Compression Codecs Compression and Input Splits Using Compression in MapReduce Serialization The Writable Interface Writable Classes iv | Table of Contents Implementing a Custom Writable Serialization Frameworks Avro File-Based Data Structures SequenceFile MapFile 5. Developing a MapReduce Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 The Configuration API Combining Resources Variable Expansion Configuring the Development Environment Managing Configuration GenericOptionsParser, Tool, and ToolRunner Writing a Unit Test Mapper Reducer Running Locally on Test Data Running a Job in a Local Job Runner Testing the Driver Running on a Cluster Packaging Launching a Job The MapReduce Web UI Retrieving the Results Debugging a Job Hadoop Logs Remote Debugging Tuning a Job Profiling Tasks MapReduce Workflows Decomposing a Problem into MapReduce Jobs JobControl Apache Oozie 6. How MapReduce Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 Anatomy of a MapReduce Job Run Classic MapReduce (MapReduce 1) YARN (MapReduce 2) Failures Failures in Classic MapReduce Failures in YARN Job Scheduling Table of Contents | v The Fair Scheduler The Capacity Scheduler Shuffle and Sort The Map Side The Reduce Side Configuration Tuning Task Execution The Task Execution Environment Speculative Execution Output Committers Task JVM Reuse Skipping Bad Records 7. MapReduce Types and Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 MapReduce Types The Default MapReduce Job Input Formats Input Splits and Records Text Input Binary Input Multiple Inputs Database Input (and Output) Output Formats Text Output Binary Output Multiple Outputs Lazy Output Database Output

1. 大数据行业现状分析与最新行业动态2. Hadoop的起源与简史（包含：Hadoop的发行版本）3. Hadoop 2.X生态体系简介：HDFS，MapReduce，Hive等4. Hadoop 3.0新特性介5. Hadoop在互联网公司的应用案例解析6. Hadoop 2.X安装部署的三种模式：集群，伪分布式，Local

升级glib解决Hadoop WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... 和 SequenceFile doesn't work with GzipCodec without native-hadoop code 问题，具体请参见博文：https://blog.csdn.net/l1028386804/article/details/88420473

先说一下环境，flume 1.9.0，hadoop 3.2.1，兼容没有问题，官方文档没什么好说的，足够详细，每一项后面都附带有例子，好评。但是在配置sink to hdfs的时候足足踩了不少雷，记录下来希望可以帮到更多的人。错误最常见的还是java.lang.NoClassDefFoundError异常，出现这个提示100%是因为flume缺少相应的组件包，下面分别说一下缺少的内容，找到对应jar之后复制到{FLUME_HOME}/lib下即可。 org/apache/hadoop/io/SequenceFile$CompressionType 缺少hadoop-common-X.jar

徐培成的课程社区_NO_1

1

社区成员

469

社区内容

发帖

与我相关

我的任务

社区管理员

加入社区

近7日
近30日
至今

加载中

查看更多榜单

社区公告

暂无公告

试试用AI创作助手写篇文章吧

+ 用AI写文章