RDD的五个特性之3

[wux_labs]

A list of dependencies on other RDDs，RDD之间的依赖关系。RDD的每次转换都会生成一个新的RDD，所以RDD之间就会形成类似于流水线一样的前后依赖关系。

rdd1 = sc.textFile("hdfs://node1:8020/words.txt")
rdd2 = rdd1.flatMap(lambda x: x.split(" "))
rdd3 = rdd2.map(lambda x: (x, 1))
rdd4 = rdd3.reduceByKey(lambda a, b: a + b)
rdd4.collect()
print(rdd4.toDebugString())

执行代码，输出内容为：

(2) PythonRDD[15] at collect at /tmp/ipykernel_43451/2680249566.py:5 []
 |  MapPartitionsRDD[14] at mapPartitions at PythonRDD.scala:145 []
 |  ShuffledRDD[13] at partitionBy at NativeMethodAccessorImpl.java:0 []
 +-(2) PairwiseRDD[12] at reduceByKey at /tmp/ipykernel_43451/2680249566.py:4 []
    |  PythonRDD[11] at reduceByKey at /tmp/ipykernel_43451/2680249566.py:4 []
    |  hdfs://node1:8020/words.txt MapPartitionsRDD[10] at textFile at NativeMethodAccessorImpl.java:0 []
    |  hdfs://node1:8020/words.txt HadoopRDD[9] at textFile at NativeMethodAccessorImpl.java:0 []