spark java api 开发_JAVA_编程开发_程序员俱乐部

?spark,又一个传说中的分布式实现,详情:http://spark-project.org/,

安装这里就不写了，因为网上已有中文介绍，这里主要是介绍一下入门，和hadoop一样，学习的时候，首先学习spark提供的字符统计例子：javaWordCount

原始代码如下：

1. import?scala.Tuple2;??
2. import?spark.api.java.JavaPairRDD;??
3. import?spark.api.java.JavaRDD;??
4. import?spark.api.java.JavaSparkContext;??
5. import?spark.api.java.function.FlatMapFunction;??
6. import?spark.api.java.function.Function2;??
7. import?spark.api.java.function.PairFunction;??
8. ??
9. import?java.util.Arrays;??
10. import?java.util.List;??
11. ??
12. public?class?JavaWordCount?{??
13. ??public?static?void?main(String[]?args)?throws?Exception?{??
14. ????if?(args.length?<?2)?{??
15. ??????System.err.println("Usage:?JavaWordCount?<master>?<file>");??
16. ??????System.exit(1);??
17. ????}??
18. ??
19. ????JavaSparkContext?ctx?=?new?JavaSparkContext(args[0],?"JavaWordCount",??
20. ????????System.getenv("SPARK_HOME"),?System.getenv("SPARK_EXAMPLES_JAR"));??
21. ????JavaRDD<String>?lines?=?ctx.textFile(args[1],?1);??
22. ??
23. ????JavaRDD<String>?words?=?lines.flatMap(new?FlatMapFunction<String,?String>()?{??
24. ??????public?Iterable<String>?call(String?s)?{??
25. ????????return?Arrays.asList(s.split("?"));??
26. ??????}??
27. ????});??
28. ??????
29. ????JavaPairRDD<String,?Integer>?ones?=?words.map(new?PairFunction<String,?String,?Integer>()?{??
30. ??????public?Tuple2<String,?Integer>?call(String?s)?{??
31. ????????return?new?Tuple2<String,?Integer>(s,?1);??
32. ??????}??
33. ????});??
34. ??????
35. ????JavaPairRDD<String,?Integer>?counts?=?ones.reduceByKey(new?Function2<Integer,?Integer,?Integer>()?{??
36. ??????public?Integer?call(Integer?i1,?Integer?i2)?{??
37. ????????return?i1?+?i2;??
38. ??????}??
39. ????});??
40. ??
41. ????List<Tuple2<String,?Integer>>?output?=?counts.collect();??
42. ????for?(Tuple2?tuple?:?output)?{??
43. ??????System.out.println(tuple._1?+?":?"?+?tuple._2);??
44. ????}??
45. ????System.exit(0);??
46. ??}??
47. }??

? 运行： ./run spark/examples/JavaWordCount? local input.txt
?local：不解析，自己查

input.txt:文件类容

1. Hello?World?Bye?World?goole??

?运行的结果和haddoop中运行的JavaWordCount? 一样

?

1. goole:?1??
2. World:?2??
3. Hello:?1??
4. Bye:?1?