桂城网站制作公司网站生成
1.8 Spark编程入门
1.8.1 通过IDEA创建Spark工程
ps:工程创建之前步骤省略,在scala中已经讲解,直接默认是创建好工程的 导入Pom文件依赖
<!-- 声明公有的属性 --><properties><maven.compiler.source>1.8</maven.compiler.source><maven.compiler.target>1.8</maven.compiler.target><encoding>UTF-8</encoding><scala.version>2.12.8</scala.version><spark.version>3.1.2</spark.version><hadoop.version>3.2.1</hadoop.version><scala.compat.version>2.12</scala.compat.version></properties>
<!-- 声明并引入公有的依赖 --><dependencies><dependency><groupId>org.scala-lang</groupId><artifactId>scala-library</artifactId><version>${scala.version}</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-core_2.12</artifactId><version>${spark.version}</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>${hadoop.version}</version></dependency></dependencies>
<!-- 配置构建信息 --><build><!-- 资源文件夹 --><sourceDirectory>src/main/scala</sourceDirectory><!-- 声明并引入构建的插件 --><plugins><!-- 用于编译Scala代码到class --><plugin><groupId>net.alchim31.maven</groupId><artifactId>scala-maven-plugin</artifactId><version>3.2.2</version><executions><execution><goals><goal>compile</goal><goal>testCompile</goal></goals><configuration><args><arg>-dependencyfile</arg><arg>${project.build.directory}/.scala_dependencies</arg></args></configuration></execution></executions></plugin><plugin><!-- 程序打包 --><groupId>org.apache.maven.plugins</groupId><artifactId>maven-shade-plugin</artifactId><version>2.4.3</version><executions><execution><phase>package</phase><goals><goal>shade</goal></goals><configuration><!-- 过滤掉以下文件，不打包 ：解决包重复引用导致的打包错误--><filters><filter><artifact>*:*</artifact><excludes><exclude>META-INF/*.SF</exclude><exclude>META-INF/*.DSA</exclude><exclude>META-INF/*.RSA</exclude></excludes></filter></filters><transformers><!-- 打成可执行的jar包 的主方法入口--><transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"><mainClass></mainClass></transformer></transformers></configuration></execution></executions></plugin></plugins></build>
1.8.2 Scala实现WordCount
package com.qianfeng.sparkcore
import org.apache.spark.{SparkConf, SparkContext}
/*** 使用Spark统计单词个数*/
object Demo01_SparkWC {def main(args: Array[String]): Unit = {//1、获取spark上下文环境 local[n] : n代表cpu核数，*代表可用的cpu数量；如果打包服务器运行，则需要注释掉.setMaster()val conf = new SparkConf().setAppName("spark-wc").setMaster("local[*]")val sc = new SparkContext(conf)//2、初始化数据val rdd = sc.textFile("/Users/liyadong/data/sparkdata/test.txt")//3、对数据进行加工val sumRDD = rdd.filter(_.length >= 10).flatMap(_.split("\t")).map((_, 1)).reduceByKey(_ + _)//4、对数据进行输出println(sumRDD.collect().toBuffer)sumRDD.foreach(println(_))
//5、关闭sc对象sc.stop()}
}
1.8.3 程序打包上传集群
在Spark安装目录中的bin目录进行提交作业操作
spark-submit \
--class com.qianfeng.sparkcore.Demo01_SparkWC \
--master yarn \
--deploy-mode client \
/home/original-hn-bigdata-1.0.jar hdfs://qianfeng01:9820/words hdfs://qianfeng01:9820/output/0901注意:如果HDFS集群中有数据文件直接使用集群的数据文件即可，如果没有的话使用【hdfs dfs -put /home/words /】从Linux系统中将文件上传到HDFS，查看集群中运行之后的结果【hdfs dfs -tail output/0901/*】
Guff_hys_python数据结构,大数据开发学习,python实训项目-CSDN博客