诚讯通网站口碑营销的步骤
请先完成HDFS的前置部署，部署方式可查看:本地部署HDFS集群
https://blog.csdn.net/m0_73641796/article/details/145998092?spm=1001.2014.3001.5502
部署说明
| 组件 | 配置文件 | 启动进程 | 备注 |
| Hadoop HDFS | 需修改 | 需启动: NameNode作为主节点 DataNode作为从节点 | 分布式文件系统 |
| Hadoop YARN | 需修改 | 需启动 ProxyServer代理服务器提供安全性 JobHistoryServer记录历史信息和日志 | 分布式资源调度 |
| Hadoop MapReduce | 需修改 | 无需启动任何进程 MapReduce程序运行在YARN容器内 | 分布式数据计算 |
集群规划
| 主机 | 角色 |
| node1 | ResourceManager ProxyServer JobHistoryServer |
| node2 | NodeManager |
| node3 | NodeManager |
修改MapReduce配置文件
在$HADOOP_HOME/etc/hadoop文件夹内，修改mapred-env.sh文件，添加如下环境变量
export JAVA_HOME=/export/server/jdk
export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=1000
export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA修改mapred-site.xml文件，添加如下配置信息
<configuration><property><name>mapreduce.framework.name</name><value>yarn</value><description></description></property><property><name>mapreduce.jobhistory.address</name><value>node1:10020</value><description></description></property><property><name>mapreduce.jobhistory.webapp.address</name><value>node1:19888</value><description></description></property><property><name>mapreduce.jobhistory.intermediate-done-dir</name><value>/data/mr-history/tmp</value><description></description></property><property><name>mapreduce.jobhistory.done-dir</name><value>/data/mr-history/done</value><description></description></property><property><name>yarn.app.mapreduce.am.env</name><value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value></property><property><name>mapreduce.map.env</name><value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value></property><property><name>mapreduce.reduce.env</name><value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value></property>
</configuration>修改YARN配置文件
在$HADOOP_HOME/etc/hadoop文件夹内，修改yarn-env.sh文件，添加如下环境变量
export JAVA_HOME=/export/server/jdk
export HADOOP_HOME=/export/server/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
修改yarn-site.xml文件，添加如下内容：
<configuration><!-- Site specific YARN configuration properties -->
<property><name>yarn.log.server.url</name><value>http://node1:19888/jobhistory/logs</value><description></description>
</property><property><name>yarn.web-proxy.address</name><value>node1:8089</value><description>proxy server hostname and port</description></property><property><name>yarn.log-aggregation-enable</name><value>true</value><description>Configuration to enable or disable log aggregation</description></property><property><name>yarn.nodemanager.remote-app-log-dir</name><value>/tmp/logs</value><description>Configuration to enable or disable log aggregation</description></property><!-- Site specific YARN configuration properties --><property><name>yarn.resourcemanager.hostname</name><value>node1</value><description></description></property><property><name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value><description></description></property><property><name>yarn.nodemanager.local-dirs</name><value>/data/nm-local</value><description>Comma-separated list of paths on the local filesystem where intermediate data is written.</description></property><property><name>yarn.nodemanager.log-dirs</name><value>/data/nm-log</value><description>Comma-separated list of paths on the local filesystem where logs are written.</description></property><property><name>yarn.nodemanager.log.retain-seconds</name><value>10800</value><description>Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled.</description></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value><description>Shuffle service that needs to be set for Map Reduce applications.</description></property>
</configuration>分发配置文件
scp mapred-env.sh mapred-site.xml yarn-env.sh yarn-site.xml node2:`pwd`/
scp mapred-env.sh mapred-site.xml yarn-env.sh yarn-site.xml node3:`pwd`/启动HDFS
start-dfs.sh启动YARN
# 一键启动YARN集群
start-yarn.sh# 一键停止YARN集群
stop-yarn.sh# 在当前机器，单独启动或停止进程
yarn --daemon start|stop resourcemanager|nodemanager|proxyserver# 历史服务器启动和停止
mapred --daemon start|stop historyserver可通过jps命令查看当前运行的进程
查看YARN的WEB UI页面
打开 http://node1:8088 即可看到YARN集群的监控页面（ResourceManager的WEB UI）