多语言网站开发公司佛山网络

当前位置： 首页 > news >正文

多语言网站开发公司,佛山网络,潮品服饰网站建设规划书,长春 网站 设计hadoop高可用#xff0c;依赖于zookeeper。 用于生产环境, 企业部署必须的模式. 1. 部署环境规划 1.1. 虚拟机及hadoop角色划分 主机名称 namenode datanode resourcemanager nodemanager zkfc journalnode zookeeper master slave1 slave2 1.2. 软件版本 java …hadoop高可用依赖于zookeeper。 用于生产环境, 企业部署必须的模式.  1. 部署环境规划 1.1. 虚拟机及hadoop角色划分 主机名称 namenode datanode resourcemanager nodemanager zkfc journalnode zookeeper master slave1 slave2
1.2. 软件版本 java jdk-1.8 Hadoop 3.3.0 zookeeper 3.7.0
1.3. 数据目录规划 名称 目录 namenode目录 /data/hadoop/dfs/name datanode目录 /data/hadoop/dfs/data hadoop临时目录 /data/hadoop/tmp zookeeper数据目录 /data/zookeeper/data

1. 免密登录 略
2. 安装jdk 略
3. zookeeper安装 4.1. 解压 解压到目录/usr/local/ 下 tar -zxvf apache-zookeeper-3.7.0-bin.tar.gz -C /usr/local/zookeeper 4.2. 环境配置 cat/etc/profile EOF export ZOOKEEPER_HOME/usr/local/zookeeper/apache-zookeeper-3.7.0-bin export PATH$ZOOKEEPER_HOME/bin:$PATH EOF source /etc/profile #创建数据/日志目录 mkdir -pv /data/zookeeper/{data,log}
   4.3. 修改配置文件 cd /usr/local/zookeeper/apache-zookeeper-3.7.0-bin/conf/ cp zoo_sample.cfg zoo.cfg 修改zoo.cfg配置文件 dataDir/data/zookeeper/data/ dataLogDir/data/zookeeper/log/ server.1master:2888:3888 server.2slave1:2888:3888 server.3slave2:2888:3888 分发到slave1,slave2节点 scp zoo.cfg slave1:/usr/local/zookeeper/apache-zookeeper-3.7.0-bin/conf/ scp zoo.cfg slave2:/usr/local/zookeeper/apache-zookeeper-3.7.0-bin/conf/ 4.4. 创建myid 根据服务器对应的数字配置相应的myidmaster配置1slave1配置2slave2配置3 #各节点配置根据server.1就是1 echo 1 /data/zookeeper/data/myid 4.5. 启动zookeeper 各个节点启动 zkServer.sh start zkServer.sh status
4. hadoop安装 5.1. 解压 tar -zxvf hadoop-3.3.0.tar.gz -C /usr/local/ 5.2. 环境配置 环境配置所有节点都执行root用户执行 chown -R hadoop:hadoop /usr/local/hadoop-3.3.0 cat/etc/profile EOF export HADOOP_HOME/usr/local/hadoop-3.3.0 export PATH$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH EOF source /etc/profile 5.3. 修改配置文件 5.3.1. hadoop-env.sh cd $HADOOP_HOME/etc/hadoop vi hadoop-env.shexport JAVA_HOME/usr/java/jdk1.8.0_311 5.3.2. core-site.xml configurationpropertynamefs.defaultFS/namevaluehdfs://mycluster//valuedescription自定义的集群名称/description/propertypropertynamehadoop.tmp.dir/namevalue/data/hadoop/tmp/valuedescriptionnamenode上本地的hadoop临时文件夹/description/propertypropertynameha.zookeeper.quorum/namevaluemaster:2181,slave1:2181,slave2:2181/valuedescription指定zookeeper地址/description/propertypropertynameha.zookeeper.session-timeout.ms/namevalue1000/valuedescriptionhadoop链接zookeeper的超时时长设置ms/description/property /configuration 5.3.3. hdfs-site.xml configurationpropertynamedfs.replication/namevalue2/valuedescriptionHadoop的备份系数是指每个block在hadoop集群中有几份系数越高冗余性越好占用存储也越多/description/propertypropertynamedfs.namenode.name.dir/namevalue/data/hadoop/dfs/name/valuedescriptionnamenode上存储hdfs名字空间元数据 /description/propertypropertynamedfs.datanode.data.dir/namevalue/data/hadoop/dfs/data/valuedescriptiondatanode上数据块的物理存储位置/description/propertypropertynamedfs.webhdfs.enabled/namevaluetrue/value/property!–指定hdfs的nameservice为myha01需要和core-site.xml中的保持一致dfs.ha.namenodes.[nameservice id]为在nameservice中的每一个NameNode设置唯一标示符。配置一个逗号分隔的NameNode ID列表。这将是被DataNode识别为所有的NameNode。例如如果使用myha01作为nameservice ID并且使用nn1和nn2作为NameNodes标示符–propertynamedfs.nameservices/namevaluemycluster/value/property!– myha01下面有两个NameNode分别是nn1nn2 –propertynamedfs.ha.namenodes.mycluster/namevaluenn1,nn2/value/property!– nn1的RPC通信地址 –propertynamedfs.namenode.rpc-address.mycluster.nn1/namevaluemaster:9000/value/property!– nn1的http通信地址 –propertynamedfs.namenode.http-address.mycluster.nn1/namevaluemaster:50070/value/property!– nn2的RPC通信地址 –propertynamedfs.namenode.rpc-address.mycluster.nn2/namevalueslave1:9000/value/property!– nn2的http通信地址 –propertynamedfs.namenode.http-address.mycluster.nn2/namevalueslave1:50070/value/property!– 指定NameNode的edits元数据的共享存储位置。也就是JournalNode列表该url的配置格式qjournal://host1:port1;host2:port2;host3:port3/journalIdjournalId推荐使用nameservice默认端口号是8485 –propertynamedfs.namenode.shared.edits.dir/namevalueqjournal://master:8485;slave1:8485;slave2:8485/mycluster/value/property!– 指定JournalNode在本地磁盘存放数据的位置 –propertynamedfs.journalnode.edits.dir/namevalue/data/hadoop/data/journaldata/value/property!– 开启NameNode失败自动切换 –propertynamedfs.ha.automatic-failover.enabled/namevaluetrue/value/property!– 配置失败自动切换实现方式 –propertynamedfs.client.failover.proxy.provider.mycluster/namevalueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value/property!– 配置隔离机制方法多个机制用换行分割即每个机制暂用一行 –propertynamedfs.ha.fencing.methods/namevaluesshfenceshell(/bin/true)/value/property!– 使用sshfence隔离机制时需要ssh免登陆 –propertynamedfs.ha.fencing.ssh.private-key-files/namevalue/home/hadoop/.ssh/id_rsa/value/property!– 配置sshfence隔离机制超时时间 –propertynamedfs.ha.fencing.ssh.connect-timeout/namevalue30000/value/propertypropertynameha.failover-controller.cli-check.rpc-timeout.ms/namevalue60000/value/property /configuration 注意 mycluster  所有地方都要一样 5.3.4. mapred-site.xml configurationpropertynamemapreduce.framework.name/namevalueyarn/valuedescriptionThe runtime framework for executing MapReduce jobs. Can be one of local, classic or yarn./descriptionfinaltrue/final/propertypropertynamemapreduce.jobtracker.http.address/namevaluemaster:50030/value/propertypropertynamemapreduce.jobhistory.address/namevaluemaster:10020/value/propertypropertynamemapreduce.jobhistory.webapp.address/namevaluemaster:19888/value/propertypropertynamemapred.job.tracker/namevaluehttp://master:9001/value/property /configuration 5.3.5. yarn-site.xml configuration!– 开启RM高可用 –propertynameyarn.resourcemanager.ha.enabled/namevaluetrue/value/property!– 指定RM的cluster id –propertynameyarn.resourcemanager.cluster-id/namevalueyrc/value/property!– 指定RM的名字 –propertynameyarn.resourcemanager.ha.rm-ids/namevaluerm1,rm2/value/property!– 分别指定RM的地址 –propertynameyarn.resourcemanager.hostname.rm1/namevalueslave1/value/propertypropertynameyarn.resourcemanager.hostname.rm2/namevalueslave2/value/property!– 指定zk集群地址 –propertynameyarn.resourcemanager.zk-address/namevaluemaster:2181,slave1:2181,slave2:2181/value/propertypropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/propertypropertynameyarn.log-aggregation-enable/namevaluetrue/value/propertypropertynameyarn.log-aggregation.retain-seconds/namevalue86400/value/property!– 启用自动恢复 –propertynameyarn.resourcemanager.recovery.enabled/namevaluetrue/value/property!– 制定resourcemanager的状态信息存储在zookeeper集群上 –propertynameyarn.resourcemanager.store.class/namevalueorg.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore/value/propertypropertynameyarn.application.classpath/name value/usr/local/hadoop-3.3.0/etc/hadoop:/usr/local/hadoop-3.3.0/share/hadoop/common/lib/:/usr/local/hadoop-3.3.0/share/hadoop/common/:/usr/local/hadoop-3.3.0/share/hadoop/hdfs:/usr/local/hadoop-3.3.0/share/hadoop/hdfs/lib/:/usr/local/hadoop-3.3.0/share/hadoop/hdfs/:/usr/local/hadoop-3.3.0/share/hadoop/mapreduce/:/usr/local/hadoop-3.3.0/share/hadoop/yarn:/usr/local/hadoop-3.3.0/share/hadoop/yarn/lib/:/usr/local/hadoop-3.3.0/share/hadoop/yarn/*/value/property /configuration 5.3.6. workers vim workers master slave1 slave2 5.4. 分发到其他服务器 scp -r /usr/local/hadoop-3.3.0/ slave1:/usr/local/ scp -r /usr/local/hadoop-3.3.0/ slave2:/usr/local/
5. 启动集群 以下顺序不能错 6.1. 启动journalnode(所有节点) hadoop-daemon.sh start journalnode 6.2. 格式化namenodemaster hadoop namenode -format 6.3. 同步元数据 scp -r /data/hadoop/dfs/name/current/ rootslave1:/data/hadoop/dfs/name/ 6.4. 格式化zkfcmaster hdfs zkfc -formatZK 6.5. 启动HDFSmaster start-yarn.sh 6.6. 查看各主节点状态hdfs/yarn hdfs haadmin -getServiceState nn1 hdfs haadmin -getServiceState nn2 yarn rmadmin -getServiceState rm1 yarn rmadmin -getServiceState rm2
6. 查看页面 hdfshttp://master:9870 接付费咨询调bug 10元一次。v644789108