ceph-ansible 安装社区版本
初始化。设置静态ip，关闭防火墙，时间同步等等，实际还有内核调优啥的，就不写了。都一样的。。搞一个初始化系统脚本跑一下就行了，，这里我只记录我的ceph集群安装过程，额！！！！没有ansible 基础，“幂等性” 就是说以前有过变更的，重新执行ansible-playbook,不会改变原有。。。会不知道我在做什么。。。有空再说吧。
官网：https://docs.ceph.com/projects/ceph-ansible/en/latest/
安装结构：
| 系统版本 | ip地址 | 服务角色 |
|---|---|---|
| Red Hat Enterprise Linux release 8.3 | 192.168.5.111-servera | ansible-control,mon,osd,grafana-server,client |
| Red Hat Enterprise Linux release 8.3 | 192.168.5.112-serverb | mon,mgr,osd |
| Red Hat Enterprise Linux release 8.3 | 192.168.5.113-serverc | mon,mgr,osd |
| Red Hat Enterprise Linux release 8.3 | 192.168.5.114-serverd | mgr,osd,grafana-server |
虽然我用的是红帽八的镜像，，但是我并没有订阅ansible和ceph ，装的是社区版本
初始化：
磁盘结构，定义osd
[root@servera ]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 8.8G 0 rom
nvme0n1 259:0 0 20G 0 disk
├─nvme0n1p1 259:1 0 1G 0 part /boot
└─nvme0n1p2 259:2 0 19G 0 part
├─rhel-root 253:0 0 17G 0 lvm /
└─rhel-swap 253:1 0 2G 0 lvm [SWAP]
nvme0n2 259:3 0 6G 0 disk
nvme0n3 259:4 0 6G 0 disk
nvme0n4 259:5 0 10G 0 disk
**网络配置： **
[root@servera ~]# ifconfig
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.5.111 netmask 255.255.255.0 broadcast 192.168.5.255
inet6 fe80::89e5:c213:b513:7546 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:b9:08:16 txqueuelen 1000 (Ethernet)
RX packets 1057054 bytes 777431925 (741.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1126861 bytes 959515474 (915.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 38562 bytes 45558189 (43.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 38562 bytes 45558189 (43.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
用户配置：
四台机器
student ,用于ansible-user
ceph, ceph集群管理
useradd student && echo "student "|passwd --stdin student
useradd ceph && echo "ceph "|passwd --stdin ceph
visudo
student ALL=(ALL) NOPASSWD: ALL
ceph ALL=(ALL) NOPASSWD: ALL
hosts解析
[student@servera ~]$ sudo cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.5.111 servera
192.168.5.112 serverb
192.168.5.113 serverc
192.168.5.114 serverd
[student@servera ~]$
ansible配置：
前置条件配置一下ceph的yum仓库，不然会从download.ceph.com下载会很慢，可以使用阿里云加速：
[root@serverb ~]# cat /etc/yum.repos.d/ceph.repo
[Ceph]
name=Ceph packages for $basearch
baseurl=https://mirrors.aliyun.com/ceph/rpm-octopus/el8/$basearch
enabled=1
gpgcheck=0
type=rpm-md
[Ceph-noarch]
name=Ceph noarch packages
baseurl=https://mirrors.aliyun.com/ceph/rpm-octopus/el8/noarch
enabled=1
gpgcheck=0
type=rpm-md
[ceph-source]
name=Ceph source packages
baseurl=https://mirrors.aliyun.com/ceph/rpm-octopus/el8/SRPMS
enabled=1
gpgcheck=0
type=rpm-md
https://mirrors.aliyun.com/ceph/rpm-octopus/el8/
或者改这里：
[student@servera ceph-defaults]$ cat defaults/main.yml |grep ceph_mirror -A 4 -B 4
# REPOSITORY: COMMUNITY VERSION
#
# Enabled when ceph_repository == 'community'
#
ceph_mirror: http://download.ceph.com
ceph_stable_key: https://download.ceph.com/keys/release.asc
ceph_stable_release: octopus
ceph_stable_repo: "{{ ceph_mirror }}/debian-{{ ceph_stable_release }}"
nfs_ganesha_stable: true # use stable repos for nfs-ganesha
nfs_ganesha_stable_branch: V3.3-stable
nfs_ganesha_stable_deb_repo: "{{ ceph_mirror }}/nfs-ganesha/deb-{{ nfs_ganesha_stable_branch }}/{{ ceph_stable_release }}"
# Use the option below to specify your applicable package tree, eg. when using non-LTS Ubuntu versions
# # for a list of available Debian distributions, visit http://download.ceph.com/debian-{{ ceph_stable_release }}/dists/
ansible 安装：
yum -y install epel-release
yum -y install ansible
下载社区版ceph-ansible包
官网有git 也一样的colne 下来
wget https://github.com/ceph/ceph-ansible/archive/refs/tags/v5.0.12.tar.gz
tar -zxvf ceph-ansible-5.0.12.tar.gz
servera免密配置：
ssh-keygen 一路回车
[student@servera ~]$ for i in servera serverb serverc serverd
> do
> ssh-copy-id student@$i
> ssh-copy-id ceph@$i
> done
配置ansible 的cfg
[root@servera ~]# grep -Ev '^$|^#' /etc/ansible/ansible.cfg
[defaults]
inventory = /etc/ansible/hosts
remote_user = student
log_path = /home/student/ceph-ansible/ansible.log
[inventory]
[privilege_escalation]
become=True
become_method=sudo
become_user=root
become_ask_pass=False
[student@servera group_vars]$ sudo cat /etc/ansible/ansible.cfg |grep log_path
log_path = /home/student/ceph-ansible/ansible.log
[student@servera group_vars]$
配置主机清单
[student@servera ~]$ sudo cat /etc/ansible/hosts
[mons]
servera
serverb
serverc
[mgrs]
serverb
serverd
serverc
[mdss]
serverb
[osds]
serverb
serverd
serverc
servera
[grafana-server]
servera
serverd
[clients]
servera
拷贝基本的yml文件
[student@servera ceph-ansible-5.0.12]$ sudo cp site.yml.sample site.yml
[student@servera ceph-ansible-5.0.12]$ cd group_vars/
[student@servera group_vars]$ sudo cp mons.yml.sample mons.yml
[student@servera group_vars]$ sudo cp osds.yml.sample osds.yml
[student@servera group_vars]$ sudo cp mgrs.yml.sample mgrs.yml
[student@servera group_vars]$ sudo cp mdss.yml.sample mdss.yml
[student@servera group_vars]$ sudo cp
配置all.yml，这里就是ceph集群的配置文件了
[student@servera group_vars]$ sudo cp all.yml.sample all.yml
[student@servera group_vars]$ sudo cat all.yml
fetch_directory: ~/ceph-ansible-keys #key 文件存放路径
ntp_service_enabled: false #ntp 禁止，因为自己配置了
ceph_origin: repository #镜像仓库 本地仓库
ceph_repository: community
ceph_stable_release: octopus #版本
monitor_interface: eth160 #网卡
journal_size: 5120 # OSD journal size in MB
public_network: 172.17.10.0/24
cluster_network: 192.168.5.0/24
ceph_conf_overrides:
global:
mon_osd_allow_primary_affinity: 1
mon_clock_drift_allowed: 0.5
osd_pool_default_size: 2 最大副本数
osd_pool_default_min_size: 1 最小副本数
mon_pg_warn_min_per_osd: 0
mon_pg_warn_max_per_osd: 0
mon_pg_warn_max_object_skew: 0
#mds_max_mds: 1
client:
rbd_default_features: 1
mds_max_mds: 1
dashboard_enabled: True
dashboard_admin_user: admin
dashboard_admin_password: password
grafana_admin_user: admin
grafana_admin_password: password
[student@servera group_vars]$
编辑osds.yml,这里我用的是同类型磁盘，没有分开日志盘和存储盘采用幷置配置
[student@servera group_vars]$ grep -Ev '^$|^#' osds.yml
---
dummy:
copy_admin_key: true
devices:
- /dev/nvme0n2
- /dev/nvme0n3
- /dev/nvme0n4
osd_auto_discovery: false
如非幷置，需定义osd 日志存储盘，是一一对应的，，可以使多块数据盘的日志存放到一个高速的ssd盘中
---
dummy:
osd_scenario: "non-collocated" #非幷置
devices:
- /dev/vdb #数据盘
- /dev/vdc
dedicated_devices:
- /dev/vdd #日志盘
- /dev/vdd
直接跑ansible-playbook
[student@servera ceph-ansible-5.0.12]$ ansible-playbook site.yml
略。。。。。。。。。。。。。。。。。。
riday 07 April 2023 09:34:28 +0800 (0:00:05.005) 0:13:41.337 **********
===============================================================================
ceph-infra : open ceph networks on monitor ------------------------------------------------------------------------------------------------------- 472.35s
check for python --------------------------------------------------------------------------------------------------------------------------------- 217.88s
install python for RedHat based OS - dnf ---------------------------------------------------------------------------------------------------------- 36.73s
ceph-infra : install firewalld python binding ----------------------------------------------------------------------------------------------------- 14.53s
ceph-infra : start firewalld ----------------------------------------------------------------------------------------------------------------------- 9.72s
gather and delegate facts -------------------------------------------------------------------------------------------------------------------------- 6.78s
ceph-infra : open monitor and manager ports -------------------------------------------------------------------------------------------------------- 5.01s
ceph-handler : check if the ceph osd socket is in-use ---------------------------------------------------------------------------------------------- 3.44s
gather facts --------------------------------------------------------------------------------------------------------------------------------------- 3.36s
ceph-validate : get devices information ------------------------------------------------------------------------------------------------------------ 3.14s
ceph-facts : check for a ceph mon socket ----------------------------------------------------------------------------------------------------------- 2.61s
ceph-facts : resolve device link(s) ---------------------------------------------------------------------------------------------------------------- 1.97s
ceph-facts : check if the ceph mon socket is in-use ------------------------------------------------------------------------------------------------ 1.43s
ceph-facts : check if podman binary is present ----------------------------------------------------------------------------------------------------- 1.40s
ceph-infra : check firewalld installation on redhat or SUSE/openSUSE ------------------------------------------------------------------------------- 1.38s
ceph-facts : get current fsid if cluster is already running ---------------------------------------------------------------------------------------- 1.34s
ceph-handler : find ceph osd socket ---------------------------------------------------------------------------------------------------------------- 1.24s
ceph-handler : check for a ceph-crash process ------------------------------------------------------------------------------------------------------ 1.01s
check for dnf-3 package manager (RedHat/Fedora/CentOS) --------------------------------------------------------------------------------------------- 0.97s
check for yum package manager (RedHat/Fedora/CentOS) ----------------------------------------------------------------------------------------------- 0.86s
[student@servera ceph-ansible-5.0.12]$
ceph 状态：
[ceph@servera ~]$ ceph -s
cluster:
id: ae325f8b-20b3-44a9-8643-6e55e99901a2
health: HEALTH_OK
services:
mon: 3 daemons, quorum servera,serverb,serverc (age 20m)
mgr: serverb(active, since 30m), standbys: serverd, serverc
mds: cephfs:1 {0=serverb=up:active}
osd: 12 osds: 12 up (since 20m), 12 in (since 16h)
data:
pools: 3 pools, 65 pgs
objects: 22 objects, 5.3 KiB
usage: 12 GiB used, 76 GiB / 88 GiB avail
pgs: 65 active+clean
ceph 版本
[ceph@servera ~]$ ceph --version
ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)
osd 状态
[ceph@servera ~]$ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.08630 root default
-9 0.02158 host servera
6 ssd 0.00589 osd.6 up 1.00000 1.00000
10 ssd 0.00589 osd.10 up 1.00000 1.00000
11 ssd 0.00980 osd.11 up 1.00000 1.00000
-5 0.02158 host serverb
0 ssd 0.00589 osd.0 up 1.00000 1.00000
3 ssd 0.00589 osd.3 up 1.00000 1.00000
7 ssd 0.00980 osd.7 up 1.00000 1.00000
-3 0.02158 host serverc
1 ssd 0.00589 osd.1 up 1.00000 1.00000
4 ssd 0.00589 osd.4 up 1.00000 1.00000
9 ssd 0.00980 osd.9 up 1.00000 1.00000
-7 0.02158 host serverd
2 ssd 0.00589 osd.2 up 1.00000 1.00000
5 ssd 0.00589 osd.5 up 1.00000 1.00000
8 ssd 0.00980 osd.8 up 1.00000 1.00000
dashboard
[ceph@servera ~]$ ceph mgr services
{
"dashboard": "https://serverb:8443/",
"prometheus": "http://serverb:9283/"
}
grafana
ceph中常用端口号
开搞开搞。。
版权声明：
本站所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自
Emporer-Linux！
喜欢就支持一下吧