Greetings,
From your master k8s node can you resolve ceph-mon.ceph.svc.cluster.local?
Please also send the output of 'cat /etc/resolv.conf' from your k8s nodes (genesis and master node). Thxs ________________________________ From: Tu, Qiaolin (NSB - CN/Hangzhou) <qiaolin.tu@nokia-sbell.com> Sent: Monday, November 12, 2018 4:39 AM To: Matthew H; airship-discuss@lists.airshipit.org Cc: Li, Maxwell (NSB - CN/Hangzhou) Subject: RE: [Airship-discuss] Airship installation Questions Hi, Add ceph rbd image related logs. root@cab23-r720-11:/var/lib/kubelet# kubectl exec -it ceph-mon-qqjzz -n ceph -- rbd ls kubernetes-dynamic-pvc-43083980-e65a-11e8-bc31-2aa831f6c6fd kubernetes-dynamic-pvc-4396e109-e65a-11e8-bc31-2aa831f6c6fd kubernetes-dynamic-pvc-443ecaac-e65a-11e8-bc31-2aa831f6c6fd root@cab23-r720-11:/var/lib/kubelet# kubectl exec -it ceph-mon-qqjzz -n ceph -- rbd info kubernetes-dynamic-pvc-43083980-e65a-11e8-bc31-2aa831f6c6fd rbd image 'kubernetes-dynamic-pvc-43083980-e65a-11e8-bc31-2aa831f6c6fd': size 5120 MB in 1280 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.113b74b0dc51 format: 2 features: layering flags: create_timestamp: Mon Nov 12 09:06:39 2018 root@cab23-r720-11:/var/lib/kubelet# kubectl exec -it ceph-mon-qqjzz -n ceph -- rbd info kubernetes-dynamic-pvc-4396e109-e65a-11e8-bc31-2aa831f6c6fd rbd image 'kubernetes-dynamic-pvc-4396e109-e65a-11e8-bc31-2aa831f6c6fd': size 256 MB in 64 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.113c74b0dc51 format: 2 features: layering flags: create_timestamp: Mon Nov 12 09:06:40 2018 root@cab23-r720-11:/var/lib/kubelet# kubectl exec -it ceph-mon-qqjzz -n ceph -- rbd info kubernetes-dynamic-pvc-443ecaac-e65a-11e8-bc31-2aa831f6c6fd rbd image 'kubernetes-dynamic-pvc-443ecaac-e65a-11e8-bc31-2aa831f6c6fd': size 5120 MB in 1280 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.113d74b0dc51 format: 2 features: layering flags: create_timestamp: Mon Nov 12 09:06:40 2018 root@cab23-r720-11:/var/lib/kubelet# kubectl exec -it ceph-mon-qqjzz -n ceph -- rbd status kubernetes-dynamic-pvc-43083980-e65a-11e8-bc31-2aa831f6c6fd Watchers: none root@cab23-r720-11:/var/lib/kubelet# kubectl exec -it ceph-mon-qqjzz -n ceph -- rbd status kubernetes-dynamic-pvc-4396e109-e65a-11e8-bc31-2aa831f6c6fd Watchers: none root@cab23-r720-11:/var/lib/kubelet# kubectl exec -it ceph-mon-qqjzz -n ceph -- rbd status kubernetes-dynamic-pvc-443ecaac-e65a-11e8-bc31-2aa831f6c6fd Watchers: none Best Regards! Qiaolin Tu NSB MN 5G ECE HZ CN2 SG04 Mobile: +86 138 057 59684 E-Mail: qiaolin.tu@nokia-sbell.com<mailto:qiaolin.tu@nokia-sbell.com> From: Tu, Qiaolin (NSB - CN/Hangzhou) Sent: Monday, November 12, 2018 5:27 PM To: Matthew H <matthew.heler@hotmail.com>; airship-discuss@lists.airshipit.org Cc: Li, Maxwell (NSB - CN/Hangzhou) <maxwell.li@nokia-sbell.com> Subject: RE: [Airship-discuss] Airship installation Questions Hi, Add ceph-mod logs and yaml files. root@cab23-r720-11:/var/lib/kubelet# kubectl exec -it ceph-mon-qqjzz -n ceph -- ceph -s cluster: id: 7b7576f4-3358-4668-9112-100440079807 health: HEALTH_OK services: mon: 1 daemons, quorum cab23-r720-11 mgr: cab23-r720-11(active) mds: cephfs-1/1/1 up {0=mds-ceph-mds-5f547b6fd7-sg49g=up:active} osd: 1 osds: 1 up, 1 in rgw: 1 daemon active data: pools: 18 pools, 93 pgs objects: 1164 objects, 3407 bytes usage: 374 MB used, 1023 GB / 1023 GB avail pgs: 93 active+clean root@cab23-r720-11:/var/lib/kubelet# kubectl exec -it ceph-mon-qqjzz -n ceph -- ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.00000 root default -2 1.00000 host cab23-r720-11 0 hdd 1.00000 osd.0 up 1.00000 1.00000 root@cab23-r720-11:/var/lib/kubelet# kubectl exec -it ceph-mon-qqjzz -n ceph -- ceph osd dump epoch 219 fsid 7b7576f4-3358-4668-9112-100440079807 created 2018-11-12 08:53:17.281208 modified 2018-11-12 09:06:40.314892 flags sortbitwise,recovery_deletes,purged_snapdirs crush_version 6 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client jewel min_compat_client hammer require_osd_release luminous pool 1 'rbd' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 last_change 219 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rbd removed_snaps [1~5] pool 2 'cephfs_metadata' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 4 pgp_num 4 last_change 216 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application cephfs pool 3 'cephfs_data' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 last_change 216 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application cephfs pool 4 '.rgw.root' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 last_change 56 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 5 'default.rgw.control' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 last_change 68 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 6 'default.rgw.data.root' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 last_change 79 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 7 'default.rgw.gc' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 last_change 89 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 8 'default.rgw.log' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 last_change 102 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 9 'default.rgw.intent-log' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 last_change 113 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 10 'default.rgw.meta' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 last_change 124 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 11 'default.rgw.usage' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 last_change 134 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 12 'default.rgw.users.keys' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 last_change 145 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 13 'default.rgw.users.email' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 last_change 155 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 14 'default.rgw.users.swift' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 last_change 168 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 15 'default.rgw.users.uid' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 last_change 179 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 16 'default.rgw.buckets.extra' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 last_change 191 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 17 'default.rgw.buckets.index' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 4 pgp_num 4 last_change 202 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 18 'default.rgw.buckets.data' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 last_change 214 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw max_osd 1 osd.0 up in weight 1 up_from 5 up_thru 209 down_at 0 last_clean_interval [0,0) 10.23.23.11:6800/6766 10.23.23.11:6801/6766 10.23.23.11:6802/6766 10.23.23.11:6803/6766 exists,up 02d8f692-709a-45ea-9f2c-75486e16e82b Best Regards! Qiaolin Tu NSB MN 5G ECE HZ CN2 SG04 Mobile: +86 138 057 59684 E-Mail: qiaolin.tu@nokia-sbell.com<mailto:qiaolin.tu@nokia-sbell.com> From: Tu, Qiaolin (NSB - CN/Hangzhou) Sent: Monday, November 12, 2018 4:25 PM To: 'Matthew H' <matthew.heler@hotmail.com<mailto:matthew.heler@hotmail.com>>; airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Cc: Li, Maxwell (NSB - CN/Hangzhou) <maxwell.li@nokia-sbell.com<mailto:maxwell.li@nokia-sbell.com>> Subject: RE: [Airship-discuss] Airship installation Questions Hi, Thanks very much for your help. After modify ceph replication parameters, ceph pods deployed successfully. It then deploy ucp related pods and it have below errors. Please check attachment log for detail, thanks very much! ucp airship-ucp-rabbitmq-rabbitmq-0 0/1 Init:0/2 0 1m ucp ingress-6cd5b89d5d-nmwpt 1/1 Running 0 18m ucp ingress-6cd5b89d5d-nr65b 1/1 Running 0 18m ucp ingress-error-pages-5c97bb46bb-2mvgm 1/1 Running 0 18m ucp ingress-error-pages-5c97bb46bb-wzzdz 1/1 Running 0 18m ucp mariadb-ingress-85b8556fbc-xpvwc 0/1 Running 0 1m ucp mariadb-ingress-85b8556fbc-zv72k 0/1 Running 0 1m ucp mariadb-ingress-error-pages-64f89dc697-2trh9 1/1 Running 0 1m ucp mariadb-server-0 0/1 Init:0/2 0 1m ucp postgresql-0 0/1 Init:0/1 0 1m root@cab23-r720-11:~# kubectl describe pod mariadb-ingress-85b8556fbc-xpvwc -n ucp Name: mariadb-ingress-85b8556fbc-xpvwc Namespace: ucp Node: cab23-r720-11/10.23.22.11 Start Time: Mon, 12 Nov 2018 08:05:00 +0000 Labels: application=mariadb component=ingress pod-template-hash=4164112967 release_group=airship-ucp-mariadb Annotations: configmap-bin-hash=eb36d47d8f7d7097cf6d488a61145f76dbfe5e558edf5b802153a00fc3389f0b configmap-etc-hash=3f45f1d8d3ddf5a09fbcd3036cb23bffb939cfa1225f8f1a0d79b390877710c1 Status: Running IP: 10.97.38.125 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m default-scheduler Successfully assigned mariadb-ingress-85b8556fbc-xpvwc to cab23-r720-11 Normal SuccessfulMountVolume 3m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "airship-ucp-mariadb-ingress-token-htf82" Normal SuccessfulMountVolume 3m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "mariadb-etc" Normal SuccessfulMountVolume 3m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "mariadb-bin" Normal Pulled 3m kubelet, cab23-r720-11 Container image "quay.io/stackanetes/kubernetes-entrypoint:v0.3.1" already present on machine Normal Created 3m kubelet, cab23-r720-11 Created container Normal Started 3m kubelet, cab23-r720-11 Started container Normal Pulled 3m kubelet, cab23-r720-11 Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0" already present on machine Normal Created 3m kubelet, cab23-r720-11 Created container Normal Started 3m kubelet, cab23-r720-11 Started container Warning Unhealthy 26s (x16 over 2m) kubelet, cab23-r720-11 Readiness probe failed: dial tcp 10.97.38.125:3306: getsockopt: connection refused root@cab23-r720-11:~# kubectl describe pod postgresql-0 -n ucp Name: postgresql-0 Namespace: ucp Node: cab23-r720-11/10.23.22.11 Start Time: Mon, 12 Nov 2018 08:04:56 +0000 Labels: application=postgresql component=server controller-revision-hash=postgresql-566fd45fd7 release_group=airship-ucp-postgresql statefulset.kubernetes.io/pod-name=postgresql-0 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulAttachVolume 4m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-c66d1b6f-e64a-11e8-bb43-080027f45d2a" Normal Scheduled 4m default-scheduler Successfully assigned postgresql-0 to cab23-r720-11 Normal SuccessfulMountVolume 4m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "postgresql-bin" Normal SuccessfulMountVolume 4m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "postgresql-token-rmkq9" Warning FailedMount 2m kubelet, cab23-r720-11 MountVolume.WaitForAttach failed for volume "pvc-c66d1b6f-e64a-11e8-bb43-080027f45d2a" : fail to check rbd image status with: (exit status 22), rbd output: (2018-11-12 16:07:01.400015 7fcc31018100 -1 did not load config file, using default settings. server name not found: ceph-mon.ceph.svc.cluster.local (Name or service not known) unable to parse addrs in 'ceph-mon.ceph.svc.cluster.local:6789' rbd: couldn't connect to the cluster! ) Warning FailedMount 19s (x2 over 2m) kubelet, cab23-r720-11 Unable to mount volumes for pod "postgresql-0_ucp(a46bc160-e651-11e8-bb43-080027f45d2a)": timeout expired waiting for volumes to attach or mount for pod "ucp"/"postgresql-0". list of unmounted volumes=[postgresql-data]. list of unattached volumes=[postgresql-data postgresql-bin postgresql-token-rmkq9] Best Regards! Qiaolin Tu NSB MN 5G ECE HZ CN2 SG04 Mobile: +86 138 057 59684 E-Mail: qiaolin.tu@nokia-sbell.com<mailto:qiaolin.tu@nokia-sbell.com> From: Matthew H <matthew.heler@hotmail.com<mailto:matthew.heler@hotmail.com>> Sent: Friday, November 09, 2018 10:43 PM To: Tu, Qiaolin (NSB - CN/Hangzhou) <qiaolin.tu@nokia-sbell.com<mailto:qiaolin.tu@nokia-sbell.com>>; airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Cc: Li, Maxwell (NSB - CN/Hangzhou) <maxwell.li@nokia-sbell.com<mailto:maxwell.li@nokia-sbell.com>> Subject: Re: [Airship-discuss] Airship installation Questions Thanks,
From what I can see you need additional overrides set to run Ceph on a single node. The overrides you need are here [1].
Let me know if this helps get you in the right direction. [1] https://github.com/openstack/airship-in-a-bottle/blob/master/deployment_file... ________________________________ From: Tu, Qiaolin (NSB - CN/Hangzhou) <qiaolin.tu@nokia-sbell.com<mailto:qiaolin.tu@nokia-sbell.com>> Sent: Friday, November 9, 2018 4:51 AM To: Matthew H; airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Cc: Li, Maxwell (NSB - CN/Hangzhou) Subject: RE: [Airship-discuss] Airship installation Questions Hi, I deployed only 1 master node(1 genesis node + 1 master node), attachment is my yaml files. Thanks very much! root@cab23-r720-11:~# kubectl exec -it ceph-mon-h2gsm -n ceph -- ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 8.00000 root default -2 8.00000 host cab23-r720-11 0 hdd 1.00000 osd.0 up 1.00000 1.00000 1 hdd 1.00000 osd.1 up 1.00000 1.00000 2 hdd 1.00000 osd.2 up 1.00000 1.00000 3 hdd 1.00000 osd.3 up 1.00000 1.00000 4 hdd 1.00000 osd.4 up 1.00000 1.00000 5 hdd 1.00000 osd.5 up 1.00000 1.00000 6 hdd 1.00000 osd.6 up 1.00000 1.00000 7 hdd 1.00000 osd.7 up 1.00000 1.00000 root@cab23-r720-11:~# kubectl exec -it ceph-mon-h2gsm -n ceph -- ceph osd dump epoch 231 fsid 7b7576f4-3358-4668-9112-100440079807 created 2018-11-07 09:08:39.208517 modified 2018-11-09 09:40:10.639284 flags sortbitwise,recovery_deletes,purged_snapdirs crush_version 21 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client jewel min_compat_client hammer require_osd_release luminous pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 40 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rbd pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 last_change 223 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application cephfs pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 223 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application cephfs pool 4 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 72 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 83 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 6 'default.rgw.data.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 93 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 7 'default.rgw.gc' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 104 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 8 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 114 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 9 'default.rgw.intent-log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 124 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 10 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 135 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 11 'default.rgw.usage' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 146 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 12 'default.rgw.users.keys' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 156 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 13 'default.rgw.users.email' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 167 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 14 'default.rgw.users.swift' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 177 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 15 'default.rgw.users.uid' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 188 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 16 'default.rgw.buckets.extra' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 199 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 17 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 211 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw pool 18 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 221 flags hashpspool,nodelete,nopgchange,nosizechange stripe_width 0 application rgw max_osd 8 osd.0 up in weight 1 up_from 229 up_thru 229 down_at 228 last_clean_interval [8,228) 10.23.23.11:6800/15964 10.23.23.11:6814/2015964 10.23.23.11:6822/2015964 10.23.23.11:6823/2015964 exists,up fea47975-0810-47c9-ad43-e76ce81764a1 osd.1 up in weight 1 up_from 229 up_thru 229 down_at 228 last_clean_interval [9,228) 10.23.23.11:6808/16162 10.23.23.11:6807/2016162 10.23.23.11:6819/2016162 10.23.23.11:6801/2016162 exists,up cec98e14-83d5-4785-b8a7-a6f201170ac4 osd.2 up in weight 1 up_from 229 up_thru 229 down_at 228 last_clean_interval [9,228) 10.23.23.11:6804/16160 10.23.23.11:6806/2016160 10.23.23.11:6811/2016160 10.23.23.11:6834/2016160 exists,up 97315996-1cb9-4942-9786-8edc5a3862e3 osd.3 up in weight 1 up_from 229 up_thru 229 down_at 228 last_clean_interval [10,228) 10.23.23.11:6812/16588 10.23.23.11:6815/2016588 10.23.23.11:6805/2016588 10.23.23.11:6817/2016588 exists,up 49082e4c-7827-4c4c-85c9-16ea134289b4 osd.4 up in weight 1 up_from 229 up_thru 229 down_at 228 last_clean_interval [13,228) 10.23.23.11:6816/17053 10.23.23.11:6803/2017053 10.23.23.11:6813/2017053 10.23.23.11:6821/2017053 exists,up 8f9a5a7d-c97d-40c6-912e-33b6ab68d9e7 osd.5 up in weight 1 up_from 229 up_thru 229 down_at 228 last_clean_interval [16,228) 10.23.23.11:6820/17600 10.23.23.11:6810/2017600 10.23.23.11:6809/2017600 10.23.23.11:6818/2017600 exists,up b4602bfb-075f-4303-9f76-946576c4ef43 osd.6 up in weight 1 up_from 16 up_thru 212 down_at 0 last_clean_interval [0,0) 10.23.23.11:6824/17601 10.23.23.11:6825/17601 10.23.23.11:6826/17601 10.23.23.11:6827/17601 exists,up 2a853bad-7d97-43de-85f3-96e0f9e16c0d osd.7 up in weight 1 up_from 20 up_thru 212 down_at 0 last_clean_interval [0,0) 10.23.23.11:6828/18682 10.23.23.11:6829/18682 10.23.23.11:6830/18682 10.23.23.11:6831/18682 exists,up dfee9a9c-7587-421b-a0dc-eda2314174d9 root@cab23-r720-11:~# kubectl exec -it ceph-mon-h2gsm -n ceph -- ceph health detail HEALTH_WARN Reduced data availability: 338 pgs inactive; Degraded data redundancy: 338 pgs undersized PG_AVAILABILITY Reduced data availability: 338 pgs inactive pg 1.47 is stuck inactive for 174532.928425, current state undersized+peered, last acting [0] pg 1.48 is stuck inactive for 174532.928425, current state undersized+peered, last acting [5] pg 1.49 is stuck inactive for 174532.928425, current state undersized+peered, last acting [3] pg 1.4a is stuck inactive for 174532.928425, current state undersized+peered, last acting [4] pg 1.4b is stuck inactive for 174532.928425, current state undersized+peered, last acting [5] pg 1.4c is stuck inactive for 174532.928425, current state undersized+peered, last acting [7] pg 1.4d is stuck inactive for 174532.928425, current state undersized+peered, last acting [7] pg 1.4e is stuck inactive for 174532.928425, current state undersized+peered, last acting [4] pg 1.4f is stuck inactive for 174532.928425, current state undersized+peered, last acting [6] pg 1.50 is stuck inactive for 174532.928425, current state undersized+peered, last acting [0] pg 1.51 is stuck inactive for 174532.928425, current state undersized+peered, last acting [2] pg 1.52 is stuck inactive for 174532.928425, current state undersized+peered, last acting [5] pg 1.53 is stuck inactive for 174532.928425, current state undersized+peered, last acting [6] pg 1.54 is stuck inactive for 174532.928425, current state undersized+peered, last acting [4] pg 1.55 is stuck inactive for 174532.928425, current state undersized+peered, last acting [3] pg 1.56 is stuck inactive for 174532.928425, current state undersized+peered, last acting [2] pg 1.57 is stuck inactive for 174532.928425, current state undersized+peered, last acting [5] pg 1.58 is stuck inactive for 174532.928425, current state undersized+peered, last acting [3] pg 1.59 is stuck inactive for 174532.928425, current state undersized+peered, last acting [7] pg 1.5a is stuck inactive for 174532.928425, current state undersized+peered, last acting [5] pg 1.5b is stuck inactive for 174532.928425, current state undersized+peered, last acting [3] pg 1.5c is stuck inactive for 174532.928425, current state undersized+peered, last acting [7] pg 1.5d is stuck inactive for 174532.928425, current state undersized+peered, last acting [3] pg 1.5e is stuck inactive for 174532.928425, current state undersized+peered, last acting [0] pg 1.5f is stuck inactive for 174532.928425, current state undersized+peered, last acting [6] pg 18.40 is stuck inactive for 174337.349457, current state undersized+peered, last acting [1] pg 18.41 is stuck inactive for 174337.349457, current state undersized+peered, last acting [6] pg 18.42 is stuck inactive for 174337.349457, current state undersized+peered, last acting [5] pg 18.43 is stuck inactive for 174337.349457, current state undersized+peered, last acting [6] pg 18.44 is stuck inactive for 174337.349457, current state undersized+peered, last acting [7] pg 18.45 is stuck inactive for 174337.349457, current state undersized+peered, last acting [7] pg 18.46 is stuck inactive for 174337.349457, current state undersized+peered, last acting [4] pg 18.47 is stuck inactive for 174337.349457, current state undersized+peered, last acting [1] pg 18.48 is stuck inactive for 174337.349457, current state undersized+peered, last acting [1] pg 18.49 is stuck inactive for 174337.349457, current state undersized+peered, last acting [7] pg 18.4a is stuck inactive for 174337.349457, current state undersized+peered, last acting [0] pg 18.4b is stuck inactive for 174337.349457, current state undersized+peered, last acting [7] pg 18.4c is stuck inactive for 174337.349457, current state undersized+peered, last acting [6] pg 18.4d is stuck inactive for 174337.349457, current state undersized+peered, last acting [4] pg 18.4e is stuck inactive for 174337.349457, current state undersized+peered, last acting [0] pg 18.4f is stuck inactive for 174337.349457, current state undersized+peered, last acting [6] pg 18.54 is stuck inactive for 174337.349457, current state undersized+peered, last acting [5] pg 18.55 is stuck inactive for 174337.349457, current state undersized+peered, last acting [6] pg 18.58 is stuck inactive for 174337.349457, current state undersized+peered, last acting [6] pg 18.59 is stuck inactive for 174337.349457, current state undersized+peered, last acting [1] pg 18.5a is stuck inactive for 174337.349457, current state undersized+peered, last acting [5] pg 18.5b is stuck inactive for 174337.349457, current state undersized+peered, last acting [6] pg 18.5c is stuck inactive for 174337.349457, current state undersized+peered, last acting [6] pg 18.5d is stuck inactive for 174337.349457, current state undersized+peered, last acting [5] pg 18.5e is stuck inactive for 174337.349457, current state undersized+peered, last acting [4] pg 18.5f is stuck inactive for 174337.349457, current state undersized+peered, last acting [7] PG_DEGRADED Degraded data redundancy: 338 pgs undersized pg 1.47 is stuck undersized for 175.198010, current state undersized+peered, last acting [0] pg 1.48 is stuck undersized for 175.208624, current state undersized+peered, last acting [5] pg 1.49 is stuck undersized for 175.220652, current state undersized+peered, last acting [3] pg 1.4a is stuck undersized for 175.187294, current state undersized+peered, last acting [4] pg 1.4b is stuck undersized for 175.208051, current state undersized+peered, last acting [5] pg 1.4c is stuck undersized for 174531.317358, current state undersized+peered, last acting [7] pg 1.4d is stuck undersized for 174531.318742, current state undersized+peered, last acting [7] pg 1.4e is stuck undersized for 175.202431, current state undersized+peered, last acting [4] pg 1.4f is stuck undersized for 174531.331123, current state undersized+peered, last acting [6] pg 1.50 is stuck undersized for 175.207213, current state undersized+peered, last acting [0] pg 1.51 is stuck undersized for 175.215944, current state undersized+peered, last acting [2] pg 1.52 is stuck undersized for 175.209013, current state undersized+peered, last acting [5] pg 1.53 is stuck undersized for 174531.331587, current state undersized+peered, last acting [6] pg 1.54 is stuck undersized for 175.202884, current state undersized+peered, last acting [4] pg 1.55 is stuck undersized for 175.222033, current state undersized+peered, last acting [3] pg 1.56 is stuck undersized for 175.215670, current state undersized+peered, last acting [2] pg 1.57 is stuck undersized for 175.196356, current state undersized+peered, last acting [5] pg 1.58 is stuck undersized for 175.218886, current state undersized+peered, last acting [3] pg 1.59 is stuck undersized for 174531.316843, current state undersized+peered, last acting [7] pg 1.5a is stuck undersized for 175.209613, current state undersized+peered, last acting [5] pg 1.5b is stuck undersized for 175.219138, current state undersized+peered, last acting [3] pg 1.5c is stuck undersized for 174531.319395, current state undersized+peered, last acting [7] pg 1.5d is stuck undersized for 175.219426, current state undersized+peered, last acting [3] pg 1.5e is stuck undersized for 175.219873, current state undersized+peered, last acting [0] pg 1.5f is stuck undersized for 174531.331739, current state undersized+peered, last acting [6] pg 18.40 is stuck undersized for 175.211281, current state undersized+peered, last acting [1] pg 18.41 is stuck undersized for 174335.530906, current state undersized+peered, last acting [6] pg 18.42 is stuck undersized for 175.189316, current state undersized+peered, last acting [5] pg 18.43 is stuck undersized for 174335.529402, current state undersized+peered, last acting [6] pg 18.44 is stuck undersized for 174335.520749, current state undersized+peered, last acting [7] pg 18.45 is stuck undersized for 174335.520646, current state undersized+peered, last acting [7] pg 18.46 is stuck undersized for 175.204679, current state undersized+peered, last acting [4] pg 18.47 is stuck undersized for 175.211629, current state undersized+peered, last acting [1] pg 18.48 is stuck undersized for 175.213907, current state undersized+peered, last acting [1] pg 18.49 is stuck undersized for 174335.521334, current state undersized+peered, last acting [7] pg 18.4a is stuck undersized for 175.218699, current state undersized+peered, last acting [0] pg 18.4b is stuck undersized for 174335.527174, current state undersized+peered, last acting [7] pg 18.4c is stuck undersized for 174335.528996, current state undersized+peered, last acting [6] pg 18.4d is stuck undersized for 175.204937, current state undersized+peered, last acting [4] pg 18.4e is stuck undersized for 175.219027, current state undersized+peered, last acting [0] pg 18.4f is stuck undersized for 174335.531066, current state undersized+peered, last acting [6] pg 18.54 is stuck undersized for 175.189185, current state undersized+peered, last acting [5] pg 18.55 is stuck undersized for 174335.531222, current state undersized+peered, last acting [6] pg 18.58 is stuck undersized for 174335.530357, current state undersized+peered, last acting [6] pg 18.59 is stuck undersized for 175.204978, current state undersized+peered, last acting [1] pg 18.5a is stuck undersized for 175.192362, current state undersized+peered, last acting [5] pg 18.5b is stuck undersized for 174335.531432, current state undersized+peered, last acting [6] pg 18.5c is stuck undersized for 174335.530149, current state undersized+peered, last acting [6] pg 18.5d is stuck undersized for 175.191412, current state undersized+peered, last acting [5] pg 18.5e is stuck undersized for 175.205141, current state undersized+peered, last acting [4] pg 18.5f is stuck undersized for 174335.527472, current state undersized+peered, last acting [7] root@cab23-r720-11:~# kubectl exec -it ceph-mon-h2gsm -n ceph -- ceph -s cluster: id: 7b7576f4-3358-4668-9112-100440079807 health: HEALTH_WARN Reduced data availability: 338 pgs inactive Degraded data redundancy: 338 pgs undersized services: mon: 1 daemons, quorum cab23-r720-11 mgr: cab23-r720-11(active) mds: cephfs-1/1/1 up {0=mds-ceph-mds-6bfb74d9c7-gqgtl=up:creating}, 1 up:standby osd: 8 osds: 8 up, 8 in data: pools: 18 pools, 338 pgs objects: 0 objects, 0 bytes usage: 3229 MB used, 8184 GB / 8187 GB avail pgs: 100.000% pgs not active 338 undersized+peered Best Regards! Qiaolin Tu NSB MN 5G ECE HZ CN2 SG04 Mobile: +86 138 057 59684 E-Mail: qiaolin.tu@nokia-sbell.com<mailto:qiaolin.tu@nokia-sbell.com> From: Matthew H <matthew.heler@hotmail.com<mailto:matthew.heler@hotmail.com>> Sent: Friday, November 09, 2018 6:18 AM To: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Cc: Tu, Qiaolin (NSB - CN/Hangzhou) <qiaolin.tu@nokia-sbell.com<mailto:qiaolin.tu@nokia-sbell.com>> Subject: Re: [Airship-discuss] Airship installation Questions Greetings, Could you run the following commands from a MON pod: ceph osd tree ceph osd dump Also how many nodes did you deploy on? one or one or more nodes? Thanks,