[Airship-discuss] Airship installation Questions

Tu, Qiaolin (NSB - CN/Hangzhou) qiaolin.tu at nokia-sbell.com
Thu Nov 8 08:04:47 UTC 2018


Hi,

root at cab23-r720-11:~# kubectl exec -it -n ceph ceph-mon-h2gsm -- ceph -s
  cluster:
    id:     7b7576f4-3358-4668-9112-100440079807
    health: HEALTH_WARN
            Reduced data availability: 338 pgs inactive
            Degraded data redundancy: 338 pgs undersized

  services:
    mon: 1 daemons, quorum cab23-r720-11
    mgr: cab23-r720-11(active)
    mds: cephfs-1/1/1 up  {0=mds-ceph-mds-6bfb74d9c7-gqgtl=up:creating}, 1 up:standby
    osd: 8 osds: 8 up, 8 in
  data:
    pools:   18 pools, 338 pgs
    objects: 0 objects, 0 bytes
    usage:   2973 MB used, 8185 GB / 8187 GB avail
    pgs:     100.000% pgs not active
             338 undersized+peered

Which I marked with red color mean ceph state have something wrong? And this lead to Readiness probe failed: Get http://10.97.38.77:8088/<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.97.38.77-3A8088_&d=DwQGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=D4ltt0SIk5Pwy68QAIKWn9EVFOIaNEjb0kjv1bpo724&s=IA_GBVFM-BWoCOlZ7Pt-MR2fraxaMFqve_EXGWlkvo4&e=>: dial tcp 10.97.38.77:8088: getsockopt: connection refused ?


Best Regards!
Qiaolin Tu
NSB MN 5G ECE HZ CN2 SG04
Mobile:  +86 138 057 59684
E-Mail:   qiaolin.tu at nokia-sbell.com<mailto:qiaolin.tu at nokia-sbell.com>


From: SKELS, KASPARS <ks3019 at att.com>
Sent: Thursday, November 08, 2018 3:49 PM
To: Tu, Qiaolin (NSB - CN/Hangzhou) <qiaolin.tu at nokia-sbell.com>; Li, Maxwell (NSB - CN/Hangzhou) <maxwell.li at nokia-sbell.com>
Cc: GORSHUNOV, ROMAN <roman.gorshunov at att.com>; MEADOWS, ALAN <am240k at att.com>; MCEUEN, MATT <MM9745 at att.com>; PACHECO, RODOLFO J <rp2723 at att.com>; 'airship-discuss at lists.airshipit.org' <airship-discuss at lists.airshipit.org>; REDDY, CHINASUBBA <cr3938 at att.com>; BIRLEY, PETE <pb269f at att.com>
Subject: RE: Airship installation Questions

Hey!

Have a look if ceph is healthy in general with following command (cluster status)
kubectl exec –it –n ceph ceph-mon-h2gsm -- ceph -s

I still see that you have 2 instances of ceph-mgr from your terminal outputs
ceph-mgr-6dc44fc75b-jtv7w                                   0/1       Pending     0          22h
ceph-mgr-6dc44fc75b-jvstm                                   1/1       Running     0          22h

There are 2 ceph-client charts, 1 for initial single node/genesis, and ceph-client-update for final state (when additional nodes join). Feel free to explore
https://github.com/openstack/airship-treasuremap/tree/master/global/software/charts/ucp/ceph

I think simplest way for you to get ceph running (if you are using single control plane) would be to just stay with initial ceph as it will also keep crush rules and other settings as needed (final state requires 3 domains/hosts)
I would say change https://github.com/openstack/airship-treasuremap/blob/master/global/software/manifests/full-site.yaml#L22
From  `ucp-ceph-update` to just `ucp-ceph` instead; this will keep using initial single node/genesis ceph and keep it healthy.

For RGW – is similar – you will need to set it to 1 replica (right now there is no site override so you would need to craft one or set on global level)
https://github.com/openstack/airship-treasuremap/blob/master/global/software/charts/ucp/ceph/ceph-rgw.yaml#L133

Kindly, Kaspars


From: Tu, Qiaolin (NSB - CN/Hangzhou) [mailto:qiaolin.tu at nokia-sbell.com]
Sent: Thursday, November 8, 2018 1:25 AM
To: SKELS, KASPARS <ks3019 at att.com<mailto:ks3019 at att.com>>; Li, Maxwell (NSB - CN/Hangzhou) <maxwell.li at nokia-sbell.com<mailto:maxwell.li at nokia-sbell.com>>
Cc: GORSHUNOV, ROMAN <roman.gorshunov at att.com<mailto:roman.gorshunov at att.com>>; MEADOWS, ALAN <am240k at att.com<mailto:am240k at att.com>>; MCEUEN, MATT <MM9745 at att.com<mailto:MM9745 at att.com>>; PACHECO, RODOLFO J <rp2723 at att.com<mailto:rp2723 at att.com>>; 'airship-discuss at lists.airshipit.org' <airship-discuss at lists.airshipit.org<mailto:airship-discuss at lists.airshipit.org>>
Subject: RE: Airship installation Questions

Hi,
Thanks very much for your reply.
I adjust the replica count on site level (set 1, instead of 2) and ceph-rgw pod still have getsockopt: connection refused error. Below is the log detail:

root at cab23-r720-11:~# kubectl describe pod ceph-rgw-5b6645c456-rsv4k -n ceph
Name:           ceph-rgw-5b6645c456-rsv4k
Namespace:      ceph
Node:           cab23-r720-11/10.23.22.11
Start Time:     Wed, 07 Nov 2018 23:07:06 +0000
Labels:         application=ceph
                component=rgw
                pod-template-hash=1622017012
                release_group=airship-ucp-ceph-rgw
Annotations:    configmap-bin-hash=aee34fa624622fb03c48b85c014e805a1012f74f772bc0f3a1e91597839ce49e
                configmap-etc-client-hash=38eb0b2eb23d31bcd3c4238c2bf9f6b8acd335659d94b3fcfff6c333ed1339b2
Status:         Running
IP:             10.97.38.77
Controlled By:  ReplicaSet/ceph-rgw-5b6645c456

Events:
  Type     Reason     Age                  From                    Message
  ----     ------     ----                 ----                    -------
  Warning  BackOff    5m (x737 over 7h)    kubelet, cab23-r720-11  Back-off restarting failed container
  Warning  Unhealthy  39s (x1154 over 7h)  kubelet, cab23-r720-11  Readiness probe failed: Get http://10.97.38.77:8088/<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.97.38.77-3A8088_&d=DwQGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=D4ltt0SIk5Pwy68QAIKWn9EVFOIaNEjb0kjv1bpo724&s=IA_GBVFM-BWoCOlZ7Pt-MR2fraxaMFqve_EXGWlkvo4&e=>: dial tcp 10.97.38.77:8088: getsockopt: connection refused


root at cab23-r720-11:~# kubectl logs -f ceph-rgw-5b6645c456-rsv4k -n ceph
+ export LC_ALL=C
+ LC_ALL=C
+ : 0
++ uname -n
+ : ceph-rgw-5b6645c456-rsv4k
+ : ''
+ : ''
+ : 0
+ : 9000
+ : 0.0.0.0
+ : /etc/ceph/ceph.client.admin.keyring
+ : /var/lib/ceph/radosgw/ceph-rgw-5b6645c456-rsv4k/keyring
+ : /var/lib/ceph/bootstrap-rgw/ceph.keyring
+ [[ ! -e /etc/ceph/ceph.conf ]]
+ '[' 0 -eq 1 ']'
+ '[' '!' -e /var/lib/ceph/radosgw/ceph-rgw-5b6645c456-rsv4k/keyring ']'
+ RGW_FRONTENDS='civetweb port=8088'
+ '[' 0 -eq 1 ']'
+ /usr/bin/radosgw --cluster ceph --setuser ceph --setgroup ceph -d -n client.rgw.ceph-rgw-5b6645c456-rsv4k -k /var/lib/ceph/radosgw/ceph-rgw-5b6645c456-rsv4k/keyring --rgw-socket-path= --rgw-zonegroup= --rgw-zone= '--rgw-frontends=civetweb port=8088'
2018-11-08 06:53:53.769321 7f3c7826de80  0 deferred set uid:gid to 64045:64045 (ceph:ceph)
2018-11-08 06:53:53.769371 7f3c7826de80  0 ceph version 12.2.3 (2dab17a455c09584f2a85e6b10888337d1ec8949) luminous (stable), process (unknown), pid 8



root at cab23-r720-11:~# kubectl get pods -n ceph
NAME                                                        READY     STATUS      RESTARTS   AGE
airship-ucp-ceph-provisioners-ceph-ns-key-generator-8vm8b   0/1       Completed   0          20h
ceph-bootstrap-mp5tn                                        0/1       Completed   0          22h
ceph-cephfs-client-key-generator-4sqrd                      0/1       Completed   0          20h
ceph-cephfs-provisioner-676684f6bd-48fq8                    1/1       Running     0          20h
ceph-cephfs-provisioner-676684f6bd-mqwnj                    1/1       Running     0          20h
ceph-mds-6bfb74d9c7-gqgtl                                   1/1       Running     0          22h
ceph-mds-6bfb74d9c7-sk4pw                                   1/1       Running     0          22h
ceph-mds-keyring-generator-m2rjj                            0/1       Completed   0          22h
ceph-mgr-6dc44fc75b-jtv7w                                   0/1       Pending     0          22h
ceph-mgr-6dc44fc75b-jvstm                                   1/1       Running     0          22h
ceph-mgr-keyring-generator-m56p5                            0/1       Completed   0          22h
ceph-mon-check-6db6b569b6-flg76                             1/1       Running     0          22h
ceph-mon-h2gsm                                              1/1       Running     0          21h
ceph-mon-keyring-generator-5phjm                            0/1       Completed   0          22h
ceph-osd-default-64779b8c-c74ms                             1/1       Running     0          22h
ceph-osd-default-6ea9de2c-hn4tm                             1/1       Running     0          22h
ceph-osd-default-70a54190-t54l7                             1/1       Running     0          22h
ceph-osd-default-7544b6da-mxfj6                             1/1       Running     1          22h
ceph-osd-default-7cfc44c1-vbqs4                             1/1       Running     0          22h
ceph-osd-default-83945928-66m9s                             1/1       Running     0          22h
ceph-osd-default-be8e8cc4-2vnxb                             1/1       Running     0          22h
ceph-osd-default-f9249fa9-8gw25                             1/1       Running     0          22h
ceph-osd-keyring-generator-fnrgk                            0/1       Completed   0          22h
ceph-rbd-pool-4dzjz                                         0/1       Completed   0          22h
ceph-rbd-provisioner-84bc5c88c7-smfcn                       1/1       Running     0          20h
ceph-rbd-provisioner-84bc5c88c7-vtc5g                       1/1       Running     0          20h
ceph-rgw-5b6645c456-rsv4k                                   0/1       Running     74         8h
ceph-rgw-5b6645c456-sgzpt                                   0/1       Running     74         8h
ceph-rgw-storage-init-pqf6q                                 0/1       Completed   0          8h
ceph-storage-keys-generator-9fx4k                           0/1       Completed   0          22h
ingress-65dc849968-9zlqn                                    1/1       Running     0          22h
ingress-65dc849968-cc8xp                                    1/1       Running     0          22h
ingress-error-pages-796b76c856-dnpcp                        1/1       Running     0          22h
ingress-error-pages-796b76c856-tt9x4                        1/1       Running     0          22h

Best Regards!
Qiaolin Tu
NSB MN 5G ECE HZ CN2 SG04
Mobile:  +86 138 057 59684
E-Mail:   qiaolin.tu at nokia-sbell.com<mailto:qiaolin.tu at nokia-sbell.com>


From: SKELS, KASPARS <ks3019 at att.com<mailto:ks3019 at att.com>>
Sent: Wednesday, November 07, 2018 4:28 AM
To: Tu, Qiaolin (NSB - CN/Hangzhou) <qiaolin.tu at nokia-sbell.com<mailto:qiaolin.tu at nokia-sbell.com>>; Li, Maxwell (NSB - CN/Hangzhou) <maxwell.li at nokia-sbell.com<mailto:maxwell.li at nokia-sbell.com>>
Cc: GORSHUNOV, ROMAN <roman.gorshunov at att.com<mailto:roman.gorshunov at att.com>>; MEADOWS, ALAN <am240k at att.com<mailto:am240k at att.com>>; MCEUEN, MATT <MM9745 at att.com<mailto:MM9745 at att.com>>; PACHECO, RODOLFO J <rp2723 at att.com<mailto:rp2723 at att.com>>; 'airship-discuss at lists.airshipit.org' <airship-discuss at lists.airshipit.org<mailto:airship-discuss at lists.airshipit.org>>
Subject: RE: Airship installation Questions

Hi Qiaolin/Maxwell,

it seems like you may be using 1 control node for your deployment.
The ceph-mgr chart is set to have 2 replicas and is trying to launch them (here you get a port clash).

Generally speaking, the manifests (by default) are targeted for 3 control node (HA) configuration, including ceph to reflect production-like HA deployment
https://github.com/openstack/airship-treasuremap/blob/master/global/software/charts/ucp/ceph/ceph-client-update.yaml#L145<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Dtreasuremap_blob_master_global_software_charts_ucp_ceph_ceph-2Dclient-2Dupdate.yaml-23L145&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=D4ltt0SIk5Pwy68QAIKWn9EVFOIaNEjb0kjv1bpo724&s=ryDNfEpcHGDMGGbSjUmlkjoTcINDh6JT426VPeJDY78&e=>

That said, you may adjust the replica count on site level (set 1, instead of 2) as well, as well as OSD count here by doing overrides (matching how many OSDs you have)
https://github.com/openstack/airship-treasuremap/blob/master/site/airship-seaworthy/software/charts/ucp/ceph/ceph-client-update.yaml<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Dtreasuremap_blob_master_site_airship-2Dseaworthy_software_charts_ucp_ceph_ceph-2Dclient-2Dupdate.yaml&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=D4ltt0SIk5Pwy68QAIKWn9EVFOIaNEjb0kjv1bpo724&s=1_cNRZV5b8U3J20kYkk-BWxvlkUVVQdYQIdjjjVvMW4&e=>

And also, adjustments in other charts may be required to set this up for a single control plane node, here is more info on reference site
https://airship-treasuremap.readthedocs.io/en/latest/seaworthy.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dtreasuremap.readthedocs.io_en_latest_seaworthy.html&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=D4ltt0SIk5Pwy68QAIKWn9EVFOIaNEjb0kjv1bpo724&s=ezzAGQJQOJd6evJtTSDs8QVivNbRIw5b2DtLrHtd0k8&e=>

Kindly, Kaspars



From: Kaspars Skels [mailto:kaspars.skels at gmail.com]
Sent: Tuesday, November 6, 2018 2:10 PM
To: qiaolin.tu at nokia-sbell.com<mailto:qiaolin.tu at nokia-sbell.com>
Cc: MEADOWS, ALAN <am240k at att.com<mailto:am240k at att.com>>; MONTEIRO, FELIPE C <fm577c at att.com<mailto:fm577c at att.com>>; BARTRA, RICK <rb560u at att.com<mailto:rb560u at att.com>>; REDDY, CHINASUBBA <cr3938 at att.com<mailto:cr3938 at att.com>>; GORSHUNOV, ROMAN <roman.gorshunov at att.com<mailto:roman.gorshunov at att.com>>; KHUNTIA, SOUMITRA <sk698p at att.com<mailto:sk698p at att.com>>; KABANOV, DMITRII <dk370c at att.com<mailto:dk370c at att.com>>; Mark Burnett <mark.m.burnett at gmail.com<mailto:mark.m.burnett at gmail.com>>; VOLKOV, ANDREY <av903u at att.com<mailto:av903u at att.com>>; avolkov at mirantis.com<mailto:avolkov at mirantis.com>; Chris Wedgwood <cw at f00f.org<mailto:cw at f00f.org>>; PALLAV GUPTA <pallavgupta84 at gmail.com<mailto:pallavgupta84 at gmail.com>>; GUPTA, SANGEET <sg774j at att.com<mailto:sg774j at att.com>>; zuul at review01.openstack.org<mailto:zuul at review01.openstack.org>; HUSSEY, SCOTT T <sh8121 at att.com<mailto:sh8121 at att.com>>; PACHECO, RODOLFO J <rp2723 at att.com<mailto:rp2723 at att.com>>; maxwell.li at nokia-sbell.com<mailto:maxwell.li at nokia-sbell.com>; SKELS, KASPARS <ks3019 at att.com<mailto:ks3019 at att.com>>
Subject: Re: Airship installation Questions



On Tue, Nov 6, 2018 at 3:42 AM Tu, Qiaolin (NSB - CN/Hangzhou) <qiaolin.tu at nokia-sbell.com<mailto:qiaolin.tu at nokia-sbell.com>> wrote:
Hi,
I also found that pod(ceph-mgr-6dc44fc75b-25n24) is always on the pending state. It report error as below:   Warning  FailedScheduling  1m (x125 over 36m)  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.


root at cab23-r720-11:~# kubectl get pods --all-namespaces
NAMESPACE     NAME                                                        READY     STATUS      RESTARTS   AGE
ceph          airship-ucp-ceph-provisioners-ceph-ns-key-generator-n5vzj   0/1       Completed   0          20m
ceph          ceph-cephfs-client-key-generator-656rd                      0/1       Completed   0          20m
ceph          ceph-cephfs-provisioner-676684f6bd-trf5h                    1/1       Running     0          20m
ceph          ceph-cephfs-provisioner-676684f6bd-zfk2s                    1/1       Running     0          20m
ceph          ceph-mds-6bfb74d9c7-btbt5                                   1/1       Running     0          37m
ceph          ceph-mds-6bfb74d9c7-qw4xl                                   1/1       Running     0          37m
ceph          ceph-mds-keyring-generator-dlqjt                            0/1       Completed   0          53m
ceph          ceph-mgr-6dc44fc75b-25n24                                   0/1       Pending     0          37m
ceph          ceph-mgr-6dc44fc75b-42mqz                                   1/1       Running     0          37m
ceph          ceph-mgr-keyring-generator-7dftx                            0/1       Completed   0          53m
ceph          ceph-mon-cfrf6                                              1/1       Running     0          41m
ceph          ceph-mon-check-6db6b569b6-67vvm                             1/1       Running     0          53m
ceph          ceph-mon-keyring-generator-55ps8                            0/1       Completed   0          53m



root at cab23-r720-11:~# kubectl describe pod ceph-mgr-6dc44fc75b-25n24 -n ceph
Name:           ceph-mgr-6dc44fc75b-25n24
Namespace:      ceph
Node:           <none>
Labels:         application=ceph
                component=mgr
                pod-template-hash=2870097316
                release_group=airship-ucp-ceph-client
Annotations:    configmap-bin-hash=d23610f5cc67014b596eb45b9d47ddb97997d6ec5fd3289fb264429a260f39a9
                configmap-etc-client-hash=0174be85f29398aa5cea5fccb9537e0e0f25ea2d2f280182e9e2506a1be1b115
Status:         Pending
IP:
Controlled By:  ReplicaSet/ceph-mgr-6dc44fc75b
Init Containers:
  init:
    Image:      quay.io/stackanetes/kubernetes-entrypoint:v0.3.1<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_stackanetes_kubernetes-2Dentrypoint-3Av0.3.1&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=QVq3KXWVyPVji_O3s5wgjBk7wnbkw3swE9ddmYytPOw&e=>
    Port:       <none>
    Host Port:  <none>
    Command:
      kubernetes-entrypoint
    Environment:
      POD_NAME:              ceph-mgr-6dc44fc75b-25n24 (v1:metadata.name<https://urldefense.proofpoint.com/v2/url?u=http-3A__metadata.name&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=DIlvVxm8OFCSVZEOPMgsZQb4rkRx8zql64UE_2OnMjU&e=>)
      NAMESPACE:             ceph (v1:metadata.namespace)
      INTERFACE_NAME:        eth0
      PATH:                  /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/
      DEPENDENCY_SERVICE:    ceph:ceph-mon
      DEPENDENCY_JOBS:       ceph-storage-keys-generator,ceph-mgr-keyring-generator
      DEPENDENCY_DAEMONSET:
      DEPENDENCY_CONTAINER:
      DEPENDENCY_POD_JSON:
      COMMAND:               echo done
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount<https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=S6tMkfnBpnWWGsjH58zRIhEcnxQ0dJINNWuoTXXeNyI&e=> from ceph-mgr-token-pgbwp (ro)
  ceph-init-dirs:
    Image:      docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04<https://urldefense.proofpoint.com/v2/url?u=http-3A__docker.io_ceph_daemon-3Atag-2Dbuild-2Dmaster-2Dluminous-2Dubuntu-2D16.04&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=wD6kGoJq_VjM3q9MaytwbkA4gjV0HuHeKt7ZJ_419m4&e=>
    Port:       <none>
    Host Port:  <none>
    Command:
      /tmp/init-dirs.sh
    Environment:
      CLUSTER:  ceph
    Mounts:
      /etc/ceph from pod-etc-ceph (rw)
      /run from pod-run (rw)
      /tmp/init-dirs.sh from ceph-client-bin (ro)
      /var/lib/ceph from pod-var-lib-ceph (rw)
      /var/run/secrets/kubernetes.io/serviceaccount<https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=S6tMkfnBpnWWGsjH58zRIhEcnxQ0dJINNWuoTXXeNyI&e=> from ceph-mgr-token-pgbwp (ro)
Containers:
  ceph-mgr:
    Image:       docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04<https://urldefense.proofpoint.com/v2/url?u=http-3A__docker.io_ceph_daemon-3Atag-2Dbuild-2Dmaster-2Dluminous-2Dubuntu-2D16.04&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=wD6kGoJq_VjM3q9MaytwbkA4gjV0HuHeKt7ZJ_419m4&e=>
    Ports:       7000/TCP, 9283/TCP
    Host Ports:  7000/TCP, 9283/TCP
    Command:
      /mgr-start.sh
    Liveness:   exec [/tmp/mgr-check.sh liveness] delay=30s timeout=5s period=10s #success=1 #failure=3
    Readiness:  exec [/tmp/mgr-check.sh readiness] delay=30s timeout=5s period=10s #success=1 #failure=3
    Environment:
      CLUSTER:          ceph
      ENABLED_MODULES:  restful
status
prometheus
    Mounts:
      /etc/ceph from pod-etc-ceph (rw)
      /etc/ceph/ceph.client.admin.keyring from ceph-client-admin-keyring (ro)
      /etc/ceph/ceph.conf from ceph-client-etc (ro)
      /mgr-start.sh from ceph-client-bin (ro)
      /run from pod-run (rw)
      /tmp/mgr-check.sh from ceph-client-bin (ro)
      /var/lib/ceph from pod-var-lib-ceph (rw)
      /var/lib/ceph/bootstrap-mgr/ceph.keyring from ceph-bootstrap-mgr-keyring (rw)
      /var/run/secrets/kubernetes.io/serviceaccount<https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=S6tMkfnBpnWWGsjH58zRIhEcnxQ0dJINNWuoTXXeNyI&e=> from ceph-mgr-token-pgbwp (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  pod-etc-ceph:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  ceph-client-bin:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ceph-client-bin
    Optional:  false
  ceph-client-etc:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ceph-client-etc
    Optional:  false
  pod-var-lib-ceph:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  pod-run:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  Memory
  ceph-client-admin-keyring:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ceph-client-admin-keyring
    Optional:    false
  ceph-bootstrap-mgr-keyring:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ceph-bootstrap-mgr-keyring
    Optional:    false
  ceph-mgr-token-pgbwp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ceph-mgr-token-pgbwp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  ceph-mgr=enabled
Tolerations:     node.kubernetes.io/not-ready:NoExecute<https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_not-2Dready-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=7UGHV8v22mtkm09xG6MoMxOg7RO9aa7lQkwQd3yYYqw&e=> for 300s
                 node.kubernetes.io/unreachable:NoExecute<https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_unreachable-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=U8nhQAM3YHPjGEUgzGDzxxM8T5YYO0V9_PBd7yfaXpU&e=> for 300s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  1m (x125 over 36m)  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.

Best Regards!
Qiaolin Tu
NSB MN 5G ECE HZ CN2 SG04
Mobile:  +86 138 057 59684
E-Mail:   qiaolin.tu at nokia-sbell.com<mailto:qiaolin.tu at nokia-sbell.com>


From: Tu, Qiaolin (NSB - CN/Hangzhou)
Sent: Tuesday, November 06, 2018 1:46 PM
To: 'MEADOWS, ALAN' <am240k at att.com<mailto:am240k at att.com>>; Kaspars Skels <kaspars.skels at gmail.com<mailto:kaspars.skels at gmail.com>>
Cc: MONTEIRO, FELIPE C <fm577c at att.com<mailto:fm577c at att.com>>; BARTRA, RICK <rb560u at att.com<mailto:rb560u at att.com>>; REDDY, CHINASUBBA <cr3938 at att.com<mailto:cr3938 at att.com>>; GORSHUNOV, ROMAN <roman.gorshunov at att.com<mailto:roman.gorshunov at att.com>>; KHUNTIA, SOUMITRA <sk698p at att.com<mailto:sk698p at att.com>>; KABANOV, DMITRII <dk370c at att.com<mailto:dk370c at att.com>>; mark.m.burnett at gmail.com<mailto:mark.m.burnett at gmail.com>; VOLKOV, ANDREY <av903u at att.com<mailto:av903u at att.com>>; avolkov at mirantis.com<mailto:avolkov at mirantis.com>; cw at f00f.org<mailto:cw at f00f.org>; pallavgupta84 at gmail.com<mailto:pallavgupta84 at gmail.com>; GUPTA, SANGEET <sg774j at att.com<mailto:sg774j at att.com>>; pete at port.direct<mailto:pete at port.direct>; zuul at review01.openstack.org<mailto:zuul at review01.openstack.org>; HUSSEY, SCOTT T <sh8121 at att.com<mailto:sh8121 at att.com>>; PACHECO, RODOLFO J <rp2723 at att.com<mailto:rp2723 at att.com>>; Li, Maxwell (NSB - CN/Hangzhou) <maxwell.li at nokia-sbell.com<mailto:maxwell.li at nokia-sbell.com>>
Subject: Airship installation Questions

Hi,
Thanks very much for your kindly support.
I also encountered a problem when install multi-node airship. It blocked by 2 ceph-rgw pods. They crash off and run again and again. Pods error log as bellow:

root at cab23-r720-11:~# kubectl describe pod ceph-rgw-6ff5ff866d-4x66n -n ceph
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 1m default-scheduler Successfully assigned ceph-rgw-6ff5ff866d-4x66n to cab23-r720-11
Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "pod-etc-ceph"
Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "pod-var-lib-ceph"
Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "pod-run"
Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "ceph-rgw-etc"
Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "ceph-rgw-bin"
Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "ceph-bootstrap-rgw-keyring"
Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "ceph-rgw-token-sjg9x"
Normal Pulled 1m kubelet, cab23-r720-11 Container image "quay.io/stackanetes/kubernetes-entrypoint:v0.3.1<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_stackanetes_kubernetes-2Dentrypoint-3Av0.3.1&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=QVq3KXWVyPVji_O3s5wgjBk7wnbkw3swE9ddmYytPOw&e=>" already present on machine

Normal Started 1m kubelet, cab23-r720-11 Started container
Normal Pulled 1m kubelet, cab23-r720-11 Container image "docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04<https://urldefense.proofpoint.com/v2/url?u=http-3A__docker.io_ceph_daemon-3Atag-2Dbuild-2Dmaster-2Dluminous-2Dubuntu-2D16.04&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=wD6kGoJq_VjM3q9MaytwbkA4gjV0HuHeKt7ZJ_419m4&e=>" already present on machine
Normal Created 1m kubelet, cab23-r720-11 Created container
Normal Started 1m kubelet, cab23-r720-11 Started container

Warning Unhealthy 18s (x6 over 1m) kubelet, cab23-r720-11 Readiness probe failed: Get http://10.97.38.105:8088/<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.97.38.105-3A8088_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=N6CGj_cMP5_eYfe9DOVCi3m-z6jWPpRQNrRRkYg6kjA&e=>: dial tcp 10.97.38.105:8088<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.97.38.105-3A8088&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=_MBsJiSpie7BI64_sUyyqznHkaLMa1UEOyINiKSIfEE&e=>: getsockopt: connection refused


Best Regards!
Qiaolin Tu
NSB MN 5G ECE HZ CN2 SG04
Mobile:  +86 138 057 59684
E-Mail:   qiaolin.tu at nokia-sbell.com<mailto:qiaolin.tu at nokia-sbell.com>


From: MEADOWS, ALAN <am240k at att.com<mailto:am240k at att.com>>
Sent: Saturday, November 03, 2018 5:39 AM
To: Kaspars Skels <kaspars.skels at gmail.com<mailto:kaspars.skels at gmail.com>>; Li, Maxwell (NSB - CN/Hangzhou) <maxwell.li at nokia-sbell.com<mailto:maxwell.li at nokia-sbell.com>>
Cc: MONTEIRO, FELIPE C <fm577c at att.com<mailto:fm577c at att.com>>; BARTRA, RICK <rb560u at att.com<mailto:rb560u at att.com>>; REDDY, CHINASUBBA <cr3938 at att.com<mailto:cr3938 at att.com>>; GORSHUNOV, ROMAN <roman.gorshunov at att.com<mailto:roman.gorshunov at att.com>>; KHUNTIA, SOUMITRA <sk698p at att.com<mailto:sk698p at att.com>>; KABANOV, DMITRII <dk370c at att.com<mailto:dk370c at att.com>>; mark.m.burnett at gmail.com<mailto:mark.m.burnett at gmail.com>; VOLKOV, ANDREY <av903u at att.com<mailto:av903u at att.com>>; avolkov at mirantis.com<mailto:avolkov at mirantis.com>; cw at f00f.org<mailto:cw at f00f.org>; pallavgupta84 at gmail.com<mailto:pallavgupta84 at gmail.com>; GUPTA, SANGEET <sg774j at att.com<mailto:sg774j at att.com>>; pete at port.direct<mailto:pete at port.direct>; zuul at review01.openstack.org<mailto:zuul at review01.openstack.org>; HUSSEY, SCOTT T <sh8121 at att.com<mailto:sh8121 at att.com>>; Tu, Qiaolin (NSB - CN/Hangzhou) <qiaolin.tu at nokia-sbell.com<mailto:qiaolin.tu at nokia-sbell.com>>; PACHECO, RODOLFO J <rp2723 at att.com<mailto:rp2723 at att.com>>
Subject: RE: Airship Version Questions

Hi Maxwell!

Great to see additional people trying Airship out.

Treasuremap does change quite frequently.  In fact, this is from a lot of the great work Kaspars has been doing – building a CI/CD pipeline to not only keep treasuremap fresh and referencing the latest working versions of Airship components but also validating that the YAML documents are able to provision a multi-node baremetal physical environment by way of third-party gates.

This gives us a high degree of confidence that no matter what version of treasuremap you have decided to start working with to deploy your own environments it should work.

To be sure, we do understand when getting started it is nice to work with a stable target that isn’t constantly changing.    Kaspars has introduced tagging to provide a human readable reference (using dates).  These are not on any particular cadence, but usually follow more in-depth tested release beyond just automated CI/CD using the manifests:

https://github.com/openstack/airship-treasuremap/releases<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Dtreasuremap_releases&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=ZyIbYVxKvMan1A5Fw3GshH0lifNZHRUnU_40MqtumxA&e=>

For right now with regard to pegleg, the best way to leverage the version of pegleg that has been tested with the treasuremap manifests is to pull it the one used in the baremetal CI/CD gate test directly from the target tag Jenkinsfile.  For example, for v18.11.01:

https://github.com/openstack/airship-treasuremap/blob/v18.11.01/tools/gate/Jenkinsfile#L15<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Dtreasuremap_blob_v18.11.01_tools_gate_Jenkinsfile-23L15&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=S0oZ2OKnKI07o5yGWGvwz6_gUToysqjc9W_EKPR8eTY&e=>

Please feel free to get on the airship mailing list (airshipit.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__airshipit.org&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=Yfn1URqa4p-w7xoxiueJJP2SNYUZZCsOA07XxDCOxEM&s=9mXi3m93VupawhG1pIMsk99_Ng5NFUCbs-6ip-2NxTY&e=>) to connect with a wider audience or our IRC channel!

Alan Meadows

From: Kaspars Skels [mailto:kaspars.skels at gmail.com]
Sent: Friday, November 02, 2018 2:27 PM
To: maxwell.li at nokia-sbell.com<mailto:maxwell.li at nokia-sbell.com>
Cc: MONTEIRO, FELIPE C <fm577c at att.com<mailto:fm577c at att.com>>; BARTRA, RICK <rb560u at att.com<mailto:rb560u at att.com>>; REDDY, CHINASUBBA <cr3938 at att.com<mailto:cr3938 at att.com>>; GORSHUNOV, ROMAN <roman.gorshunov at att.com<mailto:roman.gorshunov at att.com>>; KHUNTIA, SOUMITRA <sk698p at att.com<mailto:sk698p at att.com>>; KABANOV, DMITRII <dk370c at att.com<mailto:dk370c at att.com>>; mark.m.burnett at gmail.com<mailto:mark.m.burnett at gmail.com>; VOLKOV, ANDREY <av903u at att.com<mailto:av903u at att.com>>; avolkov at mirantis.com<mailto:avolkov at mirantis.com>; cw at f00f.org<mailto:cw at f00f.org>; pallavgupta84 at gmail.com<mailto:pallavgupta84 at gmail.com>; GUPTA, SANGEET <sg774j at att.com<mailto:sg774j at att.com>>; pete at port.direct<mailto:pete at port.direct>; zuul at review01.openstack.org<mailto:zuul at review01.openstack.org>; HUSSEY, SCOTT T <sh8121 at att.com<mailto:sh8121 at att.com>>; qiaolin.tu at nokia-sbell.com<mailto:qiaolin.tu at nokia-sbell.com>; MEADOWS, ALAN <am240k at att.com<mailto:am240k at att.com>>; PACHECO, RODOLFO J <rp2723 at att.com<mailto:rp2723 at att.com>>
Subject: Re: Airship Version Questions

+ Alan/Rodolfo



On Thu, Nov 1, 2018 at 3:37 AM Li, Maxwell (NSB - CN/Hangzhou) <maxwell.li at nokia-sbell.com<mailto:maxwell.li at nokia-sbell.com>> wrote:
Hi Airship Team:

I have a problem when I deploy airship on my server. There are new commits in airship-pegleg and airship-treasuremap, almost erveryday. And there are no tags or branches in these code repositories. Could I get a commit ID about these repositories so that I could deploy airship.

Thanks a lot!

Best Regards!
Maxwell Li

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20181108/a99a3a28/attachment-0001.html>


More information about the Airship-discuss mailing list