Hi Qiaolin/Maxwell,

 

it seems like you may be using 1 control node for your deployment.

The ceph-mgr chart is set to have 2 replicas and is trying to launch them (here you get a port clash).

 

Generally speaking, the manifests (by default) are targeted for 3 control node (HA) configuration, including ceph to reflect production-like HA deployment

https://github.com/openstack/airship-treasuremap/blob/master/global/software/charts/ucp/ceph/ceph-client-update.yaml#L145

 

That said, you may adjust the replica count on site level (set 1, instead of 2) as well, as well as OSD count here by doing overrides (matching how many OSDs you have)

https://github.com/openstack/airship-treasuremap/blob/master/site/airship-seaworthy/software/charts/ucp/ceph/ceph-client-update.yaml

 

And also, adjustments in other charts may be required to set this up for a single control plane node, here is more info on reference site

https://airship-treasuremap.readthedocs.io/en/latest/seaworthy.html

 

Kindly, Kaspars

 

 

 

From: Kaspars Skels [mailto:kaspars.skels@gmail.com]
Sent: Tuesday, November 6, 2018 2:10 PM
To: qiaolin.tu@nokia-sbell.com
Cc: MEADOWS, ALAN <am240k@att.com>; MONTEIRO, FELIPE C <fm577c@att.com>; BARTRA, RICK <rb560u@att.com>; REDDY, CHINASUBBA <cr3938@att.com>; GORSHUNOV, ROMAN <roman.gorshunov@att.com>; KHUNTIA, SOUMITRA <sk698p@att.com>; KABANOV, DMITRII <dk370c@att.com>; Mark Burnett <mark.m.burnett@gmail.com>; VOLKOV, ANDREY <av903u@att.com>; avolkov@mirantis.com; Chris Wedgwood <cw@f00f.org>; PALLAV GUPTA <pallavgupta84@gmail.com>; GUPTA, SANGEET <sg774j@att.com>; zuul@review01.openstack.org; HUSSEY, SCOTT T <sh8121@att.com>; PACHECO, RODOLFO J <rp2723@att.com>; maxwell.li@nokia-sbell.com; SKELS, KASPARS <ks3019@att.com>
Subject: Re: Airship installation Questions

 

 

 

On Tue, Nov 6, 2018 at 3:42 AM Tu, Qiaolin (NSB - CN/Hangzhou) <qiaolin.tu@nokia-sbell.com> wrote:

Hi,

I also found that pod(ceph-mgr-6dc44fc75b-25n24) is always on the pending state. It report error as below:   Warning  FailedScheduling  1m (x125 over 36m)  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.

 

 

root@cab23-r720-11:~# kubectl get pods --all-namespaces

NAMESPACE     NAME                                                        READY     STATUS      RESTARTS   AGE

ceph          airship-ucp-ceph-provisioners-ceph-ns-key-generator-n5vzj   0/1       Completed   0          20m

ceph          ceph-cephfs-client-key-generator-656rd                      0/1       Completed   0          20m

ceph          ceph-cephfs-provisioner-676684f6bd-trf5h                    1/1       Running     0          20m

ceph          ceph-cephfs-provisioner-676684f6bd-zfk2s                    1/1       Running     0          20m

ceph          ceph-mds-6bfb74d9c7-btbt5                                   1/1       Running     0          37m

ceph          ceph-mds-6bfb74d9c7-qw4xl                                   1/1       Running     0          37m

ceph          ceph-mds-keyring-generator-dlqjt                            0/1       Completed   0          53m

ceph          ceph-mgr-6dc44fc75b-25n24                                   0/1       Pending     0          37m

ceph          ceph-mgr-6dc44fc75b-42mqz                                   1/1       Running     0          37m

ceph          ceph-mgr-keyring-generator-7dftx                            0/1       Completed   0          53m

ceph          ceph-mon-cfrf6                                              1/1       Running     0          41m

ceph          ceph-mon-check-6db6b569b6-67vvm                             1/1       Running     0          53m

ceph          ceph-mon-keyring-generator-55ps8                            0/1       Completed   0          53m

 

 

 

root@cab23-r720-11:~# kubectl describe pod ceph-mgr-6dc44fc75b-25n24 -n ceph

Name:           ceph-mgr-6dc44fc75b-25n24

Namespace:      ceph

Node:           <none>

Labels:         application=ceph

                component=mgr

                pod-template-hash=2870097316

                release_group=airship-ucp-ceph-client

Annotations:    configmap-bin-hash=d23610f5cc67014b596eb45b9d47ddb97997d6ec5fd3289fb264429a260f39a9

                configmap-etc-client-hash=0174be85f29398aa5cea5fccb9537e0e0f25ea2d2f280182e9e2506a1be1b115

Status:         Pending

IP:            

Controlled By:  ReplicaSet/ceph-mgr-6dc44fc75b

Init Containers:

  init:

    Image:      quay.io/stackanetes/kubernetes-entrypoint:v0.3.1

    Port:       <none>

    Host Port:  <none>

    Command:

      kubernetes-entrypoint

    Environment:

      POD_NAME:              ceph-mgr-6dc44fc75b-25n24 (v1:metadata.name)

      NAMESPACE:             ceph (v1:metadata.namespace)

      INTERFACE_NAME:        eth0

      PATH:                  /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/

      DEPENDENCY_SERVICE:    ceph:ceph-mon

      DEPENDENCY_JOBS:       ceph-storage-keys-generator,ceph-mgr-keyring-generator

      DEPENDENCY_DAEMONSET: 

      DEPENDENCY_CONTAINER: 

      DEPENDENCY_POD_JSON:  

      COMMAND:               echo done

    Mounts:

      /var/run/secrets/kubernetes.io/serviceaccount from ceph-mgr-token-pgbwp (ro)

  ceph-init-dirs:

    Image:      docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04

    Port:       <none>

    Host Port:  <none>

    Command:

      /tmp/init-dirs.sh

    Environment:

      CLUSTER:  ceph

    Mounts:

      /etc/ceph from pod-etc-ceph (rw)

      /run from pod-run (rw)

      /tmp/init-dirs.sh from ceph-client-bin (ro)

      /var/lib/ceph from pod-var-lib-ceph (rw)

      /var/run/secrets/kubernetes.io/serviceaccount from ceph-mgr-token-pgbwp (ro)

Containers:

  ceph-mgr:

    Image:       docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04

    Ports:       7000/TCP, 9283/TCP

    Host Ports:  7000/TCP, 9283/TCP

    Command:

      /mgr-start.sh

    Liveness:   exec [/tmp/mgr-check.sh liveness] delay=30s timeout=5s period=10s #success=1 #failure=3

    Readiness:  exec [/tmp/mgr-check.sh readiness] delay=30s timeout=5s period=10s #success=1 #failure=3

    Environment:

      CLUSTER:          ceph

      ENABLED_MODULES:  restful

status

prometheus

    Mounts:

      /etc/ceph from pod-etc-ceph (rw)

      /etc/ceph/ceph.client.admin.keyring from ceph-client-admin-keyring (ro)

      /etc/ceph/ceph.conf from ceph-client-etc (ro)

      /mgr-start.sh from ceph-client-bin (ro)

      /run from pod-run (rw)

      /tmp/mgr-check.sh from ceph-client-bin (ro)

      /var/lib/ceph from pod-var-lib-ceph (rw)

      /var/lib/ceph/bootstrap-mgr/ceph.keyring from ceph-bootstrap-mgr-keyring (rw)

      /var/run/secrets/kubernetes.io/serviceaccount from ceph-mgr-token-pgbwp (ro)

Conditions:

  Type           Status

  PodScheduled   False

Volumes:

  pod-etc-ceph:

    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium: 

  ceph-client-bin:

    Type:      ConfigMap (a volume populated by a ConfigMap)

    Name:      ceph-client-bin

    Optional:  false

  ceph-client-etc:

    Type:      ConfigMap (a volume populated by a ConfigMap)

    Name:      ceph-client-etc

    Optional:  false

  pod-var-lib-ceph:

    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium: 

  pod-run:

    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium:  Memory

  ceph-client-admin-keyring:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  ceph-client-admin-keyring

    Optional:    false

  ceph-bootstrap-mgr-keyring:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  ceph-bootstrap-mgr-keyring

    Optional:    false

  ceph-mgr-token-pgbwp:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  ceph-mgr-token-pgbwp

    Optional:    false

QoS Class:       BestEffort

Node-Selectors:  ceph-mgr=enabled

Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s

                 node.kubernetes.io/unreachable:NoExecute for 300s

Events:

  Type     Reason            Age                 From               Message

  ----     ------            ----                ----               -------

  Warning  FailedScheduling  1m (x125 over 36m)  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.

 

Best Regards!

Qiaolin Tu

NSB MN 5G ECE HZ CN2 SG04

Mobile:  +86 138 057 59684

E-Mail:   qiaolin.tu@nokia-sbell.com

 

 

From: Tu, Qiaolin (NSB - CN/Hangzhou)
Sent: Tuesday, November 06, 2018 1:46 PM
To: 'MEADOWS, ALAN' <am240k@att.com>; Kaspars Skels <kaspars.skels@gmail.com>
Cc: MONTEIRO, FELIPE C <fm577c@att.com>; BARTRA, RICK <rb560u@att.com>; REDDY, CHINASUBBA <cr3938@att.com>; GORSHUNOV, ROMAN <roman.gorshunov@att.com>; KHUNTIA, SOUMITRA <sk698p@att.com>; KABANOV, DMITRII <dk370c@att.com>; mark.m.burnett@gmail.com; VOLKOV, ANDREY <av903u@att.com>; avolkov@mirantis.com; cw@f00f.org; pallavgupta84@gmail.com; GUPTA, SANGEET <sg774j@att.com>; pete@port.direct; zuul@review01.openstack.org; HUSSEY, SCOTT T <sh8121@att.com>; PACHECO, RODOLFO J <rp2723@att.com>; Li, Maxwell (NSB - CN/Hangzhou) <maxwell.li@nokia-sbell.com>
Subject: Airship installation Questions

 

Hi,

Thanks very much for your kindly support.

I also encountered a problem when install multi-node airship. It blocked by 2 ceph-rgw pods. They crash off and run again and again. Pods error log as bellow:

 

root@cab23-r720-11:~# kubectl describe pod ceph-rgw-6ff5ff866d-4x66n -n ceph

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Scheduled 1m default-scheduler Successfully assigned ceph-rgw-6ff5ff866d-4x66n to cab23-r720-11

Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "pod-etc-ceph"

Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "pod-var-lib-ceph"

Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "pod-run"

Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "ceph-rgw-etc"

Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "ceph-rgw-bin"

Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "ceph-bootstrap-rgw-keyring"

Normal SuccessfulMountVolume 1m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "ceph-rgw-token-sjg9x"

Normal Pulled 1m kubelet, cab23-r720-11 Container image "quay.io/stackanetes/kubernetes-entrypoint:v0.3.1" already present on machine

 

Normal Started 1m kubelet, cab23-r720-11 Started container

Normal Pulled 1m kubelet, cab23-r720-11 Container image "docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04" already present on machine

Normal Created 1m kubelet, cab23-r720-11 Created container

Normal Started 1m kubelet, cab23-r720-11 Started container

 

Warning Unhealthy 18s (x6 over 1m) kubelet, cab23-r720-11 Readiness probe failed: Get http://10.97.38.105:8088/: dial tcp 10.97.38.105:8088: getsockopt: connection refused

 

 

Best Regards!

Qiaolin Tu

NSB MN 5G ECE HZ CN2 SG04

Mobile:  +86 138 057 59684

E-Mail:   qiaolin.tu@nokia-sbell.com

 

 

From: MEADOWS, ALAN <am240k@att.com>
Sent: Saturday, November 03, 2018 5:39 AM
To: Kaspars Skels <kaspars.skels@gmail.com>; Li, Maxwell (NSB - CN/Hangzhou) <maxwell.li@nokia-sbell.com>
Cc: MONTEIRO, FELIPE C <fm577c@att.com>; BARTRA, RICK <rb560u@att.com>; REDDY, CHINASUBBA <cr3938@att.com>; GORSHUNOV, ROMAN <roman.gorshunov@att.com>; KHUNTIA, SOUMITRA <sk698p@att.com>; KABANOV, DMITRII <dk370c@att.com>; mark.m.burnett@gmail.com; VOLKOV, ANDREY <av903u@att.com>; avolkov@mirantis.com; cw@f00f.org; pallavgupta84@gmail.com; GUPTA, SANGEET <sg774j@att.com>; pete@port.direct; zuul@review01.openstack.org; HUSSEY, SCOTT T <sh8121@att.com>; Tu, Qiaolin (NSB - CN/Hangzhou) <qiaolin.tu@nokia-sbell.com>; PACHECO, RODOLFO J <rp2723@att.com>
Subject: RE: Airship Version Questions

 

Hi Maxwell!

 

Great to see additional people trying Airship out.

 

Treasuremap does change quite frequently.  In fact, this is from a lot of the great work Kaspars has been doing – building a CI/CD pipeline to not only keep treasuremap fresh and referencing the latest working versions of Airship components but also validating that the YAML documents are able to provision a multi-node baremetal physical environment by way of third-party gates.

 

This gives us a high degree of confidence that no matter what version of treasuremap you have decided to start working with to deploy your own environments it should work.

 

To be sure, we do understand when getting started it is nice to work with a stable target that isn’t constantly changing.    Kaspars has introduced tagging to provide a human readable reference (using dates).  These are not on any particular cadence, but usually follow more in-depth tested release beyond just automated CI/CD using the manifests:

 

https://github.com/openstack/airship-treasuremap/releases

 

For right now with regard to pegleg, the best way to leverage the version of pegleg that has been tested with the treasuremap manifests is to pull it the one used in the baremetal CI/CD gate test directly from the target tag Jenkinsfile.  For example, for v18.11.01:

 

https://github.com/openstack/airship-treasuremap/blob/v18.11.01/tools/gate/Jenkinsfile#L15

 

Please feel free to get on the airship mailing list (airshipit.org) to connect with a wider audience or our IRC channel!

 

Alan Meadows 

 

From: Kaspars Skels [mailto:kaspars.skels@gmail.com]
Sent: Friday, November 02, 2018 2:27 PM
To: maxwell.li@nokia-sbell.com
Cc: MONTEIRO, FELIPE C <fm577c@att.com>; BARTRA, RICK <rb560u@att.com>; REDDY, CHINASUBBA <cr3938@att.com>; GORSHUNOV, ROMAN <roman.gorshunov@att.com>; KHUNTIA, SOUMITRA <sk698p@att.com>; KABANOV, DMITRII <dk370c@att.com>; mark.m.burnett@gmail.com; VOLKOV, ANDREY <av903u@att.com>; avolkov@mirantis.com; cw@f00f.org; pallavgupta84@gmail.com; GUPTA, SANGEET <sg774j@att.com>; pete@port.direct; zuul@review01.openstack.org; HUSSEY, SCOTT T <sh8121@att.com>; qiaolin.tu@nokia-sbell.com; MEADOWS, ALAN <am240k@att.com>; PACHECO, RODOLFO J <rp2723@att.com>
Subject: Re: Airship Version Questions

 

+ Alan/Rodolfo

 

 

 

On Thu, Nov 1, 2018 at 3:37 AM Li, Maxwell (NSB - CN/Hangzhou) <maxwell.li@nokia-sbell.com> wrote:

Hi Airship Team:

 

I have a problem when I deploy airship on my server. There are new commits in airship-pegleg and airship-treasuremap, almost erveryday. And there are no tags or branches in these code repositories. Could I get a commit ID about these repositories so that I could deploy airship.

 

Thanks a lot!

 

Best Regards!

Maxwell Li