<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We are trying to install Airship, but ran into some issues I’d like to bring up here. Specifically, we base our config on treasuremap tag v19.02.04 and see the following two problems:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">i) The ceph-rbd-provisioner pod tries to use service kube-dns although only coredns is deployed by Airship.<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Log:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">kubectl logs -n ceph ceph-rbd-provisioner-85f5b5bb8b-8vzc7<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">I0207 11:21:51.007104 1 leaderelection.go:156] attempting to acquire leader lease...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">I0207 11:21:51.014035 1 leaderelection.go:178] successfully acquired lease to provision for pvc ucp/mysql-data-mariadb-server-0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">I0207 11:21:51.048705 1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"ucp", Name:"mysql-data-mariadb-server-0", UID:"6e479917-2ac7-11e9-ab70-246e9657cd40",
APIVersion:"v1", ResourceVersion:"10655", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "ucp/mysql-data-mariadb-server-0"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">E0207 11:21:51.249161 1 provision.go:199]
<b>error getting kube-dns service: services "kube-dns" not found<o:p></o:p></b></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">I0207 11:22:16.379111 1 leaderelection.go:204] stopped trying to renew lease to provision for pvc ucp/mysql-data-mariadb-server-1, timeout reached<o:p></o:p></span></p>
<p class="MsoNormal">Related change: <a href="https://github.com/kubernetes-incubator/external-storage/pull/861/commits/99098beac2ff91e788b027f0e27d9e7f5ec5b769">
https://github.com/kubernetes-incubator/external-storage/pull/861/commits/99098beac2ff91e788b027f0e27d9e7f5ec5b769</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We are wondering if that is a version selection problem (non-matching Kubernetes and DNS service), but we haven’t touched the versions.yaml.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">ii) The ceph-rgw pod is not started on the Genesis node (nor on any other node; this is during an early phase before other nodes are provisioned); no related error in logs; all other ceph pods seem to be there. As a result, PVCs are not
bound and database pods cannot start.<o:p></o:p></p>
<p class="MsoNormal">site/airship-seaworthy/profiles/host/cp_r630.yaml: ceph-rgw: enabled<o:p></o:p></p>
<p class="MsoNormal">site/airship-seaworthy/profiles/host/cp_r630.yaml: tenant-ceph-rgw: enabled<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"># kubectl exec -n ceph ceph-mon-p7k6d -- ceph -s<o:p></o:p></p>
<p class="MsoNormal"> cluster:<o:p></o:p></p>
<p class="MsoNormal"> id: 7b7576f4-3358-4668-9112-100440079807<o:p></o:p></p>
<p class="MsoNormal"> health: HEALTH_WARN<o:p></o:p></p>
<p class="MsoNormal"> 1 MDSs report slow metadata IOs<o:p></o:p></p>
<p class="MsoNormal"> Reduced data availability: 143 pgs inactive<o:p></o:p></p>
<p class="MsoNormal"> Degraded data redundancy: 143 pgs undersized<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> services:<o:p></o:p></p>
<p class="MsoNormal"> mon: 1 daemons, quorum salmon-11<o:p></o:p></p>
<p class="MsoNormal"> mgr: salmon-11(active)<o:p></o:p></p>
<p class="MsoNormal"> mds: cephfs-1/1/1 up {0=mds-ceph-mds-6774766c6-2n9tm=up:creating}<o:p></o:p></p>
<p class="MsoNormal"> osd: 1 osds: 1 up, 1 in<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> data:<o:p></o:p></p>
<p class="MsoNormal"> pools: 18 pools, 143 pgs<o:p></o:p></p>
<p class="MsoNormal"> objects: 0 objects, 0 B<o:p></o:p></p>
<p class="MsoNormal"> usage: 111 MiB used, 372 GiB / 372 GiB avail<o:p></o:p></p>
<p class="MsoNormal"> pgs: 100.000% pgs not active<o:p></o:p></p>
<p class="MsoNormal"> 143 undersized+peered<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Did anybody run into this issue before or could point us in the right direction?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks a lot!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best regards<o:p></o:p></p>
<p class="MsoNormal">Georg<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>