<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msochpdefault, li.msochpdefault, div.msochpdefault
{mso-style-name:msochpdefault;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.emailstyle17
{mso-style-name:emailstyle17;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi Kaspars,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you for the response and the hint regarding the ceph issue. We will revisit our ceph config and either go for 3 directories or a lower replication level.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best regards<o:p></o:p></p>
<p class="MsoNormal">Georg<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> SKELS, KASPARS <ks3019@att.com> <br>
<b>Sent:</b> Thursday, February 7, 2019 4:55 PM<br>
<b>To:</b> Georg Kunz <georg.kunz@ericsson.com>; airship-discuss@lists.airshipit.org<br>
<b>Cc:</b> Stefan Behrens <stefan.behrens@ericsson.com>; Rihab Banday <rihab.banday@ericsson.com>; BIRLEY, PETE <pb269f@att.com>; MCEUEN, MATT <MM9745@att.com>; REDDY, CHINASUBBA <cr3938@att.com>; HELER, MATTHEW <mh935s@att.com><br>
<b>Subject:</b> RE: DNS and Ceph issues during deployment<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Hi Georg,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">the kube-dns seems an issue but I wonder if this could be
<u>the failing</u> issue as the treasuremap reference (airship-seaworthy) site is being tested on frequent bases especially the tags released (here some info on it,
<a href="https://airship-treasuremap.readthedocs.io/en/latest/seaworthy.html" target="_blank">
https://airship-treasuremap.readthedocs.io/en/latest/seaworthy.html</a>).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">I believe that current ceph configuration in globals require at least 3 OSDs (to fulfill 3x replication).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Normally each OSD represents a disk - but you may use 3 folders on the same disk that could lead to a success.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">There is also possibility to fiddle with replication settings and turn them all to 1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><a href="https://github.com/openstack/openstack-helm-infra/blob/master/ceph-client/values.yaml#L186" target="_blank">https://github.com/openstack/openstack-helm-infra/blob/master/ceph-client/values.yaml#L186</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Still - would very much recommend using more disks for production like bare-metal deployments.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">FYI - the kube-dns item is likely something that should be fixed.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Kindly, Kaspars<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:black">
<hr size="2" width="100%" align="center">
</span></div>
<div id="divRpF868943">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"> Georg Kunz [georg.kunz@ericsson.com]<br>
<b>Sent:</b> Thursday, February 07, 2019 7:37 AM<br>
<b>To:</b> <a href="mailto:airship-discuss@lists.airshipit.org">airship-discuss@lists.airshipit.org</a><br>
<b>Cc:</b> Stefan Behrens; Rihab Banday<br>
<b>Subject:</b> [Airship-discuss] DNS and Ceph issues during deployment</span><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Hi all,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">We are trying to install Airship, but ran into some issues I’d like to bring up here. Specifically, we base our config on treasuremap tag v19.02.04 and see the following two problems:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">i) The ceph-rbd-provisioner pod tries to use service kube-dns although only coredns is deployed by Airship.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">Log:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">kubectl logs -n ceph ceph-rbd-provisioner-85f5b5bb8b-8vzc7</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">I0207 11:21:51.007104 1 leaderelection.go:156] attempting to acquire leader lease...</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">I0207 11:21:51.014035 1 leaderelection.go:178] successfully acquired lease to provision for pvc ucp/mysql-data-mariadb-server-0</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">I0207 11:21:51.048705 1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"ucp", Name:"mysql-data-mariadb-server-0", UID:"6e479917-2ac7-11e9-ab70-246e9657cd40",
APIVersion:"v1", ResourceVersion:"10655", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "ucp/mysql-data-mariadb-server-0"</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">E0207 11:21:51.249161 1 provision.go:199]
<b>error getting kube-dns service: services "kube-dns" not found</b></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">I0207 11:22:16.379111 1 leaderelection.go:204] stopped trying to renew lease to provision for pvc ucp/mysql-data-mariadb-server-1, timeout reached</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Related change: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes-2Dincubator_external-2Dstorage_pull_861_commits_99098beac2ff91e788b027f0e27d9e7f5ec5b769&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=rgoV-81VfVoElaVyHhCe0EhKF_bYMlpjF9rTMvG-gBc&s=OQ_twK6I4cvTerkUhyowBcmxM9j6aXqJGj4HLQ7TfVU&e=" target="_blank">
https://github.com/kubernetes-incubator/external-storage/pull/861/commits/99098beac2ff91e788b027f0e27d9e7f5ec5b769</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">We are wondering if that is a version selection problem (non-matching Kubernetes and DNS service), but we haven’t touched the versions.yaml.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">ii) The ceph-rgw pod is not started on the Genesis node (nor on any other node; this is during an early phase before other nodes are provisioned); no related error in logs; all other ceph pods seem to be there.
As a result, PVCs are not bound and database pods cannot start.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">site/airship-seaworthy/profiles/host/cp_r630.yaml: ceph-rgw: enabled<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">site/airship-seaworthy/profiles/host/cp_r630.yaml: tenant-ceph-rgw: enabled<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"># kubectl exec -n ceph ceph-mon-p7k6d -- ceph -s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> cluster:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> id: 7b7576f4-3358-4668-9112-100440079807<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> health: HEALTH_WARN<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> 1 MDSs report slow metadata IOs<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> Reduced data availability: 143 pgs inactive<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> Degraded data redundancy: 143 pgs undersized<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> services:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> mon: 1 daemons, quorum salmon-11<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> mgr: salmon-11(active)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> mds: cephfs-1/1/1 up {0=mds-ceph-mds-6774766c6-2n9tm=up:creating}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> osd: 1 osds: 1 up, 1 in<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> data:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> pools: 18 pools, 143 pgs<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> objects: 0 objects, 0 B<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> usage: 111 MiB used, 372 GiB / 372 GiB avail<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> pgs: 100.000% pgs not active<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> 143 undersized+peered<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Did anybody run into this issue before or could point us in the right direction?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Thanks a lot!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Best regards<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Georg<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>