<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Didot;}
@font-face
{font-family:"Bradley Hand";}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:897008028;
mso-list-template-ids:2039643072;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Dmitry,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To be clear I see three phases to this effort:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%">1) Just put the cloud-init for network config
<i>verbatim </i>in the BMH doc</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> for the advanced network configuration (or rather in the example
below, within a secret referenced by the BMH doc – as is the metal3-io approach today).<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%">2) Put the intended network configuration data in another CR, or enhance BMH CRD to
store it separately and then <i>generate</i> the appropriate cloud-init data to configure the networking – potentially leveraging some of the hardware detected by ironic hardware inventory discovery to help with that process. I think this is what you’re currently
thinking but I think we may be going a step ahead of what’s required <i>initially.
</i>This is my main point.</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%">3) Finally, I think the last phase is introducing IPAM support so users do not have
to explicitly define every interface address for hosts – airship, an ipam operator, or whatever can help identify the next available.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The baremetalhost_types you refer to would be populated by ironic inventory. I think trying to represent the network configuration in some structured way, and
potentially use some of the hardware configuration discovered as well, in order to
<i>generate </i>cloud-init is more in-line with #2 above.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Really, I’m talking about #1 right now and to be specific about what I’m suggesting, let me propose the following simple solution. Imagine a baremetal Ubuntu
host (which we can easily imitate with virtual machines). For this example, let’s pretend we want this host to have two interfaces – ens1 and ens4 -- both with static addresses. You can obviously expand this simple example to account for N interfaces, with
bonding, VLANs, and so on.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I would have the following cloud-init for my theoretical “host10”:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">#cloud-config<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">write_files:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">- path: /etc/network/interfaces.d/10-ens1.cfg<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> owner: root:root<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> permissions: '0644'<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> content: |<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> auto ens1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> iface ens1 inet static<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> address 10.1.1.10<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> netmask 255.255.255.0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> gateway 10.1.1.1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">- path: /etc/network/interfaces.d/11-ens4.cfg<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> owner: root:root<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> permissions: '0644'<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> content: |<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> auto ens4<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> iface ens4 inet static<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> address 10.4.1.10<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> netmask 255.255.255.0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">runcmd:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">- [ ifup, ens1]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">- [ ifup, ens4]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I would create a Secret for this user-data, “host10-user-data”. This would theoretically be where this is stored declaratively as a document in my set.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">apiVersion: v1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">data:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">
<span style="background:yellow;mso-highlight:yellow">userData: $(base64 -w 0 <<data above>>)</span><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">kind: Secret<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">metadata:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> name: host10-user-data<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> namespace: metal3<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">type: Opaque<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">I would create a BMH CR for host10:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">apiVersion: v1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">kind: Secret<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">metadata:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> name: host10-bmc-secret<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">type: Opaque<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">data:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> username: YWRtaW4=<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> password: cGFzc3dvcmQ=<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">---<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">apiVersion: metal3.io/v1alpha1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">kind: BareMetalHost<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">metadata:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> name: host10<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">spec:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> online: true<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> bootMACAddress: 00:3b:8b:0c:ec:8b<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> bmc:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> address: ipmi://192.168.111.1:6230<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"> credentialsName: master-0-bmc-secret<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">
<span style="background:yellow;mso-highlight:yellow">userData:<o:p></o:p></span></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D;background:yellow;mso-highlight:yellow"> name: host10-user-data<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D;background:yellow;mso-highlight:yellow"> namespace: metal3</span><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:Consolas;color:#1F497D">---<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">At this point, what I’ve done is fully declare the advanced network configuration for “host10” within the constraints of what exists today – in other words, this
works right now in metal3-io. I could extract this cloud-init for whatever host is being used as the ephemeral host as well for use in ISO generation.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">My point is basically I think going with #1 now so we can achieve E2E functionality seems like the most expedient approach.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Let me know if any aspect of this doesn’t make sense.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Alan Meadows<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Dmitry Ukov [mailto:dukov@mirantis.com]
<br>
<b>Sent:</b> Friday, September 20, 2019 10:18 AM<br>
<b>To:</b> MEADOWS, ALAN <am240k@att.com><br>
<b>Cc:</b> PACHECO, RODOLFO J <rp2723@att.com>; airship-discuss@lists.airshipit.org<br>
<b>Subject:</b> Re: [Airship-discuss] IPAM for Airship<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Alan, <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">> <span style="color:#1F497D">MaaS behaves as an IPAM service – for the most part all it does is enforce the static assignments we declare within Drydock BaremetalNode documents as in here:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_airshipit_treasuremap_blob_master_site_seaworthy_baremetal_nodes.yaml-23L48-2DL86&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OI-s0Rr_BtNj0OcKWGZyuA&m=j4XnOJ5N_gtPrZSJlEi7x4Gdbn474aeBXiNi4oyXMIo&s=CT5Rs_ulfgsRJ_aRLjbQn9cMYSNnTrEj1DnUfBUVmkU&e=" target="_blank">https://github.com/airshipit/treasuremap/blob/master/site/seaworthy/baremetal/nodes.yaml#L48-L86</a><br>
I've missed this part :(</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><o:p> </o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">> I missed the latest bootstrap-sig call, however when we were initially discussing this one very simple way to get <i>started</i> with this is to represent
the network configuration <i>as </i>cloud-init data in the BareMetalHost (BMH) CRD. There is support for that now without introducing any additional CRDs or fields.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Unfortunately BareMetalHost does not have all information. Here is what is supported <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_metal3-2Dio_baremetal-2Doperator_blob_master_pkg_apis_metal3_v1alpha1_baremetalhost-5Ftypes.go-23L249&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OI-s0Rr_BtNj0OcKWGZyuA&m=j4XnOJ5N_gtPrZSJlEi7x4Gdbn474aeBXiNi4oyXMIo&s=-4AQiMysvbUC-zagc0GLr9UMPmjY6bq3tTIFLZHJEW8&e=">https://github.com/metal3-io/baremetal-operator/blob/master/pkg/apis/metal3/v1alpha1/baremetalhost_types.go#L249</a>.
NIC structure is assigned as a property of HardwareDetails structure which is associated to Status property of BareMetalHost. AFAIK Status property is not supposed to be a source of data for CR defined in operator, it's filled in after reconcile event checks
resource status. So technically we can not use this property for data. Moreover there is not Default gw and dns server setting. <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><o:p> </o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">As an alternative temporary solution (until we decide what to do with IPAM) we can put DNS server in a cluster config (i.e. airshipctl config) or supply it as CLI flag for isogen
and calculate default gateway as a first address of a network<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><o:p> </o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Sep 20, 2019 at 8:14 PM MEADOWS, ALAN <<a href="mailto:am240k@att.com">am240k@att.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">I think we may intertwining two separate needs.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">The first and most urgent need is to understand how we will declare the static IP assignments for each baremetal host up front, much like we do within
Airship 1.0. I disagree that MaaS behaves as an IPAM service – for the most part all it does is enforce the static assignments we declare within Drydock BaremetalNode documents as in here:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_airshipit_treasuremap_blob_master_site_seaworthy_baremetal_nodes.yaml-23L48-2DL86&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OI-s0Rr_BtNj0OcKWGZyuA&m=j4XnOJ5N_gtPrZSJlEi7x4Gdbn474aeBXiNi4oyXMIo&s=CT5Rs_ulfgsRJ_aRLjbQn9cMYSNnTrEj1DnUfBUVmkU&e=" target="_blank">https://github.com/airshipit/treasuremap/blob/master/site/seaworthy/baremetal/nodes.yaml#L48-L86</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">I missed the latest bootstrap-sig call, however when we were initially discussing this one very simple way to get
<i>started</i> with this is to represent the network configuration <i>as </i>cloud-init data in the BareMetalHost (BMH) CRD. There is support for that now without introducing any additional CRDs or fields.
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">The second need you are addressing, where files/CRs like “common-addresses” become more important and have definitions for all the various subnets--abstracted
from specific machines, is supporting a more liberal IP assignment mechanism in Airship 2.0, where users need only define subnets, and we can support allocating addresses automatically. This is where IPAM truly becomes a requirement but I think it is premature
to look at this and feel this may be a subsequent step to delivering static functionality.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">Alan Meadows</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> PACHECO, RODOLFO J
<br>
<b>Sent:</b> Friday, September 20, 2019 9:07 AM<br>
<b>To:</b> Dmitry Ukov <<a href="mailto:dukov@mirantis.com" target="_blank">dukov@mirantis.com</a>><br>
<b>Cc:</b> <a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a><br>
<b>Subject:</b> Re: [Airship-discuss] IPAM for Airship<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Agree, we expect the document that defines this information to be generic.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Meaning it will be used by both ephemeral and target clusters.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:Didot;color:#941100">Regards</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:13.5pt;font-family:"Bradley Hand";color:#0433FF">Rodolfo Pacheco</span></b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;color:black"> </span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.5pt;color:black">Home/Office</span></b><span style="font-size:10.5pt;color:black"> 732 5337671</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="color:black">From:
</span></b><span style="color:black">Dmitry Ukov <<a href="mailto:dukov@mirantis.com" target="_blank">dukov@mirantis.com</a>><br>
<b>Date: </b>Friday, September 20, 2019 at 11:49 AM<br>
<b>To: </b>"PACHECO, RODOLFO J" <<a href="mailto:rp2723@att.com" target="_blank">rp2723@att.com</a>><br>
<b>Cc: </b>"<a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a>" <<a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a>><br>
<b>Subject: </b>Re: [Airship-discuss] IPAM for Airship</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Rodolfo,
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks for the links. This is what is needed for ephemeral node!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">What is about target cluster nodes? We need to manage their network configuration some how (assign IPs, set up bonding etc). MAAS is used as IPAM service in Airship 1.0 but it will
be replaced in favor of Metal3. So we may want to have the same mechanism of ip allocation for ephemeral cluster and for target cluster.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Fri, Sep 20, 2019 at 7:47 PM PACHECO, RODOLFO J <<a href="mailto:rp2723@att.com" target="_blank">rp2723@att.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Forgot to add, Jerome’s POC has an example that looks closer to a CR expectations
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_keleustes_treasuremap_blob_kustomize_site_seaworthy_networks_common-2Daddresses.yaml&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Bjzj_RciGLj21LnNXoD5dw&m=eSPJlb86ECKHCgBeXNfbCrcBGq7doKl90sZz1JU-HJM&s=5hGWlv6RN10N-cPbl_8lnzrGFZRmONETR55X1GdKN7g&e=" target="_blank">
https://github.com/keleustes/treasuremap/blob/kustomize/site/seaworthy/networks/common-addresses.yaml</a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:Didot;color:#941100">Regards</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:13.5pt;font-family:"Bradley Hand";color:#0433FF">Rodolfo Pacheco</span></b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;color:black"> </span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.5pt;color:black">Home/Office</span></b><span style="font-size:10.5pt;color:black"> 732 5337671</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="color:black">From:
</span></b><span style="color:black">"PACHECO, RODOLFO J" <<a href="mailto:rp2723@att.com" target="_blank">rp2723@att.com</a>><br>
<b>Date: </b>Friday, September 20, 2019 at 11:40 AM<br>
<b>To: </b>Dmitry Ukov <<a href="mailto:dukov@mirantis.com" target="_blank">dukov@mirantis.com</a>>, "<a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a>" <<a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a>><br>
<b>Subject: </b>Re: [Airship-discuss] IPAM for Airship</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Dmitry<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">In Airship 1.0 we had such a document . I expect for 2.0 we need an equivalent CR for this type of information<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Is what we have been calling Catalog or DB like Document’s<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Here is an example from treasuremap<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__opendev.org_airship_treasuremap_src_branch_master_site_seaworthy_networks_common-2Daddresses.yaml&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Bjzj_RciGLj21LnNXoD5dw&m=eSPJlb86ECKHCgBeXNfbCrcBGq7doKl90sZz1JU-HJM&s=wvOJ_Dykh2xTn3clmits97RK3Pk5sSKbS6JAdw8eE1s&e=" target="_blank">https://opendev.org/airship/treasuremap/src/branch/master/site/seaworthy/networks/common-addresses.yaml</a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I think that is what you are thinking about.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:Didot;color:#941100">Regards</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:13.5pt;font-family:"Bradley Hand";color:#0433FF">Rodolfo Pacheco</span></b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;color:black"> </span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.5pt;color:black">Home/Office</span></b><span style="font-size:10.5pt;color:black"> 732 5337671</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="color:black">From:
</span></b><span style="color:black">Dmitry Ukov <<a href="mailto:dukov@mirantis.com" target="_blank">dukov@mirantis.com</a>><br>
<b>Date: </b>Friday, September 20, 2019 at 11:37 AM<br>
<b>To: </b>"<a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a>" <<a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a>><br>
<b>Subject: </b>[Airship-discuss] IPAM for Airship</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hello,
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I'm working on cloud-init generator as a part of airshipctl bootstrap isogen command.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This cloud init will be used to deploy K8s on ephemeral node.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Following information is needed to set up ephemeral node:<o:p></o:p></p>
</div>
<div>
<ul type="disc">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
IP address<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Net mask <o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Default Gateway<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
DNS server address<o:p></o:p></li></ul>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">There is no CRD/CR that actually has this information. For example BareMetalHost resource has IP and Netmask as a part of Status field (and determined dynamically by beremetal operator)
and there is no info about Default GW and DNS.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This actually leads us to the situation where we need to have IPAM (ip address management) operator and appropriate CRs/CRDs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Unfortunately there is not much IPAM services that can act as K8s operator (I've found only one actually <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_inwinstack_ipam&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Bjzj_RciGLj21LnNXoD5dw&m=c8c5DfwYKVyHfFy7VcpEn5JOI4G1u4qxCaCy9cDsgAI&s=d7costGUiy-0u1vBkTFbV8Gf4-BIyHE3PGYKZ49NK7k&e=" target="_blank">https://github.com/inwinstack/ipam</a> and
it has limitations I guess). <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">As an alternative we have to implement IPAM as a part of Airship 2.0.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">What do you think?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Kind regards
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Dmitry Ukov<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Principal Deployment Engineer<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Mirantis, Inc.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Kind regards
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Dmitry Ukov<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Principal Deployment Engineer<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Mirantis, Inc.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">Kind regards<o:p></o:p></p>
<div>
<p class="MsoNormal">Dmitry Ukov<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Principal Deployment Engineer<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">Mirantis, Inc.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>