<div dir="ltr">Darren,<div><br></div><div>Thank you so much for your time and reply. These are very helpful and important points you mentioned in your email. </div><div><br></div><div>For [1], we are already taking care of this. Exceptions will be thrown if there is any problem with Disks. </div><div>For [2], it seems a very interesting point but according to my knowledge, these are OS level configurations. We will look into it in future but right now, it is beyond our scope. </div><div><br></div><div>Thank you for showing interest, once again. </div><span class="gmail-im" style="color:rgb(80,0,80)"><br class="gmail-Apple-interchange-newline"></span><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Warm Regards,</div><div><br></div>Zainub Wahid<div>Network Design Engineer II,</div><div><span style="color:rgb(34,34,34);font-family:Arial,sans-serif;font-size:13px;white-space:pre-wrap">xFlow Research Inc.</span><br></div><div><span style="color:rgb(34,34,34);font-family:Arial,sans-serif;font-size:13px;white-space:pre-wrap">+923044506900</span></div><div><span style="color:rgb(34,34,34);font-family:Arial,sans-serif;font-size:13px;white-space:pre-wrap"><a href="mailto:zainub.wahid@xflowresearch.com" target="_blank">zainub.wahid@xflowresearch.com</a></span></div><div><a href="http://www.xFlowResearch.com" target="_blank">www.xFlowResearch.com</a></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 1, 2021 at 7:20 PM DEJAEGER, DARREN R <<a href="mailto:dd118r@att.com">dd118r@att.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="gmail-m_-7546456153377017192WordSection1">
<p class="MsoNormal">Matt,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I agree with your thoughts on making available both approaches. A couple things to consider:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="gmail-m_-7546456153377017192MsoListParagraph" style="margin-left:0in">Regarding disk failures: would these disks even be available to be selected for any sort of RAID if they’re bad? If not, we’d quickly hit an issue with a mismatched number of disks
vs. what’s declared (at least today).<u></u><u></u></li><li class="gmail-m_-7546456153377017192MsoListParagraph" style="margin-left:0in">Can we allow other disk/raid-related config to be tuned? I’m thinking Stripe size, Read policy, Write policy?<u></u><u></u></li></ol>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Just some food for thought.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Regards,<u></u><u></u></p>
<p class="MsoNormal">Darren<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">"MCEUEN, MATT" <<a href="mailto:MM9745@att.com" target="_blank">MM9745@att.com</a>><br>
<b>Date: </b>Tuesday, June 1, 2021 at 10:04 AM<br>
<b>To: </b>Zainub Wahid <<a href="mailto:zainub.wahid@xflowresearch.com" target="_blank">zainub.wahid@xflowresearch.com</a>>, "<a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a>" <<a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a>>, "SPHICAS, PHIL G" <<a href="mailto:ps3910@att.com" target="_blank">ps3910@att.com</a>>, "OBERFIELD, GREG" <<a href="mailto:go4243@att.com" target="_blank">go4243@att.com</a>>, "DEJAEGER, DARREN R" <<a href="mailto:dd118r@att.com" target="_blank">dd118r@att.com</a>>, "STEIN, JARED
W" <<a href="mailto:js5175@att.com" target="_blank">js5175@att.com</a>><br>
<b>Cc: </b>"Williams, J T" <<a href="mailto:J.T.Williams@dell.com" target="_blank">J.T.Williams@dell.com</a>>, Airship <<a href="mailto:airship@xflowresearch.com" target="_blank">airship@xflowresearch.com</a>>, "<a href="mailto:richard_pioso@dell.com" target="_blank">richard_pioso@dell.com</a>" <<a href="mailto:richard_pioso@dell.com" target="_blank">richard_pioso@dell.com</a>><br>
<b>Subject: </b>RE: [Airship-discuss] RAID Config Implementation<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">Refreshing this in post-holiday inboxes: Phil, Craig, Greg, Darren, (& anyone else with experience with RAID config today) – please take a look at my thoughts below and let me know if that lines up with what you would find most useful
as well. Thanks!<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Also, Merging Mahnoor’s comments with the email thread:<u></u><u></u></p>
<p class="MsoNormal">(quote)<u></u><u></u></p>
<p class="MsoNormal">Thank you for your email, Matt.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">About the Airship use cases:<u></u><u></u></p>
<p class="MsoNormal">1. The strongly declarative one would be easy to support via Metal3, to the best of my knowledge. This would be perfect as long as the Airship user is okay with finding out the disk and RAID controller names through other means (using iDRAC,
for example). It would also be an easier and valuable implementation.<u></u><u></u></p>
<p class="MsoNormal">2. Metal3 will perform the disk selection itself (or delegate to Ironic). <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Providing some level of guarantee for the disk failures would contribute to a reliable user experience. For the declarative case, we can perhaps ensure that the specified disks are functional. For the specification with hints, I suggest
making the disk capacity a metric that must always be satisfied (e.g. x GB always provided). <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Input on these requirements is much appreciated, so we can carry the discussion forward with the Metal3 community and get to implementation. <u></u><u></u></p>
<p class="MsoNormal">Thank you so much!<u></u><u></u></p>
<p class="MsoNormal">(end quote)<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> MCEUEN, MATT <br>
<b>Sent:</b> Wednesday, May 26, 2021 4:17 PM<br>
<b>To:</b> Zainub Wahid <<a href="mailto:zainub.wahid@xflowresearch.com" target="_blank">zainub.wahid@xflowresearch.com</a>>; <a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a>; SPHICAS, PHIL G <<a href="mailto:ps3910@att.com" target="_blank">ps3910@att.com</a>>; OBERFIELD, GREG <<a href="mailto:go4243@att.com" target="_blank">go4243@att.com</a>>; DEJAEGER, DARREN R <<a href="mailto:dd118r@att.com" target="_blank">dd118r@att.com</a>>; STEIN, JARED W <<a href="mailto:js5175@att.com" target="_blank">js5175@att.com</a>><br>
<b>Cc:</b> Williams, J T <<a href="mailto:J.T.Williams@dell.com" target="_blank">J.T.Williams@dell.com</a>>; Airship <<a href="mailto:airship@xflowresearch.com" target="_blank">airship@xflowresearch.com</a>>; <a href="mailto:richard_pioso@dell.com" target="_blank">richard_pioso@dell.com</a><br>
<b>Subject:</b> RE: [Airship-discuss] RAID Config Implementation<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks for sharing & summarizing this, Zainub.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Phil, Craig, Greg, Darren, (& anyone else with experience with RAID config today) – here are my thoughts; please refine/agree/disagree with them so we can give the Metal3 community good feedback on the interface we need for declarative
RAID config.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">First: long term, we want to be able to declaratively configure all the BIOS settings that need to be configured. Steps like this RAID config are a valuable step in that direction.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Second: I think there are use cases for both of these setups, and I’d advocate for M3 facilitating both of them in the long run*:<u></u><u></u></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="gmail-m_-7546456153377017192MsoListParagraph" style="margin-left:0in">Strongly declarative: I want a RAID-N array on ControllerX with disks [DiskY, DiskZ]<u></u><u></u></li><li class="gmail-m_-7546456153377017192MsoListParagraph" style="margin-left:0in">Declarative with hints: I want a RAID-N array with two SSDs (and I don’t care what disk or what controller – you figure it out, M3!)<u></u><u></u></li></ol>
<p class="MsoNormal">* Rook supports something similar – “use all disks:yes/no”, or alternately “use these specific disks:…”
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Third: I think that #1 is a good fit for our most common use cases, where we put a lot of work into a reference definition, and then stamp it out many times on
<i>identical</i> hardware. This is also the simplest implementation and would probably get something usable into Airship’s hands fastest.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Does that sound right to you all?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">The other thing to consider is thresholding when disks are bad. I.e. a config item that says something like, “tolerate disk failures as long as I get Xgb” or “as long as I get 3 disks” or something. Do you think this is critical enough
to be needed on Day 1?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks!<u></u><u></u></p>
<p class="MsoNormal">Matt<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Zainub Wahid <<a href="mailto:zainub.wahid@xflowresearch.com" target="_blank">zainub.wahid@xflowresearch.com</a>>
<br>
<b>Sent:</b> Tuesday, May 25, 2021 11:50 AM<br>
<b>To:</b> <a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a><br>
<b>Cc:</b> Williams, J T <<a href="mailto:J.T.Williams@dell.com" target="_blank">J.T.Williams@dell.com</a>>; Airship <<a href="mailto:airship@xflowresearch.com" target="_blank">airship@xflowresearch.com</a>>;
<a href="mailto:richard_pioso@dell.com" target="_blank">richard_pioso@dell.com</a><br>
<b>Subject:</b> [Airship-discuss] RAID Config Implementation<u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">Hello everyone,<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">As discussed in today's meeting, this is related to the demo represented by Noor, last year, related to performing RAID configurations in Airship via Metal3 (and Ironic).<br>
Add RAID to example hardwareprofile (I50c82ce5) · Gerrit Code Review (<a href="https://urldefense.com/v3/__http:/opendev.org__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyvEP2e5U$" target="_blank">opendev.org</a>)<br>
<br>
Through Airship, the user should be able to perform RAID configurations on their hardware. The user can specify the RAID controller names and their associated disks, along with the required RAID level, the desired number of disks and the amount of storage required.
<br>
Here is the proposal for specifying the RAID configuration for a bare metal host via Metal3:<br>
<a href="https://urldefense.com/v3/__https:/github.com/metal3-io/metal3-docs/pull/148/files__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyGgo40Mo$" target="_blank">https://github.com/metal3-io/metal3-docs/pull/148/files</a><br>
<br>
In terms of the CRD, the current implementation of the RAID has the following sections in it, for hardware RAID:<br>
spec:<br>
raid:<br>
hardwareRAIDVolumes:<br>
- sizeGibiBytes: 1000<br>
level: 1<br>
name: volume-name<br>
rotational: true<br>
numberOfPhysicalDisks: 2<br>
We propose to add fields into the CRD, in the following manner:<br>
spec:<br>
raid:<br>
hardwareRAIDVolumes:<br>
- sizeGibiBytes: 1000<br>
level: 1<br>
name: volume-1<br>
rotational: true<br>
numberOfPhysicalDisks: 2<br>
physicalDisks:<br>
- diskName: Disk-1<br>
- diskName: Disk-2<br>
controller: primary-controller<br>
- sizeGibiBytes: 2000<br>
level: 0<br>
name: volume-2<br>
rotational: false<br>
numberOfPhysicalDisks: 2<br>
physicalDisks:<br>
- diskName: Disk-3<br>
- diskName: Disk-4<br>
controller: secondary-controller-2<br>
<br>
However, it remains unclear how the user would have the in-depth knowledge of the RAID controllers available to them, along with the vendor-specific disk names.
<br>
Example disk names (Dell): <br>
['Disk.Bay.0:Enclosure.Internal.0-1:RAID.Intergrated.1-1', 'Disk.Bay.2:Enclosure.Internal.0-1:RAID.Integrated.1-1', 'Disk.Bay.3:Enclosure.Internal.0-1:RAID.Integrated.1-1',...]<br>
Example disk names (HP):<br>
['2I:1:1', '2I:1:2', '2I:1:3',...]<br>
<br>
Example RAID Controllers (Dell):<br>
[‘Boss-F1 RAID Controller’, 'Dell PowerEdge RAID Controller H740P', ' Dell PowerEdge RAID Controller H830',...]<br>
Example RAID controller (HP):<br>
['HP Smart Array P400 Controller', 'HPE Smart Array E208i-a Controller', 'HPE Smart Array P816i-a Controller',...]<br>
<br>
One possible solution is to let the user specify the required RAID level, number of disks and storage capacity needed. Metal3 could then validate this request using the Ironic drivers (e.g. Redfish driver, iDRAC driver etc.) in order to ensure the required
hardware is available in the cluster. If it is available, the RAID configuration can be applied. Ifnot, the user can be informed about the error.<br>
<br>
Another solution is that Airship can perform inventory of the storage subsystems available in the cluster, and so the user can construct a RAID configuration accordingly. Conversely, the logic to select the physical disks and associated controllers may be implemented
in Airship, whilst the user just specifies the high level requirements.<br>
<br>
One issue is that Redfish requires the explicit RAID controller and disk names in order to do the RAID configuration, but it would be more convenient if the user did not have to know the low level details of their system.<br>
Here is the Redfish Action URI: {Base URI of target resource}/Actions/Volume.ChangeRAIDLayout<br>
(Provided in 6.109.4.2 of Redfish Resource and Schema Guide (<a href="https://urldefense.com/v3/__http:/dmtf.org__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYys3w9hc0$" target="_blank">dmtf.org</a>))<br>
<br>
The disk names can be generated using the Redfish logic constructing a human friendly identifier or label for hardware devices.
<a href="https://urldefense.com/v3/__https:/github.com/DMTF/Redfish-Tacklebox/blob/e429f70a79cfe288756618498ce485ab4be37131/redfish_utilities/inventory.py*L192__;Iw!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyuJLYIII$" target="_blank">
https://github.com/DMTF/Redfish-Tacklebox/blob/e429f70a79cfe288756618498ce485ab4be37131/redfish_utilities/inventory.py#L192</a><br>
<br>
Disk selection logic needs to be implemented in the most appropriate place. Airship may require that the user be unaware of the storage subsystem details when requesting the RAID config. Ironic requires a vendor independent implementation for this purpose.<br>
<br>
Here is relevant work done in metal3 to support RAID configuration in baremetal servers, using the Fujitsu iRMC driver:<br>
<a href="https://urldefense.com/v3/__https:/github.com/metal3-io/baremetal-operator/issues/206__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyAS9JYzU$" target="_blank">https://github.com/metal3-io/baremetal-operator/issues/206</a><br>
<a href="https://urldefense.com/v3/__https:/github.com/metal3-io/baremetal-operator/pull/292__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYym4_G3ew$" target="_blank">https://github.com/metal3-io/baremetal-operator/pull/292</a><br clear="all">
<u></u><u></u></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Warm Regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal">Zainub Wahid<u></u><u></u></p>
<div>
<p class="MsoNormal">Network Design Engineer II,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">xFlow Research Inc.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">+923044506900</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="mailto:zainub.wahid@xflowresearch.com" target="_blank"><span style="font-size:10pt;font-family:Arial,sans-serif">zainub.wahid@xflowresearch.com</span></a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="https://urldefense.com/v3/__http:/www.xFlowResearch.com__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyG48Ww0I$" target="_blank">www.xFlowResearch.com</a><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div>