Hello everyone, As discussed in today's meeting, this is related to the demo represented by Noor, last year, related to performing RAID configurations in Airship via Metal3 (and Ironic). Add RAID to example hardwareprofile (I50c82ce5) · Gerrit Code Review ( opendev.org) Through Airship, the user should be able to perform RAID configurations on their hardware. The user can specify the RAID controller names and their associated disks, along with the required RAID level, the desired number of disks and the amount of storage required. Here is the proposal for specifying the RAID configuration for a bare metal host via Metal3: https://github.com/metal3-io/metal3-docs/pull/148/files In terms of the CRD, the current implementation of the RAID has the following sections in it, for hardware RAID: spec: raid: hardwareRAIDVolumes: - sizeGibiBytes: 1000 level: 1 name: volume-name rotational: true numberOfPhysicalDisks: 2 We propose to add fields into the CRD, in the following manner: spec: raid: hardwareRAIDVolumes: - sizeGibiBytes: 1000 level: 1 name: volume-1 rotational: true numberOfPhysicalDisks: 2 physicalDisks: - diskName: Disk-1 - diskName: Disk-2 controller: primary-controller - sizeGibiBytes: 2000 level: 0 name: volume-2 rotational: false numberOfPhysicalDisks: 2 physicalDisks: - diskName: Disk-3 - diskName: Disk-4 controller: secondary-controller-2 However, it remains unclear how the user would have the in-depth knowledge of the RAID controllers available to them, along with the vendor-specific disk names. Example disk names (Dell): ['Disk.Bay.0:Enclosure.Internal.0-1:RAID.Intergrated.1-1', 'Disk.Bay.2:Enclosure.Internal.0-1:RAID.Integrated.1-1', 'Disk.Bay.3:Enclosure.Internal.0-1:RAID.Integrated.1-1',...] Example disk names (HP): ['2I:1:1', '2I:1:2', '2I:1:3',...] Example RAID Controllers (Dell): [‘Boss-F1 RAID Controller’, 'Dell PowerEdge RAID Controller H740P', ' Dell PowerEdge RAID Controller H830',...] Example RAID controller (HP): ['HP Smart Array P400 Controller', 'HPE Smart Array E208i-a Controller', 'HPE Smart Array P816i-a Controller',...] One possible solution is to let the user specify the required RAID level, number of disks and storage capacity needed. Metal3 could then validate this request using the Ironic drivers (e.g. Redfish driver, iDRAC driver etc.) in order to ensure the required hardware is available in the cluster. If it is available, the RAID configuration can be applied. Ifnot, the user can be informed about the error. Another solution is that Airship can perform inventory of the storage subsystems available in the cluster, and so the user can construct a RAID configuration accordingly. Conversely, the logic to select the physical disks and associated controllers may be implemented in Airship, whilst the user just specifies the high level requirements. One issue is that Redfish requires the explicit RAID controller and disk names in order to do the RAID configuration, but it would be more convenient if the user did not have to know the low level details of their system. Here is the Redfish Action URI: {Base URI of target resource}/Actions/Volume.ChangeRAIDLayout (Provided in 6.109.4.2 of Redfish Resource and Schema Guide (dmtf.org)) The disk names can be generated using the Redfish logic constructing a human friendly identifier or label for hardware devices. https://github.com/DMTF/Redfish-Tacklebox/blob/e429f70a79cfe288756618498ce48... Disk selection logic needs to be implemented in the most appropriate place. Airship may require that the user be unaware of the storage subsystem details when requesting the RAID config. Ironic requires a vendor independent implementation for this purpose. Here is relevant work done in metal3 to support RAID configuration in baremetal servers, using the Fujitsu iRMC driver: https://github.com/metal3-io/baremetal-operator/issues/206 https://github.com/metal3-io/baremetal-operator/pull/292 Warm Regards, Zainub Wahid Network Design Engineer II, xFlow Research Inc. +923044506900 zainub.wahid@xflowresearch.com www.xFlowResearch.com
Thanks for sharing & summarizing this, Zainub. Phil, Craig, Greg, Darren, (& anyone else with experience with RAID config today) – here are my thoughts; please refine/agree/disagree with them so we can give the Metal3 community good feedback on the interface we need for declarative RAID config. First: long term, we want to be able to declaratively configure all the BIOS settings that need to be configured. Steps like this RAID config are a valuable step in that direction. Second: I think there are use cases for both of these setups, and I’d advocate for M3 facilitating both of them in the long run*: 1. Strongly declarative: I want a RAID-N array on ControllerX with disks [DiskY, DiskZ] 2. Declarative with hints: I want a RAID-N array with two SSDs (and I don’t care what disk or what controller – you figure it out, M3!) * Rook supports something similar – “use all disks:yes/no”, or alternately “use these specific disks:…” Third: I think that #1 is a good fit for our most common use cases, where we put a lot of work into a reference definition, and then stamp it out many times on identical hardware. This is also the simplest implementation and would probably get something usable into Airship’s hands fastest. Does that sound right to you all? The other thing to consider is thresholding when disks are bad. I.e. a config item that says something like, “tolerate disk failures as long as I get Xgb” or “as long as I get 3 disks” or something. Do you think this is critical enough to be needed on Day 1? Thanks! Matt From: Zainub Wahid <zainub.wahid@xflowresearch.com> Sent: Tuesday, May 25, 2021 11:50 AM To: airship-discuss@lists.airshipit.org Cc: Williams, J T <J.T.Williams@dell.com>; Airship <airship@xflowresearch.com>; richard_pioso@dell.com Subject: [Airship-discuss] RAID Config Implementation Hello everyone, As discussed in today's meeting, this is related to the demo represented by Noor, last year, related to performing RAID configurations in Airship via Metal3 (and Ironic). Add RAID to example hardwareprofile (I50c82ce5) · Gerrit Code Review (opendev.org<https://urldefense.com/v3/__http:/opendev.org__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyvEP2e5U$>) Through Airship, the user should be able to perform RAID configurations on their hardware. The user can specify the RAID controller names and their associated disks, along with the required RAID level, the desired number of disks and the amount of storage required. Here is the proposal for specifying the RAID configuration for a bare metal host via Metal3: https://github.com/metal3-io/metal3-docs/pull/148/files<https://urldefense.com/v3/__https:/github.com/metal3-io/metal3-docs/pull/148/files__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyGgo40Mo$> In terms of the CRD, the current implementation of the RAID has the following sections in it, for hardware RAID: spec: raid: hardwareRAIDVolumes: - sizeGibiBytes: 1000 level: 1 name: volume-name rotational: true numberOfPhysicalDisks: 2 We propose to add fields into the CRD, in the following manner: spec: raid: hardwareRAIDVolumes: - sizeGibiBytes: 1000 level: 1 name: volume-1 rotational: true numberOfPhysicalDisks: 2 physicalDisks: - diskName: Disk-1 - diskName: Disk-2 controller: primary-controller - sizeGibiBytes: 2000 level: 0 name: volume-2 rotational: false numberOfPhysicalDisks: 2 physicalDisks: - diskName: Disk-3 - diskName: Disk-4 controller: secondary-controller-2 However, it remains unclear how the user would have the in-depth knowledge of the RAID controllers available to them, along with the vendor-specific disk names. Example disk names (Dell): ['Disk.Bay.0:Enclosure.Internal.0-1:RAID.Intergrated.1-1', 'Disk.Bay.2:Enclosure.Internal.0-1:RAID.Integrated.1-1', 'Disk.Bay.3:Enclosure.Internal.0-1:RAID.Integrated.1-1',...] Example disk names (HP): ['2I:1:1', '2I:1:2', '2I:1:3',...] Example RAID Controllers (Dell): [‘Boss-F1 RAID Controller’, 'Dell PowerEdge RAID Controller H740P', ' Dell PowerEdge RAID Controller H830',...] Example RAID controller (HP): ['HP Smart Array P400 Controller', 'HPE Smart Array E208i-a Controller', 'HPE Smart Array P816i-a Controller',...] One possible solution is to let the user specify the required RAID level, number of disks and storage capacity needed. Metal3 could then validate this request using the Ironic drivers (e.g. Redfish driver, iDRAC driver etc.) in order to ensure the required hardware is available in the cluster. If it is available, the RAID configuration can be applied. Ifnot, the user can be informed about the error. Another solution is that Airship can perform inventory of the storage subsystems available in the cluster, and so the user can construct a RAID configuration accordingly. Conversely, the logic to select the physical disks and associated controllers may be implemented in Airship, whilst the user just specifies the high level requirements. One issue is that Redfish requires the explicit RAID controller and disk names in order to do the RAID configuration, but it would be more convenient if the user did not have to know the low level details of their system. Here is the Redfish Action URI: {Base URI of target resource}/Actions/Volume.ChangeRAIDLayout (Provided in 6.109.4.2 of Redfish Resource and Schema Guide (dmtf.org<https://urldefense.com/v3/__http:/dmtf.org__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYys3w9hc0$>)) The disk names can be generated using the Redfish logic constructing a human friendly identifier or label for hardware devices. https://github.com/DMTF/Redfish-Tacklebox/blob/e429f70a79cfe288756618498ce485ab4be37131/redfish_utilities/inventory.py#L192<https://urldefense.com/v3/__https:/github.com/DMTF/Redfish-Tacklebox/blob/e429f70a79cfe288756618498ce485ab4be37131/redfish_utilities/inventory.py*L192__;Iw!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyuJLYIII$> Disk selection logic needs to be implemented in the most appropriate place. Airship may require that the user be unaware of the storage subsystem details when requesting the RAID config. Ironic requires a vendor independent implementation for this purpose. Here is relevant work done in metal3 to support RAID configuration in baremetal servers, using the Fujitsu iRMC driver: https://github.com/metal3-io/baremetal-operator/issues/206<https://urldefense.com/v3/__https:/github.com/metal3-io/baremetal-operator/issues/206__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyAS9JYzU$> https://github.com/metal3-io/baremetal-operator/pull/292<https://urldefense.com/v3/__https:/github.com/metal3-io/baremetal-operator/pull/292__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYym4_G3ew$> Warm Regards, Zainub Wahid Network Design Engineer II, xFlow Research Inc. +923044506900 zainub.wahid@xflowresearch.com<mailto:zainub.wahid@xflowresearch.com> www.xFlowResearch.com<https://urldefense.com/v3/__http:/www.xFlowResearch.com__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyG48Ww0I$>
Thank you for your email, Matt. About the Airship use cases: 1. The strongly declarative one would be easy to support via Metal3, to the best of my knowledge. This would be perfect as long as the Airship user is okay with finding out the disk and RAID controller names through other means (using iDRAC, for example). It would also be an easier and valuable implementation. 2. Metal3 will perform the disk selection itself (or delegate to Ironic). Providing some level of guarantee for the disk failures would contribute to a reliable user experience. For the declarative case, we can perhaps ensure that the specified disks are functional. For the specification with hints, I suggest making the disk capacity a metric that must always be satisfied (e.g. x GB always provided). Input on these requirements is much appreciated, so we can carry the discussion forward with the Metal3 community and get to implementation. Thank you so much! Regards,
Refreshing this in post-holiday inboxes: Phil, Craig, Greg, Darren, (& anyone else with experience with RAID config today) – please take a look at my thoughts below and let me know if that lines up with what you would find most useful as well. Thanks! Also, Merging Mahnoor’s comments with the email thread: (quote) Thank you for your email, Matt. About the Airship use cases: 1. The strongly declarative one would be easy to support via Metal3, to the best of my knowledge. This would be perfect as long as the Airship user is okay with finding out the disk and RAID controller names through other means (using iDRAC, for example). It would also be an easier and valuable implementation. 2. Metal3 will perform the disk selection itself (or delegate to Ironic). Providing some level of guarantee for the disk failures would contribute to a reliable user experience. For the declarative case, we can perhaps ensure that the specified disks are functional. For the specification with hints, I suggest making the disk capacity a metric that must always be satisfied (e.g. x GB always provided). Input on these requirements is much appreciated, so we can carry the discussion forward with the Metal3 community and get to implementation. Thank you so much! (end quote) From: MCEUEN, MATT Sent: Wednesday, May 26, 2021 4:17 PM To: Zainub Wahid <zainub.wahid@xflowresearch.com>; airship-discuss@lists.airshipit.org; SPHICAS, PHIL G <ps3910@att.com>; OBERFIELD, GREG <go4243@att.com>; DEJAEGER, DARREN R <dd118r@att.com>; STEIN, JARED W <js5175@att.com> Cc: Williams, J T <J.T.Williams@dell.com>; Airship <airship@xflowresearch.com>; richard_pioso@dell.com Subject: RE: [Airship-discuss] RAID Config Implementation Thanks for sharing & summarizing this, Zainub. Phil, Craig, Greg, Darren, (& anyone else with experience with RAID config today) – here are my thoughts; please refine/agree/disagree with them so we can give the Metal3 community good feedback on the interface we need for declarative RAID config. First: long term, we want to be able to declaratively configure all the BIOS settings that need to be configured. Steps like this RAID config are a valuable step in that direction. Second: I think there are use cases for both of these setups, and I’d advocate for M3 facilitating both of them in the long run*: 1. Strongly declarative: I want a RAID-N array on ControllerX with disks [DiskY, DiskZ] 2. Declarative with hints: I want a RAID-N array with two SSDs (and I don’t care what disk or what controller – you figure it out, M3!) * Rook supports something similar – “use all disks:yes/no”, or alternately “use these specific disks:…” Third: I think that #1 is a good fit for our most common use cases, where we put a lot of work into a reference definition, and then stamp it out many times on identical hardware. This is also the simplest implementation and would probably get something usable into Airship’s hands fastest. Does that sound right to you all? The other thing to consider is thresholding when disks are bad. I.e. a config item that says something like, “tolerate disk failures as long as I get Xgb” or “as long as I get 3 disks” or something. Do you think this is critical enough to be needed on Day 1? Thanks! Matt From: Zainub Wahid <zainub.wahid@xflowresearch.com<mailto:zainub.wahid@xflowresearch.com>> Sent: Tuesday, May 25, 2021 11:50 AM To: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Cc: Williams, J T <J.T.Williams@dell.com<mailto:J.T.Williams@dell.com>>; Airship <airship@xflowresearch.com<mailto:airship@xflowresearch.com>>; richard_pioso@dell.com<mailto:richard_pioso@dell.com> Subject: [Airship-discuss] RAID Config Implementation Hello everyone, As discussed in today's meeting, this is related to the demo represented by Noor, last year, related to performing RAID configurations in Airship via Metal3 (and Ironic). Add RAID to example hardwareprofile (I50c82ce5) · Gerrit Code Review (opendev.org<https://urldefense.com/v3/__http:/opendev.org__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyvEP2e5U$>) Through Airship, the user should be able to perform RAID configurations on their hardware. The user can specify the RAID controller names and their associated disks, along with the required RAID level, the desired number of disks and the amount of storage required. Here is the proposal for specifying the RAID configuration for a bare metal host via Metal3: https://github.com/metal3-io/metal3-docs/pull/148/files<https://urldefense.com/v3/__https:/github.com/metal3-io/metal3-docs/pull/148/files__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyGgo40Mo$> In terms of the CRD, the current implementation of the RAID has the following sections in it, for hardware RAID: spec: raid: hardwareRAIDVolumes: - sizeGibiBytes: 1000 level: 1 name: volume-name rotational: true numberOfPhysicalDisks: 2 We propose to add fields into the CRD, in the following manner: spec: raid: hardwareRAIDVolumes: - sizeGibiBytes: 1000 level: 1 name: volume-1 rotational: true numberOfPhysicalDisks: 2 physicalDisks: - diskName: Disk-1 - diskName: Disk-2 controller: primary-controller - sizeGibiBytes: 2000 level: 0 name: volume-2 rotational: false numberOfPhysicalDisks: 2 physicalDisks: - diskName: Disk-3 - diskName: Disk-4 controller: secondary-controller-2 However, it remains unclear how the user would have the in-depth knowledge of the RAID controllers available to them, along with the vendor-specific disk names. Example disk names (Dell): ['Disk.Bay.0:Enclosure.Internal.0-1:RAID.Intergrated.1-1', 'Disk.Bay.2:Enclosure.Internal.0-1:RAID.Integrated.1-1', 'Disk.Bay.3:Enclosure.Internal.0-1:RAID.Integrated.1-1',...] Example disk names (HP): ['2I:1:1', '2I:1:2', '2I:1:3',...] Example RAID Controllers (Dell): [‘Boss-F1 RAID Controller’, 'Dell PowerEdge RAID Controller H740P', ' Dell PowerEdge RAID Controller H830',...] Example RAID controller (HP): ['HP Smart Array P400 Controller', 'HPE Smart Array E208i-a Controller', 'HPE Smart Array P816i-a Controller',...] One possible solution is to let the user specify the required RAID level, number of disks and storage capacity needed. Metal3 could then validate this request using the Ironic drivers (e.g. Redfish driver, iDRAC driver etc.) in order to ensure the required hardware is available in the cluster. If it is available, the RAID configuration can be applied. Ifnot, the user can be informed about the error. Another solution is that Airship can perform inventory of the storage subsystems available in the cluster, and so the user can construct a RAID configuration accordingly. Conversely, the logic to select the physical disks and associated controllers may be implemented in Airship, whilst the user just specifies the high level requirements. One issue is that Redfish requires the explicit RAID controller and disk names in order to do the RAID configuration, but it would be more convenient if the user did not have to know the low level details of their system. Here is the Redfish Action URI: {Base URI of target resource}/Actions/Volume.ChangeRAIDLayout (Provided in 6.109.4.2 of Redfish Resource and Schema Guide (dmtf.org<https://urldefense.com/v3/__http:/dmtf.org__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYys3w9hc0$>)) The disk names can be generated using the Redfish logic constructing a human friendly identifier or label for hardware devices. https://github.com/DMTF/Redfish-Tacklebox/blob/e429f70a79cfe288756618498ce485ab4be37131/redfish_utilities/inventory.py#L192<https://urldefense.com/v3/__https:/github.com/DMTF/Redfish-Tacklebox/blob/e429f70a79cfe288756618498ce485ab4be37131/redfish_utilities/inventory.py*L192__;Iw!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyuJLYIII$> Disk selection logic needs to be implemented in the most appropriate place. Airship may require that the user be unaware of the storage subsystem details when requesting the RAID config. Ironic requires a vendor independent implementation for this purpose. Here is relevant work done in metal3 to support RAID configuration in baremetal servers, using the Fujitsu iRMC driver: https://github.com/metal3-io/baremetal-operator/issues/206<https://urldefense.com/v3/__https:/github.com/metal3-io/baremetal-operator/issues/206__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyAS9JYzU$> https://github.com/metal3-io/baremetal-operator/pull/292<https://urldefense.com/v3/__https:/github.com/metal3-io/baremetal-operator/pull/292__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYym4_G3ew$> Warm Regards, Zainub Wahid Network Design Engineer II, xFlow Research Inc. +923044506900 zainub.wahid@xflowresearch.com<mailto:zainub.wahid@xflowresearch.com> www.xFlowResearch.com<https://urldefense.com/v3/__http:/www.xFlowResearch.com__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyG48Ww0I$>
Matt, I agree with your thoughts on making available both approaches. A couple things to consider: 1. Regarding disk failures: would these disks even be available to be selected for any sort of RAID if they’re bad? If not, we’d quickly hit an issue with a mismatched number of disks vs. what’s declared (at least today). 2. Can we allow other disk/raid-related config to be tuned? I’m thinking Stripe size, Read policy, Write policy? Just some food for thought. Regards, Darren From: "MCEUEN, MATT" <MM9745@att.com> Date: Tuesday, June 1, 2021 at 10:04 AM To: Zainub Wahid <zainub.wahid@xflowresearch.com>, "airship-discuss@lists.airshipit.org" <airship-discuss@lists.airshipit.org>, "SPHICAS, PHIL G" <ps3910@att.com>, "OBERFIELD, GREG" <go4243@att.com>, "DEJAEGER, DARREN R" <dd118r@att.com>, "STEIN, JARED W" <js5175@att.com> Cc: "Williams, J T" <J.T.Williams@dell.com>, Airship <airship@xflowresearch.com>, "richard_pioso@dell.com" <richard_pioso@dell.com> Subject: RE: [Airship-discuss] RAID Config Implementation Refreshing this in post-holiday inboxes: Phil, Craig, Greg, Darren, (& anyone else with experience with RAID config today) – please take a look at my thoughts below and let me know if that lines up with what you would find most useful as well. Thanks! Also, Merging Mahnoor’s comments with the email thread: (quote) Thank you for your email, Matt. About the Airship use cases: 1. The strongly declarative one would be easy to support via Metal3, to the best of my knowledge. This would be perfect as long as the Airship user is okay with finding out the disk and RAID controller names through other means (using iDRAC, for example). It would also be an easier and valuable implementation. 2. Metal3 will perform the disk selection itself (or delegate to Ironic). Providing some level of guarantee for the disk failures would contribute to a reliable user experience. For the declarative case, we can perhaps ensure that the specified disks are functional. For the specification with hints, I suggest making the disk capacity a metric that must always be satisfied (e.g. x GB always provided). Input on these requirements is much appreciated, so we can carry the discussion forward with the Metal3 community and get to implementation. Thank you so much! (end quote) From: MCEUEN, MATT Sent: Wednesday, May 26, 2021 4:17 PM To: Zainub Wahid <zainub.wahid@xflowresearch.com>; airship-discuss@lists.airshipit.org; SPHICAS, PHIL G <ps3910@att.com>; OBERFIELD, GREG <go4243@att.com>; DEJAEGER, DARREN R <dd118r@att.com>; STEIN, JARED W <js5175@att.com> Cc: Williams, J T <J.T.Williams@dell.com>; Airship <airship@xflowresearch.com>; richard_pioso@dell.com Subject: RE: [Airship-discuss] RAID Config Implementation Thanks for sharing & summarizing this, Zainub. Phil, Craig, Greg, Darren, (& anyone else with experience with RAID config today) – here are my thoughts; please refine/agree/disagree with them so we can give the Metal3 community good feedback on the interface we need for declarative RAID config. First: long term, we want to be able to declaratively configure all the BIOS settings that need to be configured. Steps like this RAID config are a valuable step in that direction. Second: I think there are use cases for both of these setups, and I’d advocate for M3 facilitating both of them in the long run*: 1. Strongly declarative: I want a RAID-N array on ControllerX with disks [DiskY, DiskZ] 2. Declarative with hints: I want a RAID-N array with two SSDs (and I don’t care what disk or what controller – you figure it out, M3!) * Rook supports something similar – “use all disks:yes/no”, or alternately “use these specific disks:…” Third: I think that #1 is a good fit for our most common use cases, where we put a lot of work into a reference definition, and then stamp it out many times on identical hardware. This is also the simplest implementation and would probably get something usable into Airship’s hands fastest. Does that sound right to you all? The other thing to consider is thresholding when disks are bad. I.e. a config item that says something like, “tolerate disk failures as long as I get Xgb” or “as long as I get 3 disks” or something. Do you think this is critical enough to be needed on Day 1? Thanks! Matt From: Zainub Wahid <zainub.wahid@xflowresearch.com<mailto:zainub.wahid@xflowresearch.com>> Sent: Tuesday, May 25, 2021 11:50 AM To: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Cc: Williams, J T <J.T.Williams@dell.com<mailto:J.T.Williams@dell.com>>; Airship <airship@xflowresearch.com<mailto:airship@xflowresearch.com>>; richard_pioso@dell.com<mailto:richard_pioso@dell.com> Subject: [Airship-discuss] RAID Config Implementation Hello everyone, As discussed in today's meeting, this is related to the demo represented by Noor, last year, related to performing RAID configurations in Airship via Metal3 (and Ironic). Add RAID to example hardwareprofile (I50c82ce5) · Gerrit Code Review (opendev.org<https://urldefense.com/v3/__http:/opendev.org__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyvEP2e5U$>) Through Airship, the user should be able to perform RAID configurations on their hardware. The user can specify the RAID controller names and their associated disks, along with the required RAID level, the desired number of disks and the amount of storage required. Here is the proposal for specifying the RAID configuration for a bare metal host via Metal3: https://github.com/metal3-io/metal3-docs/pull/148/files<https://urldefense.com/v3/__https:/github.com/metal3-io/metal3-docs/pull/148/files__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyGgo40Mo$> In terms of the CRD, the current implementation of the RAID has the following sections in it, for hardware RAID: spec: raid: hardwareRAIDVolumes: - sizeGibiBytes: 1000 level: 1 name: volume-name rotational: true numberOfPhysicalDisks: 2 We propose to add fields into the CRD, in the following manner: spec: raid: hardwareRAIDVolumes: - sizeGibiBytes: 1000 level: 1 name: volume-1 rotational: true numberOfPhysicalDisks: 2 physicalDisks: - diskName: Disk-1 - diskName: Disk-2 controller: primary-controller - sizeGibiBytes: 2000 level: 0 name: volume-2 rotational: false numberOfPhysicalDisks: 2 physicalDisks: - diskName: Disk-3 - diskName: Disk-4 controller: secondary-controller-2 However, it remains unclear how the user would have the in-depth knowledge of the RAID controllers available to them, along with the vendor-specific disk names. Example disk names (Dell): ['Disk.Bay.0:Enclosure.Internal.0-1:RAID.Intergrated.1-1', 'Disk.Bay.2:Enclosure.Internal.0-1:RAID.Integrated.1-1', 'Disk.Bay.3:Enclosure.Internal.0-1:RAID.Integrated.1-1',...] Example disk names (HP): ['2I:1:1', '2I:1:2', '2I:1:3',...] Example RAID Controllers (Dell): [‘Boss-F1 RAID Controller’, 'Dell PowerEdge RAID Controller H740P', ' Dell PowerEdge RAID Controller H830',...] Example RAID controller (HP): ['HP Smart Array P400 Controller', 'HPE Smart Array E208i-a Controller', 'HPE Smart Array P816i-a Controller',...] One possible solution is to let the user specify the required RAID level, number of disks and storage capacity needed. Metal3 could then validate this request using the Ironic drivers (e.g. Redfish driver, iDRAC driver etc.) in order to ensure the required hardware is available in the cluster. If it is available, the RAID configuration can be applied. Ifnot, the user can be informed about the error. Another solution is that Airship can perform inventory of the storage subsystems available in the cluster, and so the user can construct a RAID configuration accordingly. Conversely, the logic to select the physical disks and associated controllers may be implemented in Airship, whilst the user just specifies the high level requirements. One issue is that Redfish requires the explicit RAID controller and disk names in order to do the RAID configuration, but it would be more convenient if the user did not have to know the low level details of their system. Here is the Redfish Action URI: {Base URI of target resource}/Actions/Volume.ChangeRAIDLayout (Provided in 6.109.4.2 of Redfish Resource and Schema Guide (dmtf.org<https://urldefense.com/v3/__http:/dmtf.org__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYys3w9hc0$>)) The disk names can be generated using the Redfish logic constructing a human friendly identifier or label for hardware devices. https://github.com/DMTF/Redfish-Tacklebox/blob/e429f70a79cfe288756618498ce485ab4be37131/redfish_utilities/inventory.py#L192<https://urldefense.com/v3/__https:/github.com/DMTF/Redfish-Tacklebox/blob/e429f70a79cfe288756618498ce485ab4be37131/redfish_utilities/inventory.py*L192__;Iw!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyuJLYIII$> Disk selection logic needs to be implemented in the most appropriate place. Airship may require that the user be unaware of the storage subsystem details when requesting the RAID config. Ironic requires a vendor independent implementation for this purpose. Here is relevant work done in metal3 to support RAID configuration in baremetal servers, using the Fujitsu iRMC driver: https://github.com/metal3-io/baremetal-operator/issues/206<https://urldefense.com/v3/__https:/github.com/metal3-io/baremetal-operator/issues/206__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyAS9JYzU$> https://github.com/metal3-io/baremetal-operator/pull/292<https://urldefense.com/v3/__https:/github.com/metal3-io/baremetal-operator/pull/292__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYym4_G3ew$> Warm Regards, Zainub Wahid Network Design Engineer II, xFlow Research Inc. +923044506900 zainub.wahid@xflowresearch.com<mailto:zainub.wahid@xflowresearch.com> www.xFlowResearch.com<https://urldefense.com/v3/__http:/www.xFlowResearch.com__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyG48Ww0I$>
Darren, Thank you so much for your time and reply. These are very helpful and important points you mentioned in your email. For [1], we are already taking care of this. Exceptions will be thrown if there is any problem with Disks. For [2], it seems a very interesting point but according to my knowledge, these are OS level configurations. We will look into it in future but right now, it is beyond our scope. Thank you for showing interest, once again. Warm Regards, Zainub Wahid Network Design Engineer II, xFlow Research Inc. +923044506900 zainub.wahid@xflowresearch.com www.xFlowResearch.com On Tue, Jun 1, 2021 at 7:20 PM DEJAEGER, DARREN R <dd118r@att.com> wrote:
Matt,
I agree with your thoughts on making available both approaches. A couple things to consider:
1. Regarding disk failures: would these disks even be available to be selected for any sort of RAID if they’re bad? If not, we’d quickly hit an issue with a mismatched number of disks vs. what’s declared (at least today). 2. Can we allow other disk/raid-related config to be tuned? I’m thinking Stripe size, Read policy, Write policy?
Just some food for thought.
Regards,
Darren
*From: *"MCEUEN, MATT" <MM9745@att.com> *Date: *Tuesday, June 1, 2021 at 10:04 AM *To: *Zainub Wahid <zainub.wahid@xflowresearch.com>, " airship-discuss@lists.airshipit.org" <airship-discuss@lists.airshipit.org>, "SPHICAS, PHIL G" <ps3910@att.com>, "OBERFIELD, GREG" <go4243@att.com>, "DEJAEGER, DARREN R" <dd118r@att.com>, "STEIN, JARED W" <js5175@att.com> *Cc: *"Williams, J T" <J.T.Williams@dell.com>, Airship < airship@xflowresearch.com>, "richard_pioso@dell.com" < richard_pioso@dell.com> *Subject: *RE: [Airship-discuss] RAID Config Implementation
Refreshing this in post-holiday inboxes: Phil, Craig, Greg, Darren, (& anyone else with experience with RAID config today) – please take a look at my thoughts below and let me know if that lines up with what you would find most useful as well. Thanks!
Also, Merging Mahnoor’s comments with the email thread:
(quote)
Thank you for your email, Matt.
About the Airship use cases:
1. The strongly declarative one would be easy to support via Metal3, to the best of my knowledge. This would be perfect as long as the Airship user is okay with finding out the disk and RAID controller names through other means (using iDRAC, for example). It would also be an easier and valuable implementation.
2. Metal3 will perform the disk selection itself (or delegate to Ironic).
Providing some level of guarantee for the disk failures would contribute to a reliable user experience. For the declarative case, we can perhaps ensure that the specified disks are functional. For the specification with hints, I suggest making the disk capacity a metric that must always be satisfied (e.g. x GB always provided).
Input on these requirements is much appreciated, so we can carry the discussion forward with the Metal3 community and get to implementation.
Thank you so much!
(end quote)
*From:* MCEUEN, MATT *Sent:* Wednesday, May 26, 2021 4:17 PM *To:* Zainub Wahid <zainub.wahid@xflowresearch.com>; airship-discuss@lists.airshipit.org; SPHICAS, PHIL G <ps3910@att.com>; OBERFIELD, GREG <go4243@att.com>; DEJAEGER, DARREN R <dd118r@att.com>; STEIN, JARED W <js5175@att.com> *Cc:* Williams, J T <J.T.Williams@dell.com>; Airship < airship@xflowresearch.com>; richard_pioso@dell.com *Subject:* RE: [Airship-discuss] RAID Config Implementation
Thanks for sharing & summarizing this, Zainub.
Phil, Craig, Greg, Darren, (& anyone else with experience with RAID config today) – here are my thoughts; please refine/agree/disagree with them so we can give the Metal3 community good feedback on the interface we need for declarative RAID config.
First: long term, we want to be able to declaratively configure all the BIOS settings that need to be configured. Steps like this RAID config are a valuable step in that direction.
Second: I think there are use cases for both of these setups, and I’d advocate for M3 facilitating both of them in the long run*:
1. Strongly declarative: I want a RAID-N array on ControllerX with disks [DiskY, DiskZ] 2. Declarative with hints: I want a RAID-N array with two SSDs (and I don’t care what disk or what controller – you figure it out, M3!)
* Rook supports something similar – “use all disks:yes/no”, or alternately “use these specific disks:…”
Third: I think that #1 is a good fit for our most common use cases, where we put a lot of work into a reference definition, and then stamp it out many times on *identical* hardware. This is also the simplest implementation and would probably get something usable into Airship’s hands fastest.
Does that sound right to you all?
The other thing to consider is thresholding when disks are bad. I.e. a config item that says something like, “tolerate disk failures as long as I get Xgb” or “as long as I get 3 disks” or something. Do you think this is critical enough to be needed on Day 1?
Thanks!
Matt
*From:* Zainub Wahid <zainub.wahid@xflowresearch.com> *Sent:* Tuesday, May 25, 2021 11:50 AM *To:* airship-discuss@lists.airshipit.org *Cc:* Williams, J T <J.T.Williams@dell.com>; Airship < airship@xflowresearch.com>; richard_pioso@dell.com *Subject:* [Airship-discuss] RAID Config Implementation
Hello everyone,
As discussed in today's meeting, this is related to the demo represented by Noor, last year, related to performing RAID configurations in Airship via Metal3 (and Ironic). Add RAID to example hardwareprofile (I50c82ce5) · Gerrit Code Review ( opendev.org <https://urldefense.com/v3/__http:/opendev.org__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyvEP2e5U$> )
Through Airship, the user should be able to perform RAID configurations on their hardware. The user can specify the RAID controller names and their associated disks, along with the required RAID level, the desired number of disks and the amount of storage required. Here is the proposal for specifying the RAID configuration for a bare metal host via Metal3: https://github.com/metal3-io/metal3-docs/pull/148/files <https://urldefense.com/v3/__https:/github.com/metal3-io/metal3-docs/pull/148/files__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyGgo40Mo$>
In terms of the CRD, the current implementation of the RAID has the following sections in it, for hardware RAID: spec: raid: hardwareRAIDVolumes: - sizeGibiBytes: 1000 level: 1 name: volume-name rotational: true numberOfPhysicalDisks: 2 We propose to add fields into the CRD, in the following manner: spec: raid: hardwareRAIDVolumes: - sizeGibiBytes: 1000 level: 1 name: volume-1 rotational: true numberOfPhysicalDisks: 2 physicalDisks: - diskName: Disk-1 - diskName: Disk-2 controller: primary-controller - sizeGibiBytes: 2000 level: 0 name: volume-2 rotational: false numberOfPhysicalDisks: 2 physicalDisks: - diskName: Disk-3 - diskName: Disk-4 controller: secondary-controller-2
However, it remains unclear how the user would have the in-depth knowledge of the RAID controllers available to them, along with the vendor-specific disk names. Example disk names (Dell): ['Disk.Bay.0:Enclosure.Internal.0-1:RAID.Intergrated.1-1', 'Disk.Bay.2:Enclosure.Internal.0-1:RAID.Integrated.1-1', 'Disk.Bay.3:Enclosure.Internal.0-1:RAID.Integrated.1-1',...] Example disk names (HP): ['2I:1:1', '2I:1:2', '2I:1:3',...]
Example RAID Controllers (Dell): [‘Boss-F1 RAID Controller’, 'Dell PowerEdge RAID Controller H740P', ' Dell PowerEdge RAID Controller H830',...] Example RAID controller (HP): ['HP Smart Array P400 Controller', 'HPE Smart Array E208i-a Controller', 'HPE Smart Array P816i-a Controller',...]
One possible solution is to let the user specify the required RAID level, number of disks and storage capacity needed. Metal3 could then validate this request using the Ironic drivers (e.g. Redfish driver, iDRAC driver etc.) in order to ensure the required hardware is available in the cluster. If it is available, the RAID configuration can be applied. Ifnot, the user can be informed about the error.
Another solution is that Airship can perform inventory of the storage subsystems available in the cluster, and so the user can construct a RAID configuration accordingly. Conversely, the logic to select the physical disks and associated controllers may be implemented in Airship, whilst the user just specifies the high level requirements.
One issue is that Redfish requires the explicit RAID controller and disk names in order to do the RAID configuration, but it would be more convenient if the user did not have to know the low level details of their system. Here is the Redfish Action URI: {Base URI of target resource}/Actions/Volume.ChangeRAIDLayout (Provided in 6.109.4.2 of Redfish Resource and Schema Guide (dmtf.org <https://urldefense.com/v3/__http:/dmtf.org__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYys3w9hc0$> ))
The disk names can be generated using the Redfish logic constructing a human friendly identifier or label for hardware devices. https://github.com/DMTF/Redfish-Tacklebox/blob/e429f70a79cfe288756618498ce48... <https://urldefense.com/v3/__https:/github.com/DMTF/Redfish-Tacklebox/blob/e429f70a79cfe288756618498ce485ab4be37131/redfish_utilities/inventory.py*L192__;Iw!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyuJLYIII$>
Disk selection logic needs to be implemented in the most appropriate place. Airship may require that the user be unaware of the storage subsystem details when requesting the RAID config. Ironic requires a vendor independent implementation for this purpose.
Here is relevant work done in metal3 to support RAID configuration in baremetal servers, using the Fujitsu iRMC driver: https://github.com/metal3-io/baremetal-operator/issues/206 <https://urldefense.com/v3/__https:/github.com/metal3-io/baremetal-operator/issues/206__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyAS9JYzU$> https://github.com/metal3-io/baremetal-operator/pull/292 <https://urldefense.com/v3/__https:/github.com/metal3-io/baremetal-operator/pull/292__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYym4_G3ew$>
Warm Regards,
Zainub Wahid
Network Design Engineer II,
xFlow Research Inc.
+923044506900
zainub.wahid@xflowresearch.com
www.xFlowResearch.com <https://urldefense.com/v3/__http:/www.xFlowResearch.com__;!!BhdT!0FCxfaRXqTKY8FTua2QbHU5D5562OD_9g_BLbFq-DtlLGrsxsjRowGYyG48Ww0I$>
participants (4)
-
DEJAEGER, DARREN R
-
Mahnoor Asghar
-
MCEUEN, MATT
-
Zainub Wahid