[Airship-discuss] Fwd: Activating Prometheus (& other monitoring tools) for aiab

[Loic Le Gal]

Thanks again Kaspars,

Adding all these charts seem too huge even for a 32 GB RAM VM (I'll try
again, in case the failure came from a latency of our infra).
Anyway, I succeed to add only Prometheus, AlertManager and Grafana by doing
this:
A - add these chartgroups in aiab bootstrap manifest
    - osh-infra-monitoring
    - osh-infra-ingress-controller
    - osh-infra-mariadb
    - osh-infra-dashboards

B - Overload the osh-infra-dashboard chartgroup to include only grafana
(using deckhand overloading mechanism instead of brute removal )
  B1. add a label at the global layer to avoid Deckhand complains in B2
+++ b/global/software/charts/osh-infra/osh-infra-dashboards/chart-group.yaml
@@ -6,6 +6,8 @@ metadata:
   layeringDefinition:
     abstract: false
     layer: global
+  labels:
+    name: osh-infra-dashboards-global
   storagePolicy: cleartext

  B2. create a new file at the type layer that only contains grafana:
+++ b/type/sloop/charts/osh-infra/dashboards-chartgroup.yaml
@@ -0,0 +1,20 @@
+---
+schema: armada/ChartGroup/v1
+metadata:
+  schema: metadata/Document/v1
+  replacement: true
+  name: osh-infra-dashboards
+  layeringDefinition:
+    abstract: false
+    layer: type
+    parentSelector:
+      name: osh-infra-dashboards-global
+    actions:
+      - method: merge
+        path: .
+  storagePolicy: cleartext
+data:
+  # disable kibana by changing the chartgroup content
+  chart_group:
+    - grafana
+...
C - Expose the services:

kubectl expose pod/prometheus-0 -n osh-infra --type=NodePort
--name=prometheus-dashboard
kubectl expose pod/alertmanager-0 -n osh-infra --type=NodePort
--name=alertmgr-dashboard
kubectl expose service/grafana-dashboard -n osh-infra --type=NodePort
--name=grafana-web

Not every Grafana Dashboards are operational in aiab but those that are are
really good !

BR, Loïc

Le mer. 7 août 2019 à 17:27, SKELS, KASPARS <ks3019 at att.com> a écrit :

> Hi,
>
> these charts are used in Airsloop/Seaworthy runs, so they should be
> tested, and especially since site/aiab uses sloop type, the resources are
> scaled down (less replicas, less storage requests), see
> https://github.com/airshipit/treasuremap/tree/master/type/sloop/charts/osh-infra.
> You should be able to deploy entire LMA (logging, monitoring, alerting)
> stack by adding this to full-site manifest
>
>     - osh-infra-ingress-controller
>     - osh-infra-ceph-config
>     - osh-infra-radosgw
>     - osh-infra-logging
>     - osh-infra-monitoring
>     - osh-infra-mariadb
>     - osh-infra-dashboards
>
> This will add things like promethius, elasticsearch, kibana, etc.
>
> Part of the reasoning for not including these charts in the default AIAB
> site was to be able to limit resources to 20G of memory (so people can get
> it up and running in VMs on their laptops, and other resources constrained
> places).
>
> I wonder if you could investigate a way we could run AIAB (airship in a
> bottle) with an option (say something like --with-lma) or similar.
>
> We are having community IRC meetings every Tuesday 9AM CST and it would be
> a good topic to discuss.
>
> Kindly, Kaspars
>
>
>
>
> ------------------------------
> *From:* Loic Le Gal [l.legal.astellia at gmail.com]
> *Sent:* Wednesday, August 07, 2019 10:06 AM
> *To:* airship-discuss at lists.airshipit.org
> *Subject:* Re: [Airship-discuss] Activating Prometheus (& other
> monitoring tools) for aiab
>
> Hi,
>
> Replying myself to close this thread.
> I finally succeed in creating prometheus-0 and alerter-0 pods by adding
> also -osh-infra-monitoring in the chart_groups: section of the
> file site/aiab/manifests/bootstrap.yaml.
> It seems to list the available charts, the deployment process looks in to
> match labels set on nodes.
>
> Le mer. 7 août 2019 à 14:39, Loic Le Gal <l.legal.astellia at gmail.com> a
> écrit :
>
>> Hi airshipers,
>>
>> I'd like to add Prometheus monitoring (+fluentbit, ...) to the new
>> treasuremap/site/aiab;
>> It seems that some charts already exist but I'm not sure the integration
>> is finished because nothing usefull found in seaworthy site def.
>>
>> Server/client labels do not exist on the Genesis node so I've added them
>> to site/aiab/profiles/genesis.yaml and launch again the whole genesis
>> process from scratch but no new pods appear. (I also tried adding
>> prometheus-global=true and prometheus-type=true labels)
>>
>> I've already searched in Armada read-the-docs.
>> Even if the WIP in armada regarding automation of metrics collection is
>> still a WIP [1), I just want to get prometheus pods (then add grafana,
>> fluentbit, ...)
>>
>> Do you know where I can find information to add/activate the monitoring
>> layer (osh-infra-monitoring) or just add a Prometheus pod ?
>>
>> [1] https://review.opendev.org/#/c/668980/
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__review.opendev.org_-23_c_668980_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=L89wsG0LNn57q1mQMoKsdP6Ws59EGdCArXPqN62UF7Y&s=8-IWFHCVVTKqC2CvGawDzxq2wQDSw4xCVexRf0-EhAE&e=>
>>
>> BR, Loïc
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190809/756c1c93/attachment.html>