Redeploy node using armada failed
Hello, What is the correct way to reprovision a node and make it join the cluster again using promenade? In the first step of deployment, using promenade, join-nodes.sh scripts are generated. In my case, I create a cluster with 10 nodes including 3 masters. What if I want to reprovision one node? I tries to do kubectl delete node ceph01 and I reinstall the node and give it the same ip and hostname as the previous one and then execute the same join script (which was generated before, and already executed on the previous node before deleting it). The problem is that calico-etcd-ceph01 pods keeps failing. Here is the log http://paste.openstack.org/show/726411/ What should I do to be able to redeploy nodes? Best,
Hello James, Thank you for trying Airship! Our developer Bryan is currently working on a user story to make Shipyard support a redeploy_server action that will tear down a node and then deploy the node. A few notes about this: 1) we're going initially for a destructive tear down, meaning that the operator has to care for dependent workload moves and control plane adjustments before invoking. 2) there is a longer term plan to make a graceful teardown that will attempt to preserve the health of the control plane proactively, and allow dependent workloads an opportunity to take action before the teardown and redeployment. As for right now, before this story is complete and available, any teardown of a node would be manual, but running an update_site action through Shipyard would trigger the steps to bring the node back in alignment with the committed design for the site. Best regards, -- Roman Gorshunov On Sun, Jul 22, 2018 at 7:50 PM, James Devon <jr8586335@gmail.com> wrote:
Hello,
What is the correct way to reprovision a node and make it join the cluster again using promenade?
In the first step of deployment, using promenade, join-nodes.sh scripts are generated.
In my case, I create a cluster with 10 nodes including 3 masters. What if I want to reprovision one node? I tries to do kubectl delete node ceph01 and I reinstall the node and give it the same ip and hostname as the previous one and then execute the same join script (which was generated before, and already executed on the previous node before deleting it).
The problem is that calico-etcd-ceph01 pods keeps failing. Here is the log http://paste.openstack.org/show/726411/
What should I do to be able to redeploy nodes?
Best,
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hi, Thanks for your answer. 1. I'm not using shipyard. Instead I am managing my environment with a configuration management tool for now. 2. How can I concretely teardown a node? Is there a command to run? I'm using only armada and promenade. Best, On Mon, Jul 23, 2018, 7:44 PM Roman Gorshunov <paye600@gmail.com> wrote:
Hello James,
Thank you for trying Airship!
Our developer Bryan is currently working on a user story to make Shipyard support a redeploy_server action that will tear down a node and then deploy the node.
A few notes about this: 1) we're going initially for a destructive tear down, meaning that the operator has to care for dependent workload moves and control plane adjustments before invoking. 2) there is a longer term plan to make a graceful teardown that will attempt to preserve the health of the control plane proactively, and allow dependent workloads an opportunity to take action before the teardown and redeployment.
As for right now, before this story is complete and available, any teardown of a node would be manual, but running an update_site action through Shipyard would trigger the steps to bring the node back in alignment with the committed design for the site.
Best regards, -- Roman Gorshunov
On Sun, Jul 22, 2018 at 7:50 PM, James Devon <jr8586335@gmail.com> wrote:
Hello,
What is the correct way to reprovision a node and make it join the cluster again using promenade?
In the first step of deployment, using promenade, join-nodes.sh scripts are generated.
In my case, I create a cluster with 10 nodes including 3 masters. What if I want to reprovision one node? I tries to do kubectl delete node ceph01 and I reinstall the node and give it the same ip and hostname as the previous one and then execute the same join script (which was generated before, and already executed on the previous node before deleting it).
The problem is that calico-etcd-ceph01 pods keeps failing. Here is the log http://paste.openstack.org/show/726411/
What should I do to be able to redeploy nodes?
Best,
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hi James, You should be able to remove a node via /user/local/bin/promenade-teardown. That script is used in the resiliency test that is run via tools/gate.sh in the promenade repo. You can find the template for that script here: https://github.com/openstack/airship-promenade/blob/9e0ab1871aad273787cb40f0... Best, Mark On Mon, Jul 23, 2018 at 2:05 PM James Devon <jr8586335@gmail.com> wrote:
Hi,
Thanks for your answer.
1. I'm not using shipyard. Instead I am managing my environment with a configuration management tool for now.
2. How can I concretely teardown a node? Is there a command to run? I'm using only armada and promenade.
Best,
On Mon, Jul 23, 2018, 7:44 PM Roman Gorshunov <paye600@gmail.com> wrote:
Hello James,
Thank you for trying Airship!
Our developer Bryan is currently working on a user story to make Shipyard support a redeploy_server action that will tear down a node and then deploy the node.
A few notes about this: 1) we're going initially for a destructive tear down, meaning that the operator has to care for dependent workload moves and control plane adjustments before invoking. 2) there is a longer term plan to make a graceful teardown that will attempt to preserve the health of the control plane proactively, and allow dependent workloads an opportunity to take action before the teardown and redeployment.
As for right now, before this story is complete and available, any teardown of a node would be manual, but running an update_site action through Shipyard would trigger the steps to bring the node back in alignment with the committed design for the site.
Best regards, -- Roman Gorshunov
On Sun, Jul 22, 2018 at 7:50 PM, James Devon <jr8586335@gmail.com> wrote:
Hello,
What is the correct way to reprovision a node and make it join the cluster again using promenade?
In the first step of deployment, using promenade, join-nodes.sh scripts are generated.
In my case, I create a cluster with 10 nodes including 3 masters. What if I want to reprovision one node? I tries to do kubectl delete node ceph01 and I reinstall the node and give it the same ip and hostname as the previous one and then execute the same join script (which was generated before, and already executed on the previous node before deleting it).
The problem is that calico-etcd-ceph01 pods keeps failing. Here is the log http://paste.openstack.org/show/726411/
What should I do to be able to redeploy nodes?
Best,
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
participants (3)
-
James Devon
-
Mark Burnett
-
Roman Gorshunov