Migrating git repos to OpenDev
Hi, As discussed in November[1], the OpenStack project infrastructure is being rebranded as "OpenDev" to better support a wider community of projects. We are nearly ready to perform the part of this transition with the largest impact: moving the authoritative git repositories for existing projects. In this email, I'd like to introduce the new hosting system we are preparing, discuss the transition, and invite projects to work with us on the logistics of the change. Gerrit ====== Gerrit is the core of our system and it will remain so in OpenDev. As part of this move, we will rename the gerrit server from review.openstack.org to review.opendev.org. As part of the transition, we will automatically merge appropriate changes to all branches of all repositories updating .gitreview and Zuul configuration files. Any further changes (README files, etc.) we expect to be made by individual project contributors. Repository Browsing =================== Currently our canonical *public* repository system is the cgit server at https://git.openstack.org/ (and git.airshipit.org, git.starlingx.io, and git.zuul-ci.org). This is a load balanced cluster of several servers which is designed to handle all the public git repository traffic, as it scales much better than Gerrit (and has a more friendly domain name).
From a technical standpoint, it's excellent, but its usability could be improved.
Therefore, as part of this transition, we will replace the cgit servers with a new system based on Gitea. Gitea is a complete development collaboration system, but it's very flexible and will allow us to disable components which we aren't using. We will operate it in a read-only configuration where it will act as the public mirror for Gerrit. The advantages it has over the current system are: * Shorter domain name in project URLs: https://git.openstack.org/openstack/nova vs https://opendev.org/openstack/nova * Clone and browsing URLs are the same (with cgit, the browsing URL has an extra path component) * More visually pleasing code browsing * Integrated code searching * Ability to highlight multiple lines in links When we perform the transition we will install redirects from git.openstack.org (and the other git sites) to opendev.org, and will maintain those redirects for the foreseeable future. We will construct them so that even existing deep links to individual files in individual commits to cgit will redirect to the correct location on opendev.org. This system is up and running now with a live mirror of data from Gerrit, and you can start testing it out today at https://opendev.org/ Please let us know if you encounter any problems. If you would like to read more about the design of this system and the transition, see the infra-spec[2]. GitHub ====== Currently all OpenStack projects are replicated to GitHub. We do not plan on changing that during the transition, however, any projects outside of the openstack*/ namespaces will not automatically be replicated to GitHub, and we do not plan on adding that in the future. We do, however, support projects using Zuul to run post-merge jobs to push updates to GitHub or any other third-party mirrors with their own credentials. We will be happy to work with anyone interested in that to help set up jobs to do so. We are adopting this approach so that individual projects can have more control over how they are represented in social media, and to give us more flexibility in supporting our own organizational namespaces on OpenDev without assuming they map directly to GitHub. Eventually we plan on moving the OpenStack project to that system as well and retiring direct replication from Gerrit to GitHub completely. But we will defer that work until after this transition. Logistics ========= We can prepare much of the system in advance (as we have for the hosting system on opendev.org), but the actual transition and renaming of the Gerrit server will need to happen at once during an outage window. We need to schedule that outage and begin preparing for it. Since all of the project git URLs are going to change (to replace git.openstack.org with opendev.org and review.openstack.org with review.opendev.org), we can additionally take the opportunity to reorganize projects into different organizations. For example, during the transition we will rename Zuul, and it's associated projects, from the "openstack-infra" org to "zuul". So their new names will be "zuul/zuul", "zuul/nodepool", etc. This is an excellent time for the rest of the OpenStack Foundation pilot projects to do the same. If the OpenStack project desires this, it would also be a good time to move unofficial projects out of the openstack/ namespace. Therefore, we need your help: Action Items ============ We need each of the following projects: * OpenStack * Airship * StarlingX * Zuul To nominate a single point of contact to work with us on the transition. It would be helpful for that person to attend the next (and possibly next several) openstack infra team meetings in IRC [3]. We will work with those people on scheduling the transition, as well as finalizing the list of projects which should be renamed as part of the transition. If you manage an unofficial project and would like to take the opportunity to move or rename your project, please add it to this ethercalc[4]. [1] http://lists.openstack.org/pipermail/openstack-dev/2018-November/136403.html [2] http://specs.openstack.org/openstack-infra/infra-specs/specs/opendev-gerrit.... [3] http://eavesdrop.openstack.org/#Project_Infrastructure_Team_Meeting [4] https://ethercalc.openstack.org/opendev-transition
On Thu, Mar 7, 2019, at 1:47 PM, James E. Blair wrote:
Hi,
As discussed in November[1], the OpenStack project infrastructure is being rebranded as "OpenDev" to better support a wider community of projects.
We are nearly ready to perform the part of this transition with the largest impact: moving the authoritative git repositories for existing projects.
In this email, I'd like to introduce the new hosting system we are preparing, discuss the transition, and invite projects to work with us on the logistics of the change.
Gerrit ======
Gerrit is the core of our system and it will remain so in OpenDev. As part of this move, we will rename the gerrit server from review.openstack.org to review.opendev.org. As part of the transition, we will automatically merge appropriate changes to all branches of all repositories updating .gitreview and Zuul configuration files. Any further changes (README files, etc.) we expect to be made by individual project contributors.
Repository Browsing ===================
Currently our canonical *public* repository system is the cgit server at https://git.openstack.org/ (and git.airshipit.org, git.starlingx.io, and git.zuul-ci.org). This is a load balanced cluster of several servers which is designed to handle all the public git repository traffic, as it scales much better than Gerrit (and has a more friendly domain name). From a technical standpoint, it's excellent, but its usability could be improved.
Therefore, as part of this transition, we will replace the cgit servers with a new system based on Gitea. Gitea is a complete development collaboration system, but it's very flexible and will allow us to disable components which we aren't using. We will operate it in a read-only configuration where it will act as the public mirror for Gerrit. The advantages it has over the current system are:
* Shorter domain name in project URLs: https://git.openstack.org/openstack/nova vs https://opendev.org/openstack/nova * Clone and browsing URLs are the same (with cgit, the browsing URL has an extra path component) * More visually pleasing code browsing * Integrated code searching * Ability to highlight multiple lines in links
When we perform the transition we will install redirects from git.openstack.org (and the other git sites) to opendev.org, and will maintain those redirects for the foreseeable future. We will construct them so that even existing deep links to individual files in individual commits to cgit will redirect to the correct location on opendev.org.
This system is up and running now with a live mirror of data from Gerrit, and you can start testing it out today at https://opendev.org/
Please let us know if you encounter any problems.
If you would like to read more about the design of this system and the transition, see the infra-spec[2].
GitHub ======
Currently all OpenStack projects are replicated to GitHub. We do not plan on changing that during the transition, however, any projects outside of the openstack*/ namespaces will not automatically be replicated to GitHub, and we do not plan on adding that in the future. We do, however, support projects using Zuul to run post-merge jobs to push updates to GitHub or any other third-party mirrors with their own credentials. We will be happy to work with anyone interested in that to help set up jobs to do so.
We are adopting this approach so that individual projects can have more control over how they are represented in social media, and to give us more flexibility in supporting our own organizational namespaces on OpenDev without assuming they map directly to GitHub.
Eventually we plan on moving the OpenStack project to that system as well and retiring direct replication from Gerrit to GitHub completely. But we will defer that work until after this transition.
Logistics =========
We can prepare much of the system in advance (as we have for the hosting system on opendev.org), but the actual transition and renaming of the Gerrit server will need to happen at once during an outage window. We need to schedule that outage and begin preparing for it.
Since all of the project git URLs are going to change (to replace git.openstack.org with opendev.org and review.openstack.org with review.opendev.org), we can additionally take the opportunity to reorganize projects into different organizations.
For example, during the transition we will rename Zuul, and it's associated projects, from the "openstack-infra" org to "zuul". So their new names will be "zuul/zuul", "zuul/nodepool", etc.
This is an excellent time for the rest of the OpenStack Foundation pilot projects to do the same.
If the OpenStack project desires this, it would also be a good time to move unofficial projects out of the openstack/ namespace.
Therefore, we need your help:
Action Items ============
We need each of the following projects:
* OpenStack * Airship * StarlingX * Zuul
To nominate a single point of contact to work with us on the transition. It would be helpful for that person to attend the next (and possibly next several) openstack infra team meetings in IRC [3]. We will work with those people on scheduling the transition, as well as finalizing the list of projects which should be renamed as part of the transition.
If you manage an unofficial project and would like to take the opportunity to move or rename your project, please add it to this ethercalc[4].
[1] http://lists.openstack.org/pipermail/openstack-dev/2018-November/136403.html [2] http://specs.openstack.org/openstack-infra/infra-specs/specs/opendev-gerrit.... [3] http://eavesdrop.openstack.org/#Project_Infrastructure_Team_Meeting [4] https://ethercalc.openstack.org/opendev-transition
We've made good progress in preparing this change and are still on track to do this Friday April 19, 2019. Our project liasons have been drawing up lists of projects to rename during the outage. One big thing to keep in mind is unofficial OpenStack projects will no longer be in the "openstack" namespace. They will be placed in the 'x/' namespace instead which intends to indicate no endorsement or special ownership. One side effect of this change (as noted in Jim's earlier email) is that we will stop replicating projects that move out of the openstack namespace. Projects that wish to be replicated to Github or anywhere else they like can do so following the steps that David Moreau-Simard put together for us here [5]. This transition is likely to be a bit bumpy particularly at the start. We'll be around after the transition to help fix unexpected errors and are likely to spend a fair bit of time at the PTG improving things as well. Finally, to be extra clear, we intend to put http redirects in place so that all your old http(s) urls continue to work. Fungi has set this up for testing with details here [6] if you would like to ensure your urls redirect properly. [5] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005007.htm... [6] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004921.htm... Thank you for your patience and feel free to reach out with any questions you might have. Clark
On Thu, Apr 11, 2019, at 3:20 PM, Clark Boylan wrote:
On Thu, Mar 7, 2019, at 1:47 PM, James E. Blair wrote:
Hi,
As discussed in November[1], the OpenStack project infrastructure is being rebranded as "OpenDev" to better support a wider community of projects.
We are nearly ready to perform the part of this transition with the largest impact: moving the authoritative git repositories for existing projects.
In this email, I'd like to introduce the new hosting system we are preparing, discuss the transition, and invite projects to work with us on the logistics of the change.
Gerrit ======
Gerrit is the core of our system and it will remain so in OpenDev. As part of this move, we will rename the gerrit server from review.openstack.org to review.opendev.org. As part of the transition, we will automatically merge appropriate changes to all branches of all repositories updating .gitreview and Zuul configuration files. Any further changes (README files, etc.) we expect to be made by individual project contributors.
Repository Browsing ===================
Currently our canonical *public* repository system is the cgit server at https://git.openstack.org/ (and git.airshipit.org, git.starlingx.io, and git.zuul-ci.org). This is a load balanced cluster of several servers which is designed to handle all the public git repository traffic, as it scales much better than Gerrit (and has a more friendly domain name). From a technical standpoint, it's excellent, but its usability could be improved.
Therefore, as part of this transition, we will replace the cgit servers with a new system based on Gitea. Gitea is a complete development collaboration system, but it's very flexible and will allow us to disable components which we aren't using. We will operate it in a read-only configuration where it will act as the public mirror for Gerrit. The advantages it has over the current system are:
* Shorter domain name in project URLs: https://git.openstack.org/openstack/nova vs https://opendev.org/openstack/nova * Clone and browsing URLs are the same (with cgit, the browsing URL has an extra path component) * More visually pleasing code browsing * Integrated code searching * Ability to highlight multiple lines in links
When we perform the transition we will install redirects from git.openstack.org (and the other git sites) to opendev.org, and will maintain those redirects for the foreseeable future. We will construct them so that even existing deep links to individual files in individual commits to cgit will redirect to the correct location on opendev.org.
This system is up and running now with a live mirror of data from Gerrit, and you can start testing it out today at https://opendev.org/
Please let us know if you encounter any problems.
If you would like to read more about the design of this system and the transition, see the infra-spec[2].
GitHub ======
Currently all OpenStack projects are replicated to GitHub. We do not plan on changing that during the transition, however, any projects outside of the openstack*/ namespaces will not automatically be replicated to GitHub, and we do not plan on adding that in the future. We do, however, support projects using Zuul to run post-merge jobs to push updates to GitHub or any other third-party mirrors with their own credentials. We will be happy to work with anyone interested in that to help set up jobs to do so.
We are adopting this approach so that individual projects can have more control over how they are represented in social media, and to give us more flexibility in supporting our own organizational namespaces on OpenDev without assuming they map directly to GitHub.
Eventually we plan on moving the OpenStack project to that system as well and retiring direct replication from Gerrit to GitHub completely. But we will defer that work until after this transition.
Logistics =========
We can prepare much of the system in advance (as we have for the hosting system on opendev.org), but the actual transition and renaming of the Gerrit server will need to happen at once during an outage window. We need to schedule that outage and begin preparing for it.
Since all of the project git URLs are going to change (to replace git.openstack.org with opendev.org and review.openstack.org with review.opendev.org), we can additionally take the opportunity to reorganize projects into different organizations.
For example, during the transition we will rename Zuul, and it's associated projects, from the "openstack-infra" org to "zuul". So their new names will be "zuul/zuul", "zuul/nodepool", etc.
This is an excellent time for the rest of the OpenStack Foundation pilot projects to do the same.
If the OpenStack project desires this, it would also be a good time to move unofficial projects out of the openstack/ namespace.
Therefore, we need your help:
Action Items ============
We need each of the following projects:
* OpenStack * Airship * StarlingX * Zuul
To nominate a single point of contact to work with us on the transition. It would be helpful for that person to attend the next (and possibly next several) openstack infra team meetings in IRC [3]. We will work with those people on scheduling the transition, as well as finalizing the list of projects which should be renamed as part of the transition.
If you manage an unofficial project and would like to take the opportunity to move or rename your project, please add it to this ethercalc[4].
[1] http://lists.openstack.org/pipermail/openstack-dev/2018-November/136403.html [2] http://specs.openstack.org/openstack-infra/infra-specs/specs/opendev-gerrit.... [3] http://eavesdrop.openstack.org/#Project_Infrastructure_Team_Meeting [4] https://ethercalc.openstack.org/opendev-transition
We've made good progress in preparing this change and are still on track to do this Friday April 19, 2019. Our project liasons have been drawing up lists of projects to rename during the outage. One big thing to keep in mind is unofficial OpenStack projects will no longer be in the "openstack" namespace. They will be placed in the 'x/' namespace instead which intends to indicate no endorsement or special ownership.
One side effect of this change (as noted in Jim's earlier email) is that we will stop replicating projects that move out of the openstack namespace. Projects that wish to be replicated to Github or anywhere else they like can do so following the steps that David Moreau-Simard put together for us here [5].
This transition is likely to be a bit bumpy particularly at the start. We'll be around after the transition to help fix unexpected errors and are likely to spend a fair bit of time at the PTG improving things as well.
Finally, to be extra clear, we intend to put http redirects in place so that all your old http(s) urls continue to work. Fungi has set this up for testing with details here [6] if you would like to ensure your urls redirect properly.
[5] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005007.htm... [6] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004921.htm...
The infra/OpenDev team continues to make good progress towards this transition and plans to perform the transition on April 19, 2019 as previously scheduled. We will begin the transition at 15:00UTC and users should plan for intermittent Gerrit and git repo outages through the day. We expect most of those will be closer to 15:00UTC than 23:00UTC. Fungi has generated a master list of project renames for the openstack namespaces: http://paste.openstack.org/show/749402/. If you have a moment please quickly review these planned renames for any obvious errors or issues. For the airship, starlingx, and zuul repo renames the repositories listed at git.airshipit.org, git.starlingx.io, and git.zuul-ci.org were used placing repos in airship/, starlingx/ and zuul/ namespaces. Any repo name prefix (like stx- and airship-) is dropped. As always we are happy to answer any questions you might have or address any concerns. Feel free to reach out. Thank you for your patience, Clark
On Tue, Apr 16, 2019, at 5:11 PM, Clark Boylan wrote:
The infra/OpenDev team continues to make good progress towards this transition and plans to perform the transition on April 19, 2019 as previously scheduled. We will begin the transition at 15:00UTC and users should plan for intermittent Gerrit and git repo outages through the day. We expect most of those will be closer to 15:00UTC than 23:00UTC.
Fungi has generated a master list of project renames for the openstack namespaces: http://paste.openstack.org/show/749402/. If you have a moment please quickly review these planned renames for any obvious errors or issues.
For the airship, starlingx, and zuul repo renames the repositories listed at git.airshipit.org, git.starlingx.io, and git.zuul-ci.org were used placing repos in airship/, starlingx/ and zuul/ namespaces. Any repo name prefix (like stx- and airship-) is dropped.
Hello everyone! The Infra team, no, OpenDev Infra team, has completed the initial work to migrate to a more flexible git hosting platform. Gerrit is now hosted at review.opendev.org (with redirects from review.openstack.org in place). Our git mirroring has transitioned from git.openstack.org, git.airshipit.org, git.starlingx.io, and git.zuul-ci.org to http(s)://opendev.org. We have put in place redirects from these old domains to the new domain. If you see James Blair at the summit and appreciate these redirects: buy him a beverage. As part of this transition we've also moved git repos around. Airship, Zuul, OpenDev, StarlingX and others now have their own top level "orgs" which we hope cuts down on confusion over what is or isn't part of a certain project. You will likely want to update your git remotes to the new canonical locations (where ever you get redirected to using the old urls) in the near future. For unofficial projects that were hosted under openstack/ the OpenStack TC decided to move them out into a different org (namespace). The chosen namespace was x/ because it is short and doesn't convey any particular meaning. In time we can continue to use more meaningful namespaces, but for now I think we want to stabilize after the recent moves. The Zuul tenant configs report no errors (Jeremy Stanley gets the beverages for this); however, that does not mean all jobs will be happy. We will be around to help work people through job failures related to this migration. Please reach out and we'll go from there. On the infrastructure side of things we will also need to work to get our continuous deployment running again. This was a bit of an inception move for us as we do all our work through Gerrit and we had to take Gerrit down and "move" it. Finally thank you all for your patience today and thank you to everyone that helped make this possible. Clark
participants (2)
-
Clark Boylan
-
corvus＠inaugust.com