Improve migrations to support migrating milestones/labels/issues/comments/pullrequests #6290
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| --- | ||
| date: "2019-04-15T17:29:00+08:00" | ||
| title: "Advanced: Migrations Interfaces" | ||
| slug: "migrations-interfaces" | ||
| weight: 30 | ||
| toc: true | ||
| draft: false | ||
| menu: | ||
| sidebar: | ||
| parent: "advanced" | ||
| name: "Migrations Interfaces" | ||
| weight: 55 | ||
| identifier: "migrations-interfaces" | ||
| --- | ||
|
|
||
| # Migration Features | ||
|
|
||
| The new migration features were introduced in Gitea 1.9.0. It defines two interfaces to support migrating | ||
| repositories data from other git host platforms to gitea or, in the future migrating gitea data to other | ||
| git host platforms. Currently, only the migrations from github via APIv3 to Gitea is implemented. | ||
lunny marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| First of all, Gitea defines some standard objects in packages `modules/migrations/base`. They are | ||
lunny marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| `Repository`, `Milestone`, `Release`, `Label`, `Issue`, `Comment`, `PullRequest`. | ||
|
|
||
| ## Downloader Interfaces | ||
|
|
||
| To migrate from a new git host platform, there are two steps to be updated. | ||
lunny marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| - You should implement a `Downloader` which will get all kinds of repository informations. | ||
lunny marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - You should implement a `DownloaderFactory` which is used to detect if the URL matches and | ||
lunny marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| create a Downloader. | ||
lunny marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - You'll need to register the `DownloaderFactory` via `RegisterDownloaderFactory` on init. | ||
lunny marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ```Go | ||
| type Downloader interface { | ||
| GetRepoInfo() (*Repository, error) | ||
| GetMilestones() ([]*Milestone, error) | ||
| GetReleases() ([]*Release, error) | ||
| GetLabels() ([]*Label, error) | ||
| GetIssues(start, limit int) ([]*Issue, error) | ||
| GetComments(issueNumber int64) ([]*Comment, error) | ||
| GetPullRequests(start, limit int) ([]*PullRequest, error) | ||
| } | ||
| ``` | ||
|
|
||
| ```Go | ||
| type DownloaderFactory interface { | ||
| Match(opts MigrateOptions) (bool, error) | ||
| New(opts MigrateOptions) (Downloader, error) | ||
| } | ||
| ``` | ||
|
|
||
| ## Uploader Interface | ||
|
|
||
| Currently, only a `GiteaLocalUploader` is implemented, so we only save downloaded | ||
lunny marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| data via this `Uploader` on the local Gitea instance. Other uploaders are not supported | ||
lunny marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| and will be implemented in future. | ||
|
|
||
| ```Go | ||
| // Uploader uploads all the informations | ||
| type Uploader interface { | ||
| CreateRepo(repo *Repository, includeWiki bool) error | ||
| CreateMilestone(milestone *Milestone) error | ||
| CreateRelease(release *Release) error | ||
| CreateLabel(label *Label) error | ||
| CreateIssue(issue *Issue) error | ||
| CreateComment(issueNumber int64, comment *Comment) error | ||
| CreatePullRequest(pr *PullRequest) error | ||
| Rollback() error | ||
| } | ||
|
|
||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| // Copyright 2019 The Gitea Authors. All rights reserved. | ||
| // Use of this source code is governed by a MIT-style | ||
| // license that can be found in the LICENSE file. | ||
|
|
||
| package models | ||
|
|
||
| import "github.com/go-xorm/xorm" | ||
|
|
||
| // InsertIssue insert one issue to database | ||
| func InsertIssue(issue *Issue, labelIDs []int64) error { | ||
|
There was a problem hiding this comment. How is this different to There was a problem hiding this comment. This will not trigger notifications. There was a problem hiding this comment. But then I'd prefer to have There was a problem hiding this comment. I want to do that, but it's so different between theses two functions I would like let it duplicated currently. Maybe send a refactor PR after this PR. |
||
| sess := x.NewSession() | ||
| if err := sess.Begin(); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| if err := insertIssue(sess, issue, labelIDs); err != nil { | ||
| return err | ||
| } | ||
| return sess.Commit() | ||
| } | ||
|
|
||
| func insertIssue(sess *xorm.Session, issue *Issue, labelIDs []int64) error { | ||
| if issue.MilestoneID > 0 { | ||
| sess.Incr("num_issues") | ||
| if issue.IsClosed { | ||
| sess.Incr("num_closed_issues") | ||
| } | ||
| if _, err := sess.ID(issue.MilestoneID).NoAutoTime().Update(new(Milestone)); err != nil { | ||
| return err | ||
| } | ||
| } | ||
| if _, err := sess.NoAutoTime().Insert(issue); err != nil { | ||
| return err | ||
| } | ||
| var issueLabels = make([]IssueLabel, 0, len(labelIDs)) | ||
| for _, labelID := range labelIDs { | ||
| issueLabels = append(issueLabels, IssueLabel{ | ||
| IssueID: issue.ID, | ||
| LabelID: labelID, | ||
| }) | ||
| } | ||
| if _, err := sess.Insert(issueLabels); err != nil { | ||
| return err | ||
| } | ||
| if !issue.IsPull { | ||
| sess.ID(issue.RepoID).Incr("num_issues") | ||
| if issue.IsClosed { | ||
| sess.Incr("num_closed_issues") | ||
| } | ||
| } else { | ||
| sess.ID(issue.RepoID).Incr("num_pulls") | ||
| if issue.IsClosed { | ||
| sess.Incr("num_closed_pulls") | ||
| } | ||
| } | ||
| if _, err := sess.NoAutoTime().Update(issue.Repo); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| sess.Incr("num_issues") | ||
| if issue.IsClosed { | ||
| sess.Incr("num_closed_issues") | ||
| } | ||
| if _, err := sess.In("id", labelIDs).Update(new(Label)); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| if issue.MilestoneID > 0 { | ||
| if _, err := sess.ID(issue.MilestoneID).SetExpr("completeness", "num_closed_issues * 100 / num_issues").Update(new(Milestone)); err != nil { | ||
| return err | ||
| } | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
|
|
||
| // InsertComment inserted a comment | ||
| func InsertComment(comment *Comment) error { | ||
| sess := x.NewSession() | ||
| defer sess.Close() | ||
| if err := sess.Begin(); err != nil { | ||
| return err | ||
| } | ||
| if _, err := sess.NoAutoTime().Insert(comment); err != nil { | ||
| return err | ||
| } | ||
| if _, err := sess.ID(comment.IssueID).Incr("num_comments").Update(new(Issue)); err != nil { | ||
| return err | ||
| } | ||
| return sess.Commit() | ||
| } | ||
|
|
||
| // InsertPullRequest inserted a pull request | ||
| func InsertPullRequest(pr *PullRequest, labelIDs []int64) error { | ||
| sess := x.NewSession() | ||
| defer sess.Close() | ||
| if err := sess.Begin(); err != nil { | ||
| return err | ||
| } | ||
| if err := insertIssue(sess, pr.Issue, labelIDs); err != nil { | ||
| return err | ||
| } | ||
| pr.IssueID = pr.Issue.ID | ||
| if _, err := sess.NoAutoTime().Insert(pr); err != nil { | ||
| return err | ||
| } | ||
| return sess.Commit() | ||
| } | ||
|
|
||
| // MigrateRelease migrates release | ||
| func MigrateRelease(rel *Release) error { | ||
| sess := x.NewSession() | ||
| if err := sess.Begin(); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| var oriRel = Release{ | ||
| RepoID: rel.RepoID, | ||
| TagName: rel.TagName, | ||
| } | ||
| exist, err := sess.Get(&oriRel) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| if !exist { | ||
| if _, err := sess.NoAutoTime().Insert(rel); err != nil { | ||
| return err | ||
| } | ||
| } else { | ||
| rel.ID = oriRel.ID | ||
| if _, err := sess.ID(rel.ID).Cols("target, title, note, is_tag, num_commits").Update(rel); err != nil { | ||
| return err | ||
| } | ||
| } | ||
|
|
||
| for i := 0; i < len(rel.Attachments); i++ { | ||
| rel.Attachments[i].ReleaseID = rel.ID | ||
| } | ||
|
|
||
| if _, err := sess.NoAutoTime().Insert(rel.Attachments); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| return sess.Commit() | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| // Copyright 2019 The Gitea Authors. All rights reserved. | ||
| // Copyright 2018 Jonas Franz. All rights reserved. | ||
| // Use of this source code is governed by a MIT-style | ||
| // license that can be found in the LICENSE file. | ||
|
|
||
| package base | ||
|
|
||
| import "time" | ||
|
|
||
| // Comment is a standard comment information | ||
| type Comment struct { | ||
| PosterName string | ||
| PosterEmail string | ||
| Created time.Time | ||
| Content string | ||
| Reactions *Reactions | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.