Skip to content

Issue search support elasticsearch #9428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 13, 2020

Conversation

lunny
Copy link
Member

@lunny lunny commented Dec 19, 2019

Replace #6150

This PR introduce a new issue indexer type - elasticsearch.

It will connect to an elasticsearch server or cluster to store issue indexer and search by keyword.
This will have better performance than db or bleve on a bigger gitea instance.
Especially there are serval gitea instances to share indexer.

This PR also added a test for elastic search issue indexer.

@lunny lunny added the type/feature Completely new functionality. Can only be merged if feature freeze is not active. label Dec 19, 2019
@lunny lunny added this to the 1.12.0 milestone Dec 19, 2019
@codecov-io
Copy link

codecov-io commented Dec 19, 2019

Codecov Report

Merging #9428 into master will increase coverage by 1.31%.
The diff coverage is 44.93%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #9428      +/-   ##
==========================================
+ Coverage   42.36%   43.67%   +1.31%     
==========================================
  Files         619      577      -42     
  Lines       81003    79869    -1134     
==========================================
+ Hits        34313    34885     +572     
+ Misses      42482    40676    -1806     
- Partials     4208     4308     +100
Impacted Files Coverage Δ
modules/structs/repo.go 36.36% <ø> (ø) ⬆️
routers/org/setting.go 0% <ø> (ø) ⬆️
modules/templates/helper.go 40.36% <ø> (ø) ⬆️
models/helper_environment.go 90% <ø> (ø) ⬆️
routers/api/v1/org/member.go 16.01% <0%> (ø) ⬆️
routers/init.go 65.16% <0%> (-2.65%) ⬇️
routers/repo/setting.go 15.01% <0%> (+0.02%) ⬆️
routers/api/v1/admin/user.go 30.19% <0%> (-0.4%) ⬇️
modules/queue/bytefifo.go 0% <0%> (ø)
cmd/serv.go 2.84% <0%> (-0.05%) ⬇️
... and 97 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17722d7...d965f11. Read the comment docs.

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Dec 19, 2019
@lunny lunny force-pushed the lunny/issue_indexer_es2 branch from 09b7458 to c23ffc6 Compare December 19, 2019 15:06
@lunny lunny force-pushed the lunny/issue_indexer_es2 branch from c23ffc6 to 580e2cb Compare December 20, 2019 05:50
@lunny
Copy link
Member Author

lunny commented Dec 25, 2019

Partilly fix #6648

Copy link
Member

@6543 6543 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for my understanding it works

@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Dec 25, 2019
@6543
Copy link
Member

6543 commented Jan 10, 2020

I think your rebase broke something ...

@lunny lunny force-pushed the lunny/issue_indexer_es2 branch from 565f0ea to d31ba64 Compare January 10, 2020 07:56
@lunny
Copy link
Member Author

lunny commented Jan 10, 2020

@6543 fixed. :)

@lunny lunny force-pushed the lunny/issue_indexer_es2 branch from d31ba64 to 51a65a2 Compare January 19, 2020 05:21
"mappings": {
"properties": {
"id": {
"type": "integer",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this compatible with int64?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think yes.

if len(repoIDs) > 0 {
var repoStrs = make([]interface{}, 0, len(repoIDs))
for _, repoID := range repoIDs {
repoStrs = append(repoStrs, repoID)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this intended to be a string, or just []interface{}?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because NewTermsQuery needs a slice of interface as parameters.

@@ -310,6 +323,7 @@ func DeleteRepoIssueIndexer(repo *models.Repository) {
}

// SearchIssuesByKeyword search issue ids by keywords and repo id
// WARNNING: You have to ensure user have permission to visit repoIDs' issues
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This messes up with our paging, but I don't see much way around it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I left it for future PR.

@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Feb 11, 2020
@lunny lunny force-pushed the lunny/issue_indexer_es2 branch from d965f11 to e41403b Compare February 12, 2020 01:30
@lunny
Copy link
Member Author

lunny commented Feb 13, 2020

make L-G-T-M work.

@lunny lunny merged commit 5dbf36f into go-gitea:master Feb 13, 2020
@lunny lunny deleted the lunny/issue_indexer_es2 branch February 13, 2020 06:06
@go-gitea go-gitea locked and limited conversation to collaborators Nov 24, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/feature Completely new functionality. Can only be merged if feature freeze is not active.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants