Make admin git gc tasks run in background #10629

zeripath · 2020-03-06T05:31:45Z

The admin git gc and git fsck tasks should run in the background rather than run on the request scope.

Any errors should then be put in to the notices.

Fix #10299

guillep2k

A couple of comments, just in case they're relevant

modules/repository/check.go

options/locale/locale_en-US.ini

zeripath · 2020-03-06T05:50:19Z

Now should we be putting a system notice to say that we've completed the gc/fsck?

zeripath · 2020-03-06T05:52:09Z

None of the other tasks seem to do this. So perhaps it would be better to do it as another PR

lunny · 2020-03-06T06:05:10Z

@zeripath I would like to create a record on task table with a new type and track the status of them.

zeripath · 2020-03-06T06:11:21Z

You could leverage the process table - there's nothing that requires that to be an actual running external process

silverwind · 2020-03-06T06:25:57Z

Now should we be putting a system notice to say that we've completed the gc/fsck?

I'd go for it. Given that it can only be started manually, a notice would not be unexpected.

lunny

I think we should use a unique queue to keep only one gc once for a repository.

zeripath · 2020-03-06T12:45:22Z

We can just run it with WithUnique(...) like we do for Cronable events.

silverwind · 2020-03-06T18:23:31Z

Tried this locally, UI showed GC was started but in the log I see this error (no stack or anything):

2020/03/06 19:21:35 ...uters/admin/admin.go:186:func1() [E] Error whilst running git gc: 0x1c97a40

Any way to show more error detail?

codecov-io · 2020-03-06T20:29:53Z

Codecov Report

❗ No coverage uploaded for pull request base (master@345515a). Click here to learn what that means.
The diff coverage is 41.3%.

@@            Coverage Diff            @@
##             master   #10629   +/-   ##
=========================================
  Coverage          ?   43.61%           
=========================================
  Files             ?      588           
  Lines             ?    82502           
  Branches          ?        0           
=========================================
  Hits              ?    35980           
  Misses            ?    42060           
  Partials          ?     4462

Impacted Files	Coverage Δ
routers/admin/admin.go	`10.21% <0%> (ø)`
modules/repository/check.go	`0% <0%> (ø)`
modules/cron/cron.go	`50.49% <61.29%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 345515a...9af8901. Read the comment docs.

zeripath · 2020-03-06T20:49:40Z

@silverwind There was a bug in the previous version try now

silverwind · 2020-03-06T22:05:58Z

modules/repository/check.go

@@ -72,13 +72,18 @@ func GitGcRepos(ctx context.Context) error {
 			if err := repo.GetOwner(); err != nil {
 				return err
 			}
+			log.Trace("Running git gc on %v", repo)


What logging config do I need to see these? I have LEVEL = Trace and tried both file/stdout logging. Also, I'd suggest raising this and similar ones to Info.

zeripath · 2020-03-07T10:27:22Z

OK. We have two options - keep this a simple just background the GC tasks PR - or properly sort out these admin and cron tasks.

That is:

Admin and cron task should appear in Processes - with cancellable contexts
They should report when they're done - presumably into system notices?

As an extension cron tasks need restructuring - they should probably be merged with the admin tasks - and their config moved to the db?

silverwind · 2020-03-07T10:36:31Z

They should report when they're done - presumably into system notices?

Yes, please. I don't feel like one should have to dig into logs for this. I'd say also report when admin actions start in the system notices.

Additional "nicities" I could see:

Add the username who started it to the start notice
Add either a report that lists repo size before/after gc or just the raw CLI output to the finish notice.

zeripath · 2020-03-16T22:34:03Z

I'm going to close this in favour of an all singing all dancing refactor...

zeripath · 2020-03-16T22:48:07Z

See #10745

somera · 2020-06-18T21:58:27Z

How is the status here? My hope was to get this in 1.12.0.

zeripath added the type/enhancement An improvement of existing functionality label Mar 6, 2020

zeripath added this to the 1.12.0 milestone Mar 6, 2020

Make admin git gc/fsck tasks run in background

334df6a

zeripath force-pushed the background-admin-git-gc-task branch from b727957 to 334df6a Compare March 6, 2020 05:36

guillep2k approved these changes Mar 6, 2020

View reviewed changes

modules/repository/check.go Show resolved Hide resolved

options/locale/locale_en-US.ini Outdated Show resolved Hide resolved

GiteaBot added the lgtm/need 1 This PR needs approval from one additional maintainer to be merged. label Mar 6, 2020

lunny approved these changes Mar 6, 2020

View reviewed changes

GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Mar 6, 2020

Update options/locale/locale_en-US.ini

9d54340

zeripath added 2 commits March 6, 2020 07:04

Merge branch 'master' into background-admin-git-gc-task

4f791e0

Merge branch 'master' into background-admin-git-gc-task

8e7faec

lafriks approved these changes Mar 6, 2020

View reviewed changes

lunny requested changes Mar 6, 2020

View reviewed changes

silverwind mentioned this pull request Mar 6, 2020

CLI command to trigger repo garbage collection #10530

Closed

use WithUnique for admin tasks

9af8901

Merge branch 'master' into background-admin-git-gc-task

ebd1e76

silverwind reviewed Mar 6, 2020

View reviewed changes

Merge branch 'master' into background-admin-git-gc-task

e12b6b9

zeripath closed this Mar 16, 2020

zeripath deleted the background-admin-git-gc-task branch September 13, 2020 12:34

go-gitea locked and limited conversation to collaborators Nov 24, 2020

Uh oh!

Make admin git gc tasks run in background #10629

Make admin git gc tasks run in background #10629

Uh oh!

Conversation

zeripath commented Mar 6, 2020

Uh oh!

guillep2k left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zeripath commented Mar 6, 2020

Uh oh!

zeripath commented Mar 6, 2020

Uh oh!

lunny commented Mar 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zeripath commented Mar 6, 2020

Uh oh!

silverwind commented Mar 6, 2020

Uh oh!

lunny left a comment

Choose a reason for hiding this comment

Uh oh!

zeripath commented Mar 6, 2020

Uh oh!

silverwind commented Mar 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-io commented Mar 6, 2020

Codecov Report

Uh oh!

zeripath commented Mar 6, 2020

Uh oh!

silverwind Mar 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zeripath commented Mar 7, 2020

Uh oh!

silverwind commented Mar 7, 2020

Uh oh!

zeripath commented Mar 16, 2020

Uh oh!

zeripath commented Mar 16, 2020

Uh oh!

somera commented Jun 18, 2020

Uh oh!

Uh oh!

lunny commented Mar 6, 2020 •

edited

Loading

silverwind commented Mar 6, 2020 •

edited

Loading

silverwind Mar 6, 2020 •

edited

Loading