GitVersion Performance Issues

[LegacyEbenZhang]

Create a pull request, so people may help test or improve. It's not done yet, at least have to fix the unit tests.

This pull request uses git command line to replace the libgit2sharp to find the branches for a commit/tag. Will be happy to see if someone knows how to do the same thing in libgit2sharp.

[robdmoore]

It looks like this requires having 'git' available on the command line? Based on my (possibly incorrect) understanding, GitVersion is currently independent and doesn't require that?

[EbenZhang]

If git is not there it will use the original code.

[asbjornu]

Perhaps @nulltoken knows how to do this in libgit2sharp?

[nulltoken]

The (untested) following code may be what you're after:

repo.Refs.ReachableFrom(new[] { commit })
   .Where(r => r.IsLocalBranch || r.IsTag)

[EbenZhang]

OK, all tests passed.

I disabled the CommitMessageIncrementMode by default. Feel free to remove that commit if you guys think it's important.

Don't know how to deal with the TrackMergeTargetBaseVersionStrategy, so if you're using GitFlow, you will probably still have the issue.

git executable is required in the PATH to speed it up.

[orjan]

This is a breaking change, and I think it's better to handle this in a separate issue!

On 15 Jan 2016, at 15:40, EbenZhang [email protected] wrote:

OK, all test passed.

I disabled the CommitMessageIncrementMode by default. Feel free to remove that commit if you guys think it's important.

—
Reply to this email directly or view it on GitHub.

[EbenZhang]

This is a breaking change

Ok, breaking change removed.

[JakeGinnivan]

@EbenZhang could you try @nulltoken's code? It would be great to get the perf boost without having to have git installed.

[EbenZhang]

@JakeGinnivan, testing it now. Already 10 minutes passed.
There must be something special in git's database, so the command line returns immediately.

[JakeGinnivan]

@nulltoken any ideas on the perf side of things? Seems odd we should have to drop to the command line.

I wonder if the newer versions of libgit2sharp are quicker in this regard though. I have this arvo blocked out to close out as many issues and investigate stuff as I can

[EbenZhang]

It takes 41m:50s for gitversion to finish.

[EbenZhang]

@nulltoken, the code i used was copied from the libgit2sharp's Uncyclo page to get the branches for commit/tag (roughly 79 branches, 5666 commits results from git rev-list --all --count)

private IEnumerable<Branch> ListBranchesContainingCommit(Repository repo, string commitSha)
{
    var localHeads = repo.Refs.Where(r => r.IsLocalBranch());

    var commit = repo.Lookup<Commit>(commitSha);
    var localHeadsContainingTheCommit = repo.Refs.ReachableFrom(localHeads, new[] { commit });

    return localHeadsContainingTheCommit
        .Select(branchRef => repo.Branches[branchRef.CanonicalName]);
}

[nulltoken]

@EbenZhang Can you please share the following information:

• a public repository against which this is reproducible
• the commit sha that you've been testing
• the duration of the execution of the ListBranchesContainingCommit method

Thanks in advance.

[EbenZhang]

Hi @nulltoken and @JakeGinnivan ,

I can't share the private repo with you guys, but fortunately it seems not hard to reproduce.

After some more tests, I've found something really interesting, hope someone can explain.

Env:

• I've create 70 branches on my GitExtensions repo. The commit being tested is 9fe628226c51f2087e3a9ac0218595fee154b258 which is a very old tag 1.08, tagged at 7/Jan/2009. (I don't understand why GitVersion has to look for such an old tag).
• Build machine profile: AMD FX 8350 Eight Core 4.02G CPU, 2G memory, 32bit windows7 Professional

Code:

I tested 3 different ways to get the branches for a commit.

• A: a new console application using the same version of libgit2sharp that GitVersion is using, the nativebinaries are also the same. and also same config: AnyCPU, Release Version, .NET4.0.

        private static void Main(string[] args)
        {
            Repository repo = new Repository(@"MyGitextensionsRepoPath");
            var w = new Stopwatch();
            w.Start();
            var x = ListBranchesContainingCommit(repo, "9fe628226c51f2087e3a9ac0218595fee154b258").ToList();
            w.Stop();
            using(var test  = new StreamWriter("test.log"))
            {
                test.Write(w.ElapsedMilliseconds);
            }
        }

• B: On the GitVersion side, I added the same ListBranchesContainingCommit to the LibGitExtensions.cs and called it as below in its GetBranchesContainingCommit method

            var w = new Stopwatch();
            w.Start();
            var k = ListBranchesContainingCommit(repository, "9fe628226c51f2087e3a9ac0218595fee154b258").ToList();
            w.Stop();
            using (var test = new StreamWriter("test1.log"))
            {
                test.Write(w.ElapsedMilliseconds);
            }
            Environment.Exit(0); // exit GitVersion as I only want to get the time for the commit.

• C: the GitVersion using git command line, which is what this pull request is doing.

Result

A took 838ms, B took 19994ms, C took 423ms

Provide that I'm not wrong about GitVersion that it will look for branches all tags and also branches for some commits, B will take at least 3315 minutes (20sAverage 10s/tag * 100 tags in GitExtensions repository) to get the version.

On a 8GB memory Windows10 64-bit PC, B took 16843ms.

Question

• A and B are the same but why B took way longer?
• Is it possible to reduce the number of calls to that function?

[EbenZhang]

Profiling Result:

• A: The faster one (link to vsps file created by vs2015 profiler)

• B: The slower one

[JakeGinnivan]

@EbenZhang can you give me more info how you are hitting these perf issues?

I am trying to replicate against the git extensions repo without issues

[EbenZhang]

can you give me more info how you are hitting these perf issues?

You may want to remove the .Where(r => r.IsLocalBranch()); from repo.Refs.Where(r => r.IsLocalBranch()); because all the branches are remote branches. Or you can create some local branches youself to test the ListBranchesContainingCommit function.

GitVersion doesn't really have a problem to get the version for GitExtensions repo. I used it to profile the performance of the ListBranchesContainingCommit function.

• If I put the code in GitVersion, it takes about 20s to get the branches for commit 9fe628226c51f2087e3a9ac0218595fee154b258.
• If create a new project, it takes less than 1s to do the same thing.
• Later today I found if I use new Repository(repoPath) instead of using the IRepository in GitVersion, it works as good as the new project.

[JakeGinnivan]

@EbenZhang I am more interested in how you get GitVersion.exe to perform badly.

Optimising a function may not be the way to go, most of the perf improvements have come from re-thinking the problem and coming up with a better solution.

[JakeGinnivan]

@EbenZhangEmbed I am keen to work with you on this, should I just cherry pick the non git command line stuff out of here?

[EbenZhang]

Tried again today on the problematic pull request, unfortunately, I'm not able to reproduce it now (probably because some branches were deleted or pull requests closed, or maybe new tags created). I'll config one of the build agent to use the official GitVersion and keep an eye on its performance.

Maybe you can have a look of the log to see if there's any clue. Here is the log with confidential stuff removed.

should I just cherry pick the non git command line stuff out of here?

I'll extract them and create another pull request because without the git command line improvement I should remove the duplicate code between TrackMergeTargetBaseVersionStrategy and its base class.

[JakeGinnivan]

0_o, that is some bad perf!

If the changes you have made (without the cmd line stuff) improves the perf or simplifies things then I would love to pull it in.

I will also have a bit of a look into possible causes for the timing you have seen.

[EbenZhang]

#765 created, I will refresh this pull request to get rid of the commits from #765 once #765 gets merged

[JakeGinnivan]

Done

[asbjornu]

@EbenZhang #765 has been merged. Could you please update this PR? :)

[JakeGinnivan]

Id like to remove the calling of the git exe. The library shouldn't be faster.

Gonna close this one out, then if we can provide a example repo which causes the perf issues we can investigate.