by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Mark Waite
    @MarkEWaite
    Any objections to a git client plugin release to deliver the redundant fetch optimization? We'll plan for a later release of git plugin and git client plugin with the GitToolChooser
    4 replies
    Francisco Javier Fernandez
    @fcojfernandez
    @rishabhBudhouliya I won't make tomorrow's meeting. An unexpected but super-grateful last minute visit (an old friend from childhood), so I won't be available from 3pm CEST (I think your 6.30pm?). Before that time, I will be around here just in case you need something from me!
    rishabhBudhouliya
    @rishabhBudhouliya
    @fcojfernandez Have fun! I'll be pushing some changes which you may check tomorrow morning.
    Thanks for the help!
    1 reply
    Mark Waite
    @MarkEWaite
    @rishabhBudhouliya thanks for joining the Platform SIG meeting today. I learned new things from your question. I've prepared a change to deploy incremental builds from the git plugin CI job. Is that OK for you or would you prefer to submit the pull request yourself?
    rishabhBudhouliya
    @rishabhBudhouliya
    @MarkEWaite I was looking at the git client plugin Jenkinsfile and I have a question, does the buildPlugin take care of incremental builds deployment?
    Mark Waite
    @MarkEWaite
    Yes, it does. I haven't been willing to spend the time to understand why the more complicated version in git plugin does not work.
    4 replies
    rishabhBudhouliya
    @rishabhBudhouliya
    @MarkEWaite Here is the PR:jenkinsci/git-plugin#934
    1 reply
    rishabhBudhouliya
    @rishabhBudhouliya
    Here's the PR for Phase 2 Blog:jenkins-infra/jenkins.io#3576
    Mark Waite
    @MarkEWaite
    I've released git client plugin 3.3.2 https://github.com/jenkinsci/git-client-plugin/releases/tag/git-client-3.3.2 . Includes a fix for a bug introduced in git client plugin 3.1.0 refactoring to reduce deprecation warnings. A warning about the non-zero costs of refactoring with a legacy code base.
    1 reply
    Also a warning that there is always a way to slip something past our best attempts to test software...
    rishabhBudhouliya
    @rishabhBudhouliya
    @fcojfernandez I have pushed some commits related to the suggestions you made after your review.
    8 replies
    rishabhBudhouliya
    @rishabhBudhouliya
    @MarkEWaite @fcojfernandez Since git clone is not implemented yet in the git plugin, if I want to benchmark the difference between a git fetch and git clone, I would have to use this functionality: 
    launchCommand("git", "tag", "--force", tag);
    For clone this would be something like 
    launchCommand("git", "clone", <url>);
    I've seen this functionality in the GitAPITestCase
    rishabhBudhouliya
    @rishabhBudhouliya

    Hi all, Git Plugin GSoC Project 2020 Second Weekly sync is about to start in 20 minutes:
    Zoom Link: https://us02web.zoom.us/j/87507840768?pwd=UVljaS9YeVVKd01Rd0FCc1FCaWRMQT09

    Meeting Agenda:https://docs.google.com/document/d/1ov4ug9WfbcTYNHL1DBcsxyRKgCi7EnFVIywdiP36CSk/edit?usp=sharing

    Oleg Nenashev
    @oleg-nenashev

    @rishabhBudhouliya @MarkEWaite @justinharringa @omkar-dsd @fcojfernandez Please review the LinkedIn post draft:

    Git Plugin Performance project update by Rishabh Budhouliya: bench-marking of Git client implementations and a new GitToolChooser engine which recommends a Git implementation on the basis of the repository size.

    "What we’ve learned so far in this project is that a git fetch is highly correlated to the size of the remote repository. In order to make fetch improvements in this plugin, our task was to find the difference in performance for the two available git implementations in the Git Plugin, git and JGit. Our major finding was that git performs much better than JGit when it comes to a large sized repository (>100 MB). Interestingly, JGit performs better than git when size of the repository is less than 100 MB. In this phase, we were successful in coding this derived knowledge from the benchmarks into a new functionality called the GitToolChooser".

    Learn more at https://www.jenkins.io/blog/2020/07/29/git-performance-improvement-phase2/

    #jenkins #git #gsoc #performancetesting #jmh #benchmarking
    Mark Waite
    @MarkEWaite
    +1 from me for that post to LinkedIn
    Recording of git plugin office hours will be available at https://youtu.be/VTr02ZCCqkk (after YouTube processing completes)
    rishabhBudhouliya
    @rishabhBudhouliya
    Looks great, +1 @oleg-nenashev
    rishabhBudhouliya
    @rishabhBudhouliya
    @MarkEWaite Apologies for disturbing you on a weekend, I am not able to ssh into your local setup. Is it running fine?
    5 replies
    rishabhBudhouliya
    @rishabhBudhouliya
    Hi all, in the last meeting, we had a discussion on the topic "Taking the responsibility to know if a GitSCMExtension (additional behavior) is supported by the git implementations"
    rishabhBudhouliya
    @rishabhBudhouliya

    Some points of discussion:

    • Currently, only at the level of JGitAPIImpl and CliGitAPIImpl (implementation level) the information of whether an operation is supported or not is held and the decision to throw an exception in case of calling an unsupported feature is taken there only.

    • Now the issue is that GitToolChooser needs that information before recommending a git implementation. @MarkEWaite I think it may not be possible to take this information from the implementation as it assumes that a client has been created by the time those methods are called. (Even the decorate-xx methods expect a git client).

    9 replies
    rishabhBudhouliya
    @rishabhBudhouliya

    Hi all, I've been facing some difficulties while creating a generalised way to get the information related to the compatibility of a git implementation w.r.t a git extension.

    @MarkEWaite In the discussion we had, GitClientimplementations were the best place to access the information we need.
    The issue with that approach:

    • Currently, JGitAPIImpl or CliGitAPIImpl are designed to implement GitCommand interfaces like the CloneCommand or CheckoutCommand. These interfaces collect the information which would tell us if a certain extension is implemented by JGit or not.
    • These interfaces are designed in a way to set properties, not to get them. For example, if I were to determine if LFS pull is supported by JGit or not, I would need the field value of String lfsRemote which is set in the CheckoutCommand implementation within the client.
    • Currently, there is no way to get this information.
    13 replies
    One possible solution I see is to change the GitCommand behavior within the git client implementations, but would that be an appropriate choice? @MarkEWaite @fcojfernandez
    rishabhBudhouliya
    @rishabhBudhouliya
    Hi all, I have created a new PR for adding a new command called the UnsupportedCommand
    jenkinsci/git-client-plugin#594
    This command will not be implemented by JGitAPIImpl or CliGitAPIImpl. It can be instantiated directly into the Git Plugin, can be decorated by GitSCMExtensions to add availability of a feature in JGit.
    It returns a boolean, if JGit should be used or not.
    That boolean can be used by GitToolChooser to make the right choice.
    @MarkEWaite @fcojfernandez @justinharringa If this looks like an acceptable solution, I will go forward with creating the necessary changes in the Git Plugin.
    Mark Waite
    @MarkEWaite
    I think you should go forward. I haven't yet understood how it will be used in context but assume that I will be able to see that usage either through tests you create in git client plugin or through usage from the git plugin.
    rishabhBudhouliya
    @rishabhBudhouliya
    Fine by me, I'll work on that.
    rishabhBudhouliya
    @rishabhBudhouliya

    Hi all, I have created two new pull requests:
    rishabhBudhouliya/git-plugin#6 -> In my forked repository, it contains the addition of UnsupportedCommand feature in git plugin only

    jenkinsci/git-plugin#937 -> In the original git plugin repo, it contains GitToolChooser source code + UnsupportedCommand feature

    2 replies
    rishabhBudhouliya
    @rishabhBudhouliya
    If anyone's interested in looking at the benchmark results of Git Clone vs Git Fetch, here is the doc:
    https://docs.google.com/document/d/1o49hjgecIDwhcA-prObHgnXjJRE3NGoUFgZQwwGde2c/edit?usp=sharing
    rishabhBudhouliya
    @rishabhBudhouliya

    Hi all, Git Plugin performance improvement Weekly Sync will start in about 30 minutes:
    Meeting agenda: https://docs.google.com/document/d/1ov4ug9WfbcTYNHL1DBcsxyRKgCi7EnFVIywdiP36CSk/edit?usp=sharing

    Zoom link:
    https://us02web.zoom.us/j/89630615756?pwd=aXJ0QzBjREtZZjhKTzVLUDRQc2NHZz09

    rishabhBudhouliya
    @rishabhBudhouliya
    I have a design question: For the API endpoint we provide in GitToolChooser,
    if we wish to add the support of using user credentials to get size of private repositories -
    • we should let the branch source plugin take that responsibility as it would know the credentials as well as the type of credential required to authenticate a private user api endpoint
    • The git plugin should send the credentials as it is acting as a client which is requesting for an information without giving full user/project details
    I think @justinharringa said something about the choice of using ssh credentials or a token, according to that discussion, the branch source plugin would be a better judge, is that the right conclusion? @MarkEWaite @fcojfernandez @omkar-dsd
    Mark Waite
    @MarkEWaite
    I think the branch source plugin is best suited to decide which credentials it should use to query repository size. However, I'm not sure how it will choose that credential unless an administrator has already assigned a credential to a repository or a credential to a GitHub organization. Try it and see?
    rishabhBudhouliya
    @rishabhBudhouliya
    Yes, I'm doing that. Since the extension class is a static class, I am not sure if we will be able to bring non-static objects like credentials id and context of the job from the branch source plugin.
    Rather, we can send the (Item) context and CredentialsId so that then the plugin can match the appropriate credential.
    rishabhBudhouliya
    @rishabhBudhouliya

    Hi everyone!
    We will have the GSoC project second weekly sync in about 30 mins:

    Meeting agenda: https://docs.google.com/document/d/1ov4ug9WfbcTYNHL1DBcsxyRKgCi7EnFVIywdiP36CSk/edit?usp=sharing

    Zoom link:
    https://us02web.zoom.us/j/87507840768?pwd=UVljaS9YeVVKd01Rd0FCc1FCaWRMQT09

    Omkar Deshpande
    @omkar-dsd
    I think @rishabhBudhouliya has covered all the deliverables for the GSOC. Definitely on track ! Keep it up Rishabh. :D
    1 reply
    rishabhBudhouliya
    @rishabhBudhouliya
    Thanks @omkar-dsd and @justinharringa . Mentors are kind enough to say this, I think we should do more in terms of achieving the ultimate goal of improvement in performance for every user out there.
    Of course we need to deliver what we promised first :smile:
    Martin d'Anjou
    @martinda
    Is there a diagram or a presentation that explains the Jenkins internal caches of Git repositories? Does the Jenkins main server transfer the git cache content to agents?
    Mark Waite
    @MarkEWaite
    AbstractGitSCMSource implements a cache of git repositories on the master ( https://github.com/jenkinsci/git-plugin/blob/fee213002d58753ddb66d71449631a80222ec3d5/src/main/java/jenkins/plugins/git/AbstractGitSCMSource.java#L339 ). AbstractGitSCMSource is used by multibranch pipeline implementations to decide if changes are detected on any branch. The cache is not transferred to agents. A Google Summer of Code 2020 project proposal describes some of the topic that you might find helpful. https://www.jenkins.io/projects/gsoc/2020/project-ideas/git-plugin-caching/ .
    AbstractGitSCMSource is not used by Freestyle or Matrix jobs.
    rishabhBudhouliya
    @rishabhBudhouliya
    @MarkEWaite @fcojfernandez @justinharringa @omkar-dsd Hi all, I have updated the GitToolChooser to support to get size of private remote repos. Please refer to this comment on the PR for more details.
    I have interactively tested this change by adding a username:password type of credentials for a freestyle job with a personal private repository, GitToolChooser is able to recommend as expected.
    rishabhBudhouliya
    @rishabhBudhouliya

    @MarkEWaite I was unable to find a functionality within the Github branch source plugin which would allow us to search for valid credentials with just the repository url.

    I will be exploring more branch source plugins to find a workable solution for our case, as with this change, the responsibility to provide credentials is on the git plugin which might limit GitToolChooser's ability to recommend tool in certain cases. (It will only work for username:pwd credentials)

    As far as I have studied the Credentials API, for one to look for a valid credential in the credentials store, two things are necessary:
    • Item context
    • CredentialsId
    Mark Waite
    @MarkEWaite
    Thanks for checking @rishabhBudhouliya . That supports your earlier idea that we must include credentials in the request.
    1 reply