Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Mar 31 22:21
    dependabot[bot] labeled #68
  • Mar 31 22:21
    dependabot[bot] opened #68
  • Mar 31 22:21

    dependabot[bot] on maven

    Bump guava from 28.2-jre to 29.… (compare)

  • Oct 12 2020 17:03
    markyjackson-taulia commented #35
  • Aug 29 2020 13:49

    Loghijiaha on master

    code cleanup Added HTML view (compare)

  • Aug 29 2020 13:49
    Loghijiaha closed #67
  • Aug 28 2020 18:21
    Loghijiaha commented #67
  • Aug 28 2020 18:20
    Loghijiaha synchronize #67
  • Aug 28 2020 17:48
    Loghijiaha opened #67
  • Aug 28 2020 07:48

    Loghijiaha on master

    Fixed unnecessary SEVERE log (compare)

  • Aug 28 2020 07:48
    Loghijiaha closed #66
  • Aug 27 2020 17:45
    Loghijiaha opened #66
  • Aug 24 2020 20:32

    Loghijiaha on master

    Updated release draft Updated changelog (compare)

  • Aug 24 2020 20:32
    Loghijiaha closed #65
  • Aug 24 2020 20:22
    Loghijiaha commented #65
  • Aug 24 2020 15:44
    Loghijiaha synchronize #65
  • Aug 24 2020 15:43
    Loghijiaha opened #65
  • Aug 24 2020 10:48

    Loghijiaha on master

    [maven-release-plugin] prepare … (compare)

  • Aug 24 2020 10:48

    Loghijiaha on machine-learning-1.0.1

    (compare)

  • Aug 24 2020 10:47

    Loghijiaha on master

    [maven-release-plugin] prepare … (compare)

Loghi
@Loghijiaha
Hi @oleg. The open graph is fantastic. I'd 100 for that. The linkedIn draft is looking good for me. How can I add those three images in the blog post in a aligned line ? Should I merge three of them or ?
Oleg Nenashev
@oleg-nenashev
machine-learning.png
Loghi
@Loghijiaha
okay thanks, @oleg I will give PR on that
Oleg Nenashev
@oleg-nenashev
Yeah. Just follow the example in other blogs. I will launch the LinkedIn post, it does not need opengraph metadata
Ioannis K. Moutsatsos
@imoutsatsos
Nice presentation today @Loghijiaha ! It was great to also realize that our DevOps colleagues are also interested in ML analysis of Jenkins artifacts and builds. Hadn't thought about that @oleg-nenashev @markearlwaite. Thanks!
Shivay Lamba
@shivaylamba
Yes great presentation Loghi
Loghi
@Loghijiaha
Thank you @imoutsatsos @shivaylamba .Will finish this and release a stable/RC release soon. :cheers:
Bruno P. Kinoshita
@kinow
@Loghijiaha for testing in a distributed environment, I think first you can try writing a simple pipeline job https://www.jenkins.io/doc/book/pipeline/docker/#execution-environment
But instead of the node command, try running some Python. You'll also need to check if the machine-learning-plugin is compatible with pipelines, or if we need to add the dependency + a few changes in the Java API.
If that works, then you should be able to create a test using the Jenkins testing API (that you are already using in other tests): https://www.jenkins.io/doc/developer/testing/#within-a-pipeline
Bruno P. Kinoshita
@kinow
Here's an example from a plugin that contains a test for the head node, and another for running in an agent: https://github.com/jenkinsci/cmakebuilder-plugin/blob/273b72bcd732718bd17de4c98932ebf806fc5ea4/src/test/java/hudson/plugins/cmake/CmakeBuilderBuildStepTest.java
Loghi
@Loghijiaha
@kinow @imoutsatsos @shivaylamba I found an API Zeppelin-jupyter where we could specify the kernel then it will run the code where you chooses. I have tested with python and R. It works :full_moon:
However, users need to input which kernel they want. For that can we have a drop-down or a textbox? Or we can discuss if we will be able to set up another meeting this week?
Loghi
@Loghijiaha
R_build.png
R_output.png
Loghi
@Loghijiaha
This feature has implemented in 0.9.0-preview1 Zeppelin. More details http://zeppelin.apache.org/docs/0.9.0-preview1/interpreter/jupyter.html
Loghi
@Loghijiaha
It is working for every kernel that already installed in my system(Eg: javascript, R, etc).
javascript_build.png
javascript_output.png
Shivay Lamba
@shivaylamba
Sure we can have a meeting.
A drop down menu should be fine
Bruno P. Kinoshita
@kinow

@Loghijiaha I think configuring the kernel is fine. As long as the user can choose the kernel in the job configuration, that should be enough. Probably the kernel needs to be selected per build step, so you could have

  • a build step to pre-process data with Python/Groovy
  • a build step to train the model with Python/C/C++
  • a build step to validate the model with Shell

etc. For each build step, s/he would just need to pick the right kernel.

Really looking forward to seeing this in action @Loghijiaha !
I have a meeting tonight, and work tomorrow only until 1/2 PM my time. Then on leave for the rest of the week. Can try to join the meeting, but cannot promise. But do have the meeting if other mentors are available. I'm sure @shivaylamba and @imoutsatsos will have great feedback!
Loghi
@Loghijiaha
That's awesome @shivay. I will work on that soon @kinow. For jupyter interpreter, there are a lot of kernels available here https://github.com/jupyter/jupyter/wiki/Jupyter-kernels. Should I include all these manually in the select option? or any special way to do this ?
Bruno P. Kinoshita
@kinow
@Loghijiaha ideally, it would be nice if Jenkins could figure out what are the kernels I have installed.
Here's what I have at the moment.
(seaglass) kinow@ranma:/tmp$ jupyter kernelspec list --debug --json
{
  "kernelspecs": {
    "javascript": {
      "resource_dir": "/home/kinow/.local/share/jupyter/kernels/javascript",
      "spec": {
        "argv": [
          "ijskernel",
          "--hide-undefined",
          "{connection_file}",
          "--protocol=5.1"
        ],
        "env": {},
        "display_name": "Javascript (Node.js)",
        "language": "javascript",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    },
    "kinow-notebooks": {
      "resource_dir": "/home/kinow/.local/share/jupyter/kernels/kinow-notebooks",
      "spec": {
        "argv": [
          "/home/kinow/Development/python/workspace/notebooks/venv/bin/python",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "kinow-notebooks",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    },
    "tooling": {
      "resource_dir": "/home/kinow/.local/share/jupyter/kernels/tooling",
      "spec": {
        "argv": [
          "/home/kinow/Development/python/tooling/venv/bin/python",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "tooling",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    },
    "python3": {
      "resource_dir": "/home/kinow/Development/python/miniconda3/envs/seaglass/share/jupyter/kernels/python3",
      "spec": {
        "argv": [
          "/home/kinow/Development/python/miniconda3/envs/seaglass/bin/python",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "Python 3",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    }
  }
}

4 kernels. See the language attribute. If Jenkins showed me Julia or R, that would probably result in an error, as I don't have the Kernels installed. It would be nice if it gave me the choice to choose the kernel, like when you start Jupyter Lab/Notebook.

And also displayed the language of the Kernel.

Also, if I wrote a script or uploaded a script in JavaScript, would be nice if it gave me the option to pick from a list of kernels that support javascript.

Said all that, I think we should start as simple as possible. So will take a look at your pull request, and see if I can get JS and Python running @Loghijiaha

Loghi
@Loghijiaha
@kinow I was thinking about that. But when it comes to a distributive environment it is hard to get the kernel spec before we start building.
Bruno P. Kinoshita
@kinow
Hmmm, good point (and you are already thinking about this kind of issues, well done @Loghijiaha !!!).
Bruno P. Kinoshita
@kinow
@Loghijiaha sorry was preparing a few things today for a long holiday. Might still have time to review things, but with less bandwidth until Monday 10th o/
Loghi
@Loghijiaha
That's fine @kinow, I am fixing that CI issue. Later will work on persistent data testing.
And enjoy the holiday !! :hotsprings:
Loghi
@Loghijiaha
Im sorry. My battery was down. I didnt see the power. I will join soon
Loghi
@Loghijiaha
JIRA for kernel configuration -> https://issues.jenkins-ci.org/browse/JENKINS-63363
Loghi
@Loghijiaha

ML plugin can run different kernel codes(pure scripts). When I tried to run a javascript based notebook(JS.ipynb), that is not successful. Because of markdown elements should be replaced with the comment tag(# for python) of the language we are running.

     Note n = new JupyterUtil().getNote(inputStreamReader, "python", "\n", "#");
            Gson gson = new Gson();
            JsonElement tree = gson.toJsonTree(n);
            JsonObject obj = tree.getAsJsonObject();

This works only for python. not working for other languages.
https://issues.jenkins-ci.org/browse/JENKINS-63379

Loghi
@Loghijiaha
I found a way. Fixing it. will give PR tommorow.
Bruno P. Kinoshita
@kinow
Great progress so far @Loghijiaha
@imoutsatsos just tested the plugin during a meeting, and it worked on Windows
Anaconda Prompt, with Python 3.7.1. Windows 10 Pro. JDK 1.8
image.png
Bruno P. Kinoshita
@kinow
image.png
image.png
All I did was
  1. start prompt
  2. python -m venv venv (I have always used venv/virtualenv, so I feel more comfortable using it, rather than the conda environments...)
  3. pip install -r requirements.txt
  4. (venv) (base) C:\Users\kinow\Development\java\jenkins\machine-learning-plugin>c:\Users\kinow\Development\java\apache-maven-3.6.1\bin\mvn.cmd clean install
  5. (venv) (base) C:\Users\kinow\Development\java\jenkins\machine-learning-plugin>c:\Users\kinow\Development\java\apache-maven-3.6.1\bin\mvn.cmd hpi:run
Used latest code in master. Hope that helps.
Loghi
@Loghijiaha
Nice work @kinow