Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Mar 31 22:21
    dependabot[bot] labeled #68
  • Mar 31 22:21
    dependabot[bot] opened #68
  • Mar 31 22:21

    dependabot[bot] on maven

    Bump guava from 28.2-jre to 29.… (compare)

  • Oct 12 2020 17:03
    markyjackson-taulia commented #35
  • Aug 29 2020 13:49

    Loghijiaha on master

    code cleanup Added HTML view (compare)

  • Aug 29 2020 13:49
    Loghijiaha closed #67
  • Aug 28 2020 18:21
    Loghijiaha commented #67
  • Aug 28 2020 18:20
    Loghijiaha synchronize #67
  • Aug 28 2020 17:48
    Loghijiaha opened #67
  • Aug 28 2020 07:48

    Loghijiaha on master

    Fixed unnecessary SEVERE log (compare)

  • Aug 28 2020 07:48
    Loghijiaha closed #66
  • Aug 27 2020 17:45
    Loghijiaha opened #66
  • Aug 24 2020 20:32

    Loghijiaha on master

    Updated release draft Updated changelog (compare)

  • Aug 24 2020 20:32
    Loghijiaha closed #65
  • Aug 24 2020 20:22
    Loghijiaha commented #65
  • Aug 24 2020 15:44
    Loghijiaha synchronize #65
  • Aug 24 2020 15:43
    Loghijiaha opened #65
  • Aug 24 2020 10:48

    Loghijiaha on master

    [maven-release-plugin] prepare … (compare)

  • Aug 24 2020 10:48

    Loghijiaha on machine-learning-1.0.1

    (compare)

  • Aug 24 2020 10:47

    Loghijiaha on master

    [maven-release-plugin] prepare … (compare)

Loghi
@Loghijiaha
JIRA for kernel configuration -> https://issues.jenkins-ci.org/browse/JENKINS-63363
Loghi
@Loghijiaha

ML plugin can run different kernel codes(pure scripts). When I tried to run a javascript based notebook(JS.ipynb), that is not successful. Because of markdown elements should be replaced with the comment tag(# for python) of the language we are running.

     Note n = new JupyterUtil().getNote(inputStreamReader, "python", "\n", "#");
            Gson gson = new Gson();
            JsonElement tree = gson.toJsonTree(n);
            JsonObject obj = tree.getAsJsonObject();

This works only for python. not working for other languages.
https://issues.jenkins-ci.org/browse/JENKINS-63379

Loghi
@Loghijiaha
I found a way. Fixing it. will give PR tommorow.
Bruno P. Kinoshita
@kinow
Great progress so far @Loghijiaha
@imoutsatsos just tested the plugin during a meeting, and it worked on Windows
Anaconda Prompt, with Python 3.7.1. Windows 10 Pro. JDK 1.8
image.png
Bruno P. Kinoshita
@kinow
image.png
image.png
All I did was
  1. start prompt
  2. python -m venv venv (I have always used venv/virtualenv, so I feel more comfortable using it, rather than the conda environments...)
  3. pip install -r requirements.txt
  4. (venv) (base) C:\Users\kinow\Development\java\jenkins\machine-learning-plugin>c:\Users\kinow\Development\java\apache-maven-3.6.1\bin\mvn.cmd clean install
  5. (venv) (base) C:\Users\kinow\Development\java\jenkins\machine-learning-plugin>c:\Users\kinow\Development\java\apache-maven-3.6.1\bin\mvn.cmd hpi:run
Used latest code in master. Hope that helps.
Loghi
@Loghijiaha
Nice work @kinow
Bruno P. Kinoshita
@kinow
Thanks @Loghijiaha !
Loghi
@Loghijiaha
A quick question. Should we need to save images and HTML in different directories associated with build number?
Bruno P. Kinoshita
@kinow
I don't think so. I think we should just output the files in the workspace. If users want, they can use plug-ins or organize their builds to archive artefacts or move the files elsewhere, etc. But maybe @Loghijiaha @imoutsatsos have use cases for that?
1 reply
Loghi
@Loghijiaha
jenkinsci/machine-learning-plugin#51 opened PR for the last task to be finished :hand:
Loghi
@Loghijiaha
Today's meeting :
Zoom link : https://zoom.us/j/98822238159
(LogLevel, not Level I think)
Arg, that's from the plug-in. Let me open the IDE to confirm what methods are aavailable
Bruno P. Kinoshita
@kinow

https://github.com/jenkinsci/jira-trigger-plugin/blob/7b96a75ae7aba5ca5f2828825f1470b9093b77c3/src/integrationTest/groovy/com/ceilfors/jenkins/plugins/jiratrigger/integration/JulLogLevelRule.groovy#L35

It looks like calling the java.util.Logger setLevel works for the NullPointerException. But there's another exception that persists even after this change. I wonder if it wouldn't be the case to dig into the code of Zeppelin to understand why both exceptions are happening @Loghijiaha ? Perhaps it's a parameter that is missing, or some method that was not invoked?

4 replies
Loghi
@Loghijiaha
One of Zeppelin maintainer commented on that issue. https://issues.apache.org/jira/browse/ZEPPELIN-4569
@kinow
Bruno P. Kinoshita
@kinow
Great initiative @Loghijiaha . I've replied to his comment too. Let's see if they know how to silence those exceptions.
Loghi
@Loghijiaha
@all still there is no reply to the issue in https://issues.apache.org/jira/browse/ZEPPELIN-4569. If we are set with the pending issue, We may release the stable version in the next few days.
Bruno P. Kinoshita
@kinow
Let's wait and see if they reply, and keep investigating in the meantime @Loghijiaha . Found this one bug debugging the plug-in: https://issues.jenkins-ci.org/browse/JENKINS-63452
Loghi
@Loghijiaha
Ah. Nice catch @kinow will look into that
Bruno P. Kinoshita
@kinow
@Loghijiaha spent some more time debugging it. Still no luck. But maybe this one can give us some clue: grpc/grpc-java#1577
Loghi
@Loghijiaha
And also we can remove the directory changing line and let it as default.
Bruno P. Kinoshita
@kinow
Yup. I don't think this is really a blocker for a release @Loghijiaha . We can set a deadline, something like Friday or Saturday your time, up to you.
If we can't find a solution, you can prepare a release before the demo.
That way after others watch your demo they can grab this latest release, and you can work on this issues with more calm. I believe we have everything from our design doc submitted to GSoC right?
Loghi
@Loghijiaha
Yh, we all set with the GSoC design doc with some minor changes.
Bruno P. Kinoshita
@kinow
Brilliant. I'll keep looking at this logging issue in the spare time. But in the worst case we just need to tell users about it.
Loghi
@Loghijiaha
Great @kinow :heavy_check_mark:
Loghi
@Loghijiaha

Demo 3 schedule:

Part 1, 2PM UTC, Aug 27 (Thu)

  • Rishabh
  • Kezhi
  • Sladyn

Part 2, 2PM UTC, Aug 28 (Fri)

  • Loghi
  • Buddhika
  • Sumit
Loghi
@Loghijiaha
@kinow there is an issue with default directory settings. Notebooks/scripts are not recognizing the files/dataset eg: example_with_images. We should at least have to change the directory to the workspace for python.
Loghi
@Loghijiaha
procEnvir.png
How can we change the env only for this process this.jupyterKernelProcessLauncher = new JupyterKernelInterpreter.JupyterKernelProcessLauncher(cmd, envs);?
Bruno P. Kinoshita
@kinow
@Loghijiaha most command line libraries have methods to define the working directory. With the Kernels, we rely on Zeppelin providing a method to specify the working directory… let me have a look (just drinking the first cup of coffee of the day.. might need a second one for this issue haha)
@Loghijiaha let us know if you need help with the final work submission, please. Related to this guideline: https://developers.google.com/open-source/gsoc/help/work-product
Loghi
@Loghijiaha
Haha i will let you know, if I need help on final submission.
Bruno P. Kinoshita
@kinow

@Loghijiaha I've rebased the code with your last changes, including the one that removes the error with javascript kernel (trying to change working directory).

With the following diff, the GRPC internal log is hidden, without disabling the logger for other plug-ins. WDYT? Feel free to use this snippet in a PR in case that works.

diff --git a/src/main/java/io/jenkins/plugins/ml/IPythonBuilder.java b/src/main/java/io/jenkins/plugins/ml/IPythonBuilder.java
index 3ae8e04..71423cf 100644
--- a/src/main/java/io/jenkins/plugins/ml/IPythonBuilder.java
+++ b/src/main/java/io/jenkins/plugins/ml/IPythonBuilder.java
@@ -60,6 +60,7 @@ import java.io.InputStreamReader;
 import java.io.Serializable;
 import java.nio.charset.Charset;
 import java.util.List;
+import java.util.logging.Level;
 import java.util.stream.Stream;

 /**
@@ -70,6 +71,8 @@ public class IPythonBuilder extends Builder implements SimpleBuildStep, Serializ
     private static final long serialVersionUID = 1L;

     private static final Logger LOGGER = LoggerFactory.getLogger(IPythonBuilder.class);
+    private static final java.util.logging.Logger GRPC_IO_LOGGER = java.util.logging.Logger.getLogger("io.grpc.internal");
+    private static final Level GRPC_IO_LOGGER_ORIGINAL_LEVEL = GRPC_IO_LOGGER.getLevel();

     private final String code;
     private final String filePath;
@@ -105,6 +108,8 @@ public class IPythonBuilder extends Builder implements SimpleBuildStep, Serializ
     @Override
     @SuppressFBWarnings("NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE")
     public void perform(@Nonnull Run<?, ?> run, @Nonnull FilePath ws, @Nonnull Launcher launcher, @Nonnull TaskListener listener) throws AbortException {
+        // Disable GRPC internal exceptions due to SEVERE log message when closing kernel interpreter
+        GRPC_IO_LOGGER.setLevel(Level.OFF);
         try {
             if (parserType.isEmpty() || task.isEmpty() || kernelName.isEmpty()) {
                 throw new AbortException("IPython builder is mis-configured ");
@@ -131,12 +136,13 @@ public class IPythonBuilder extends Builder implements SimpleBuildStep, Serializ
             } else {
                 run.addAction(new ResultAction(run, ws));
             }
-
-
         } catch (Throwable e) {
             e.printStackTrace(listener.getLogger());
             throw  new AbortException(e.getMessage());
         }
+        finally {
+            GRPC_IO_LOGGER.setLevel(GRPC_IO_LOGGER_ORIGINAL_LEVEL);
+        }
     }

     @Nullable
diff --git a/src/main/java/io/jenkins/plugins/ml/IPythonInterpreterManager.java b/src/main/java/io/jenkins/plugins/ml/IPythonInterpreterManager.java
index e019d2d..61477e6 100644
--- a/src/main/java/io/jenkins/plugins/ml/IPythonInterpreterManager.java
+++ b/src/main/java/io/jenkins/plugins/ml/IPythonInterpreterManager.java
@@ -157,7 +157,7 @@ public class IPythonInterpreterManager extends InterpreterManager {
         try {
             kernelInterpreter.shutdown();
         } catch (InterpreterException e) {
-            e.printStackTrace();
+            LOGGER.trace("Failed to close interpreter", e);
         }
     }
 }
Might need some further tests, check if everything makes sense, etc. But at least with this I don't see anything not expected in the Jenkins logs.
Bruno P. Kinoshita
@kinow
As for the working directory @Loghijiaha , I think there are two options at least. One is to create an issue in JIRA for Zeppelin asking to have a way to set the working directory, and maybe a PR to simply use a new value from that options-hashmap, and just call the method to setWorkingDirectory over here https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/util/ProcessLauncher.java#L89

The other option is to find a way to trick the JVM into changing the working directory. Since Zeppelin will run new File("."), the only way would be to bypass that.

If we changed the working directory to the workspace, "." would be correct (instead of the IDE directory for example).

Another way... maybe... would be to monkey-patch the DefaultExecutor code, which is a bit trickier/riskier I think.
Bruno P. Kinoshita
@kinow

@Loghijiaha here's what I meant with monkey-patching... there's another way for that... classpath shadowing? Can't recall it sorry. Spent the day looking at JavaScript code, and quickly switched to Java in lunch break and a bit after work, so didn't have much time to do a good job. Sorry.

The pull request also accidentally includes the logging diff I pasted above hehe. Anyhoo, it worked for me. But this is the kind of code I'm never proud of. It works, but that's really not elegant. Feel free to take the decision on how to fix the log & the working directory. I just want to give a few options, but they are definitely not the ultimate (or even unique) solutions.

Have a great weekend!
Cheers

jenkinsci/machine-learning-plugin#57

Loghi
@Loghijiaha
Thanks @kinow That works. will find a way to change the working directory cheers