jenkinsci/templating-engine-plugin

Welcome! Check out the docs: https://boozallen.github.io/sdp-docs/jte/2.0.1/index.html

silabeer

@silabeer

Hey. I started use JTE. And i have a questions. In my jenkins shared library i use kubernetes agent. May i use shared workspace between different steps in "Provide default pipeline template”. Sorry for my english :D

2 replies

silabeer

@silabeer

And may i use JTE library like Jenkins shared library from file
"file:///var/lib/jenkins/mp-ci”

6 replies

yoav-klein

@yoav-klein

Hi @steven-terrana , where can I find information about JTE Primitive Namespacing and Reverse Library Precedence you mentioned when you first announced JTE 2.0?

2 replies

silabeer

@silabeer

hi. How can I define one variable and use it in all libraries?

12 replies

Madhuri1518

@Madhuri1518_gitlab

Hi @steven-terrana and @cokieffebah
I have a pipeline taking pipeline template from governance tier and pipelien configuration from application repo
Initially build has run successfully. Build duration is 33 min
After sometime pipeline is resuming again and throwing error

[Pipeline] End of Pipeline
[Checks API] No suitable checks publisher found.
Finished: SUCCESS
Resuming build at Mon Feb 22 13:10:03 UTC 2021 after Jenkins restart
[Pipeline] End of Pipeline
[Checks API] No suitable checks publisher found.
java.io.FileNotFoundException: /var/jenkins_home/jobs/***/jobs/sample-spring-boot/builds/15/program.dat (No such file or directory)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader.openStreamAt(RiverReader.java:188)
    at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader.restorePickles(RiverReader.java:136)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.loadProgramAsync(CpsFlowExecution.java:784)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:750)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:691)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:550)
    at hudson.model.RunMap.retrieve(RunMap.java:225)
    at hudson.model.RunMap.retrieve(RunMap.java:57)
    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
    at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
    at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:346)
    at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:276)
    at jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:242)
    at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:240)
    at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:103)
    at org.jenkinsci.plugins.prometheus.JobCollector.appendJobMetrics(JobCollector.java:262)
    at org.jenkinsci.plugins.prometheus.JobCollector.lambda$collect$0(JobCollector.java:220)
    at org.jenkinsci.plugins.prometheus.util.Jobs.forEachJob(Jobs.java:14)
    at org.jenkinsci.plugins.prometheus.JobCollector.collect(JobCollector.java:214)
    at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:190)
    at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:223)
    at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:144)
    at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
    at org.jenkinsci.plugins.prometheus.service.DefaultPrometheusMetrics.collectMetrics(DefaultPrometheusMetrics.java:51)
    at org.jenkinsci.plugins.prometheus.service.PrometheusAsyncWorker.execute(PrometheusAsyncWorker.java:41)
    at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:100)
Caused: java.io.IOException: Failed to load build state
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:865)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:863)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:917)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)

3 replies

jetole

@jetole

How can I use a container for the whole pipeline with JTE?

14 replies

Vincent Letarouilly

@Opa-

Hello everyone,
JTE 2 is producing a lot of "Unreadable Data" in our Jenkins installation. When I go to JENKINS_URL/administrativeMonitor/OldData/manage I have a lot of messages.
Here's a pastebin with 3 of the most encountered errors: https://pastebin.com/JwxkaLZj
I have to manually discard the unreadable data from Jenkins from time to time which is a bit cumbersome ^^
If someone have an idea that would be great :D

Note: I upgraded today from 2.0.1 to 2.0.3 and it had no effect whatsoever

1 reply

Carlos OKieffe

@cokieffebah

@Opa- Sorry about that and thanks for the feedback. our team will be able to get back to fixing the 2.x issues after tomorrow.

1 reply

steven-terrana

@steven-terrana

@/all - it’s awesome seeing all the traffic in this channel. @cokieffebah and I were talking through how to best triage requests here.

some guidance:

If you think your issue is something broken in JTE, please open an GitHub Issue! it’ll help us triage and refer back to bug fixes
Bigger picture feature ideas, feel free to use GitHub Discussions!
Feel free to continue to use this channel for quick questions, etc

jetole

@jetole

Can I use jte { pipeline_template } to include two templates such as run this template than that template?

4 replies

jetole

@jetole

@steven-terrana said "My general advice for this situation would probably be to create two different pipelines for the same repository. One responsible for the C++ code and one for the Java code. This way - you can leverage two different pipeline configurations for the same source code repository." in a github comment. I don't understand how I would implement two pipelines for the same repository.

jetole

@jetole

Is there any way to set environment variables in pipeline_config.groovy ?

1 reply

Arie Heinrich

@ArieHein

@steven-terrana said "My general advice for this situation would probably be to create two different pipelines for the same repository. One responsible for the C++ code and one for the Java code. This way - you can leverage two different pipeline configurations for the same source code repository." in a github comment. I don't understand how I would implement two pipelines for the same repository.

why not use a template that has both ?

6 replies

Serpa

@serpa-dystematic

Hey! Is is possible to reference a field from other library?
For example, I already have a field registry in my docker library, but I need to reference it from other library. I don't want to repeat the input for both libraries if possible.

4 replies

Sanchit Lohia

@sanchitlohia2711

I have two folders A and B in Jenkins at the same hierarchy. Both points to their own Pipeline Configuration and Library Sources for JTE. Both the folder contains a Github organization job pointing to different Github org. I created folder A first and then I created B. The branches that are getting build within the GitHub organization in folder B are picking up the Pipeline Configuration of folder B and Library Sources of Folder A. It is not able to load the library sources of Folder B. Any idea what might be the issue here.

2 replies

Carlos OKieffe

@cokieffebah

@sanchitlohia2711 can you show/take an image of the configurations in question?

Sanchit Lohia

@sanchitlohia2711

@cokieffebah ^^

Mike Deitrick

@mdeitrick

Hello. I am working on creating a declarative pipeline that leverages JTE and I'm running into an issue where the property any does not exist. The templating engine (v2.0.3) is installed in Jenkins. Also, the stack trace in the console output points directly to the line agent any. Would you happen to have any insight into why this is happening?

Pipeline:

pipeline {
    agent any
    options {
        buildDiscarder(logRotator(numToKeepStr: '15', artifactNumToKeepStr: '15'))
        disableConcurrentBuilds()
        timeout(time: 1, unit: 'HOURS')
    }
    stages {
        ...
    }
}

Console Logs (Extract):
hudson.remoting.ProxyException: groovy.lang.MissingPropertyException: No such property: any for class: WorkflowScript

4 replies

namcxn

@namcxn_twitter

hello all, any idea using JTE with monorepo? tks

9 replies

Nayan Bhana

@nayanbhana

Hmmm are there reserved keywords? We have an application_environment we are trying to define called "release" but after the configuration gets added to the release app env, i get an error "Application Environment release already defined". I have searched all our code and we are not defining this anywhere.

6 replies

Jamin

@benjcleveland

Hi everyone! Are there any plans to support a yaml version of the pipeline_config file? We have some use cases where there is duplicate information outside of the pipeline and it would be nice if this could be stored in a single location

3 replies

addihorn

@addihorn

hi @steven-terrana hi @cokieffebah
we have found some kind of strange behavior that suggest, that variables inside the jte-library-code are in global scope.

we have two libaries that both define the same variable filesToCheck

when we ran the libararies as parallel stages eg.

parallel (
    'JobA': {
                job_a (
                    filesToCheck: '.'
                )
            },
    'JobA': {
                job_b(
                    filesToCheck: 'app.py'
                )
            }
)

In both libraries is a line

filesToCheck = args.filesToCheck ?: '.'

Depending on which job executes faster, the value of variable filesToCheck is the same in both executions.
Is there a way, to have the parameters defined in localc scope of the library/call()-method?
Is this even expected bahavior?

Thx

Carlos OKieffe

@cokieffebah

@addihorn the template binding is shared across all libraries. Your libraries are sharing a global (to you) filesToCheck variable that is stored in the binding. if that is not the behaviour you wish in the library, then scope the variable locally by defining it there String filesToCheck = args.filesToCheck ?: '.'

11 replies

Cai Chao

@caichao1103

Hi Eeveryone! Can JTE leverages jenkins shared library ? if i create a global variable in shared library, can i call it from JTE? thx.

Serpa

@serpa-dystematic

Hey guys, is it possible to have stages defined outside of the main pipeline_config.groovy?
I'm wondering if I can split that file so that it doesn't become too big..

Carlos OKieffe

@cokieffebah

@serpa-dystematic currently we don't aggregrate config files that way, horizontally across a project or tier.

Serpa

@serpa-dystematic

hmmm, ok, so I can't aggregate config files, that's fine.
But what is the recommended way of creating stages (groups of steps)?

We want to abstract some steps which are common to some pipelines, but we also have other pipelines which are a different paradigm (ie: infrastructure pipelines) and don't include the libraries that fulfil the steps aggregated in the stage.

As the general pipeline_config.groovy includes those steps, I get an error in the infrastructure pipelines.

Not sure if I'm doing something dumb. Thanks.

4 replies

Serpa

@serpa-dystematic

For what I understand from the docs, this stages should be included in the general pipeline_config.groovy, right?

Tushar Sisode

@tusharsisode

I recently upgraded templating engine plugin from 1.7 to 2.0.4. Did all the necessary DSL updates and pipeline_config updates required for 2.0.4. While running the pipeline, all the libraries are loading successfully but facing below error at various stages the pipeline:

Tushar Sisode

@tusharsisode

java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.Workflowrun*

Thinking that it might be some plugin version conflicting, did below upgrades to the plugins:

Pipeline: Job - 2.40
Job: DSL - 1.77
Junit - 1.49
Checks Api - 1.6.1
Plugin Utilities API - 2.0.0
Pipeline: Supporting APIs - 3.8
Pipeline: SCM step - 2.12
Artifactory - 3.10.6
Pipeline: Groovy - 2.90
Pipeline: Shared Groovy Libraries - 2.18
GIT Server plugin - 1.9
Pipeline: Multibranch: 2.22
Pipeline: API - 2.42
Branch API - 2.6.2
GitHub Branch Source - 2.9.7

However, none of these upgrades helped resolve the issue. The jenkins version currently that I am using is 2.263.4

Any advice on what could be going wrong?

3 replies

addihorn

@addihorn

hi @cokieffebah @steven-terrana
is there a way to fall back to the "default" Jenkins-Steps-Functionality, if have overwritten a step

e.g. I overwrite the build()-Step from my libraries, but want some other job to call the original build()-functionality that is provided from Jenkins.

[JTE] JTE Primitives overrode Plugin provided steps and/or variables: 
[JTE] - build

3 replies

Jamin

@benjcleveland

This might be more of a general Jenkins pipeline question but we are seeing some very long library load times when the JTE pipelines first start. This typically happens when there are a lot of other pipeline jobs running at the same time (Jenkins CPU usage is really high so the longer load times make sense). Does anyone have advice or tips on how to scale Jenkins so it can handle more concurrent pipelines? I've seen single take 7-8 minutes 12:09:14 12:16:09 [2021-04-02T19:16:09.041Z] [JTE] Loading Library ... We are currently loading the libraries from a git repository, would it help if they were loaded from a plugin instead?

Carlos OKieffe

@cokieffebah

@benjcleveland, my default thought is you need 'agents', other than your master Jenkins, to run the additional job loads.
Basically, delegating work to other handlers/machines/containers. https://www.jenkins.io/doc/book/using/using-agents/

12 replies

Matteo Cartuccia

@cartucciam

Hi all, is it possible to use Restart from Stage with JTE?

9 replies

Alex Joual

@AlexandreJoual

Hello everyone, I’m a new user looking to implement JTE for my pipelines. I went through the docs and got setup with a gouvernance tier, library and have an application on hand eager to get built.

I’m running into a small problem : I have an existing Declarative Pipeline that uses a pre-built docker image on Artifactory to run the steps. It looks something like this :

stage('Test stage'){
    agent{
        docker{
            image 'artifactorylinktoimage/npm:versionnumber'
            registryurl 'artifactoryregistryurl'
            registrycredentials 'registrycreds'
            args 'args'
            label 'jenkinsnodelabel'
        }
    }
    steps {
        sh 'command to run goes here --version'
    }
}

I have a simple library step for the sh command in my repo, the simple println statements from the docs run just fine so my whole config is good. However I just can't seem to fathom out how to specify the Artifactory docker part. I just want to pull that specific image and run the step in it.

Any guidance on how I can go about doing this ?

20 replies

Jamin

@benjcleveland

@cokieffebah @steven-terrana Is there an ETA when the patch with the fix for jenkinsci/templating-engine-plugin#195 will be available?

Carlos OKieffe

@cokieffebah

@benjcleveland ^ Next week, steven is out the remainder of this week. Sorry for the delay, other work became priority.

1 reply

Carlos OKieffe

@cokieffebah

@/all The latest Jenkins Security Advisory includes a high vulnerability in JTE.
The vulnerability pertains to the possibility of remote code execution via JTE’s pipeline configuration DSL.
To patch, please upgrade to JTE 2.2 as soon as possible.

Kishore

@krrishkishore

Team, I am new to the Jenkins Template Engine - I have started with the jte learning labs and I am running jenkins container on top of Windows and I am getting the following error when I ran the very first job - /var/jenkins_home/workspace/validate@tmp/durable-a8074292/script.sh: 1: /var/jenkins_home/workspace/validate@tmp/durable-a8074292/script.sh: docker: Permission denied

Hope, I am in the right place for help

Kishore

@krrishkishore

I have used the default windows command which is provided in the documentation - docker run --name jenkins -v /usr/local/bin/docker:/usr/bin/docker -v //var/run/docker.sock:/var/run/docker.sock --privileged --user root -p 50000:50000 -p 8080:8080 -d jenkins/jenkins:lts

Kishore

@krrishkishore

I have encountered another issue similar to this thread - https://www.gitmemory.com/issue/jenkinsci/templating-engine-plugin/200/825178412 with 2.2 version

Kishore

@krrishkishore

I was able to fix the issue on my end. Issue was with the volume we are mounting /usr/local/bin/docker this was needed to be /usr/bin/docker

1 reply

Carlos OKieffe

@cokieffebah

@/all. the latest release, 2.2.1.1 should resolve the TemplatePrimitiveCollector issues.

Gregory Paciga

@gpaciga

Hi, I'm trying to upgrade to from JTE 1.7 to JTE 2+. One issue I'm running into is that my configuration provider doesn't define a Jenkinsfile. With JTE 1.7, pipelines would gracefully fall back to using the Jenkinsfile defined in the project's own repo. With JTE 2.2.1.1, I get an error that the Jenkinsfile can't be found when loading my configuration provider. Is this a new requirement? How do I get around it?

25 replies

Tushar Sisode

@tusharsisode

I recently upgraded to JTE 2.2.1. How do we check if a particular step is present? getBinding.hasStep no longer works which worked until 1.7.1. I know there has been some major code refactor and hasStep has been removed. Is there any alternate way to check if step exists? Cheers!

5 replies

Gregory Paciga

@gpaciga

I'm running into the same issue again, where JTE fails to check out the configured git repo and tries to fall back on running the default Jenkinsfile in jte-config. This time I'm just running on a regular branch, not a PR, on an individual/adhoc pipeline instead of a multibranch one. The issue seems to be that the branchSpec needs to start with an asterisk, so */feature/something rather than just feature/something. Presumably origin/feature/something woud also work. This is unlike when a regular pipeline from scm configuration is used. In Job DSL, JTE requires gitSCM{} but regular pipelines use just git{} which does accept just feature/something. Any thoughts on why JTE uses a different way of configuring git repos?

one consequence of this seems to be that checkout scm does not work in a JTE pipeline

Where communities thrive

People

Repo info

Activity