yes, the cleanup plugin can be used in conjunction with EWM, but it's optional
@raghunathkrish_twitter the workspace should be downloaded only once on your common diskpool, afterwards, if you configure the plugin correctly, you should have access to that workspace via mount points to that diskpool.
@alexsomai No worries. I am researching an update to EWM that will clear external workspaces once all builds "pointing" to it have been deleted by routine build discards. My current proposed implementation will read fingerprint build data and ask the Workspace Cleanup Plugin to clear an external workspace once no fingerprints contain a given external workspace directory. Does this implementation seem compatible with EWM processes? Are you aware if there might be an accessible point of implementation for this within the preexisting Workspace Cleanup plugin support in EWM?
@BenjaminBeggs there is some fingerprint support implemented within EWM, maybe that could help you at some degree. I don't think the Workspace Cleanup plugin is aware of, or has access in any way to EWM.
There were some discussions about what you are trying to achieve in the build discarder plugin. Somehow, that plugin to be extended to support EWM (or the other way around, EWM to be compatible with build discarder plugin). Would this help you maybe?
There were some discussions about what you are trying to achieve in the build discarder plugin. Somehow, that plugin to be extended to support EWM (or the other way around, EWM to be compatible with build discarder plugin). Would this help you maybe?
@raghunathkrish_twitter the Jenkins master, and every Jenkins slave should be mounted to the same disk pool. In this way, if one job downloads the workspace from the remote repository, then other Jenkins slaves will have access to the workspace, because they are all mounted to it.
So to answer your question, the slave is mounted to the disk pool, therefore it has access to the workspace. No additional download should be needed.
So to answer your question, the slave is mounted to the disk pool, therefore it has access to the workspace. No additional download should be needed.
hi @martinda. I've merged jenkinsci/external-workspace-manager-plugin#68. I'm thinking to do a release with this change, to have support for JCasC, but not very sure when. Also, a double check after the release would be helpful, just to make sure we didn't break anything.
What do you think if I do the release tomorrow sometime? Or would it be better to wait for the next week?
What do you think if I do the release tomorrow sometime? Or would it be better to wait for the next week?
Just want to warn users of the external workspace manager plugin of a problem since Jenkins LTS 2.150.x.
If you write code like this:
If you write code like this:
exws(ws) { stage() { ...} } then you have to switch it around to stage() { exws(ws) { ...} } because the stage appears to reset the workspace path.
Same for
withEnv and withCredentials.
It appears to be related to a change in the core, since Jenkins LTS 2.138.x does not have the problem.
@alexsomai
Hi Martin @martinda. Are you asking:
- why I did the upgrade?
- or if I have any issues on that particular PR?
The answers:- I did the upgrade because jenkinsci/run-selector-plugin#22 would not work otherwise. Not very sure if this is really needed or not actually.
- I don't have any other issues on this PR (number 23), the upgrade seems to work, but I was waiting for at least 1 thumbs up from someone within the Jenkins community.
and btw, with regards to what you said previous days, about the compatibility issues between
exws() and stage(). Do you think we should do something about it in the plugin? Or just something to be aware of?
Yes I think something needs to be done. See https://groups.google.com/forum/#!topic/jenkinsci-dev/TXBLRt5ZxjI
I started working on it but I have not tried the “BOM” yet.
But some of the clean up tests have become flaky. I posted to the mailing list about it.
node('dockerserver') {
withCredentials([
usernamePassword(
credentialsId: 'svc',
usernameVariable: 'SVC_AUTHENTICATION_USR',
passwordVariable: 'SVC_AUTHENTICATION_PSW'
),
string(
credentialsId: 'bb',
variable: 'BB_PAT'
),
string(
credentialsId: 'nlt',
variable: 'NPM_LOCAL_TOKEN'
)
]) {
withEnv([
"HOME=${workspace}"
]) {
exws(extWorkspace) {
docker.image('node-builder')
.inside('--tmpfs /.config --ipc=host', cl)
}
}
}
}
Hi @JoA-MoS I do not use docker. Maybe you have already seen this page it explains how to allocate, then share a workspace. In your code example, where do you define the value of
${workspace}? I guess that might be part of your question as well. You should try to allocate a workspace first with exwsAllocate, then you can use it with exws. Hope this helps...
Thank @martinda - I am allocating the workspace in an earlier step but you did identify the problem ${workspace} is the default jenkins job workspace not the
exws is there a way to get the computed exws I would like home to be set to the same value as the exws path ? If I hardcode HOME = /mount-point/job/path/12 my pipeline works so if I can just get this from exws. I tried extWorkspace.pathOnDisk but got an error that the method isn't allowed to be called by a script.
how does it handle locking and resource contention which was a known issue with regular workspaces, and the reason they had to employ the dynamic workspaces with workspace@1 and workspace@2 ?
also, does this work across multiple agents?
in general it seems very hard to orchestrate a shared folder across multiple jobs all trying to read/write at the same time. Supporting multiple hosts would be yet another major layer of complexity.
Hi @solvingj Yes, the plugin works across multiple agents, as long as all of them mount the same disk (via NFS, for example). Please make sure that you have the prerequisites set-up correctly before running the plugin.
Now, related to potential multithreading issues that many agents may read/write the same resource - I have to say I didn't have any problems with that. The plugin doesn't do anything special to handle these cases, but I guess there is some resource locking and handling at OS level. Moreover, it depends how you use the plugin. The common approach, I think, is to have one master Jenkins job that writes on the shared disk, then it chains other downstream Jenkins job that reuse the same workspace, but don't necessary run in parallel.
Hello.
Why the following doesn't work I can't figure out:
def extWorkspace = exwsAllocate "diskpool1"
node('R584_CrossTests(LSB_x86)_Debian.9.8-i686') {
exws (extWorkspace) {
sh 'ls -la'
sh 'touch test1.txt'
sh 'ls -la'
}
}
node('R584_CrossTests(LSB_LicenseFree_x86)_Debian.9.8-i686') {
exws (extWorkspace) {
sh 'ls -la'
sh 'touch test2.txt'
sh 'ls -la'
}
}I get:
Disk allocation strategy was not provided as step parameter. Fallback to the strategy defined in the Jenkins global config
Using Disk allocation strategy: 'Select the Disk with the most usable space'
Selected Disk ID 'disk1' from the Disk Pool ID 'diskpool1'
The path on Disk is: folder1/for_test/Fazlyev/upstream/85
[Pipeline] node
Running on R584_CrossTests(LSB_x86)_Debian.9.8-i686 in /home/tester/workspace/for_test/Fazlyev/upstream
[Pipeline] {
[Pipeline] exws
Searching for disk definitions in the External Workspace Templates from Jenkins global config
Searching for disk definitions in the Node config
Running in /mnt/disk1/folder1/for_test/Fazlyev/upstream/85
[Pipeline] {
[Pipeline] sh
- ls -la
итого 8
drwxr-xr-x 2 root root 4096 июл 17 09:32 .
drwxr-xr-x 14 root root 4096 июл 17 09:32 ..
[Pipeline] sh - touch test1.txt
[Pipeline] sh - ls -la
итого 8
drwxr-xr-x 2 root root 4096 июл 17 09:32 .
drwxr-xr-x 14 root root 4096 июл 17 09:32 ..
-rw-r--r-- 1 root root 0 июл 17 09:32 test1.txt
[Pipeline] }
[Pipeline] // exws
[Pipeline] }
[Pipeline] // node
[Pipeline] node
Running on R584_CrossTests(LSB_LicenseFree_x86)_Debian.9.8-i686 in /home/tester/workspace/for_test/Fazlyev/upstream
[Pipeline] {
[Pipeline] exws
Searching for disk definitions in the External Workspace Templates from Jenkins global config
Searching for disk definitions in the Node config
Running in /mnt/disk1/folder1/for_test/Fazlyev/upstream/85
[Pipeline] {
[Pipeline] sh - ls -la
итого 8
drwxr-xr-x 2 root root 4096 июл 17 09:32 .
drwxr-xr-x 14 root root 4096 июл 17 09:32 ..
[Pipeline] sh - touch test2.txt
[Pipeline] sh - ls -la
итого 8
drwxr-xr-x 2 root root 4096 июл 17 09:32 .
drwxr-xr-x 14 root root 4096 июл 17 09:32 ..
-rw-r--r-- 1 root root 0 июл 17 09:32 test2.txt
[Pipeline] }
[Pipeline] // exws
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
Why doesn't the second agent see the file created by the first agent?
Hi @ainur-fa, based on the console output, it looks correct. I just want to double check, did you set-up correctly the prerequisites, especially the
Node Properties: https://github.com/jenkinsci/external-workspace-manager-plugin/blob/master/doc/PREREQUISITES.md#node-properties? And please make sure that the mounts are correctly working. Can you list the same files when accessing the mnt path on your nodes?
5 replies
I wrote pipeline script this way, but there was an error(" cannot find '.' ") in Test stage.