by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Aug 09 18:38
    stellargo review_requested #57
  • Aug 09 18:38
    stellargo review_requested #57
  • Aug 09 18:38
    stellargo review_requested #56
  • Aug 09 18:36
    stellargo review_requested #14
  • Aug 09 18:36
    stellargo review_requested #15
  • Aug 09 18:24
    stellargo synchronize #56
  • Aug 09 18:24
    stellargo synchronize #56
  • Aug 09 18:24
    stellargo synchronize #56
  • Aug 09 18:22
    stellargo synchronize #16
  • Aug 09 18:15
    stellargo review_requested #16
  • Aug 09 18:14
    stellargo synchronize #16
  • Aug 09 17:40
    stellargo edited #16
  • Aug 09 17:39
    stellargo review_requested #16
  • Aug 09 17:39
    stellargo opened #16
  • Aug 09 14:24
    stellargo unlabeled #12
  • Aug 09 14:19
    stellargo labeled #57
  • Aug 09 14:19
    stellargo reopened #56
  • Aug 09 14:17
    stellargo closed #56
  • Aug 09 14:16
    stellargo review_requested #57
  • Aug 09 14:16
    stellargo opened #57
stellargo
@stellargo
I can see redis-fingerprint-storage.hpi here : https://updates.jenkins.io/experimental/latest/
Would that mean it came through?
Oleg Nenashev
@oleg-nenashev
@stellargo yes, it means that the release was successful :+1: I was just surprised to not see commits in the master branch. Maybe I missed them
stellargo
@stellargo
So I was thinking how fingerprint cleanup would work in a system with external storage configured, as well as some old fingerprints lying around on disk. In such case, we can either:
  • Run the cleanup ONLY on external storage. Gradually some fingerprints would get migrated when used, but others would remain there for eternity.
  • Run the fingerprint cleanup on both external and file system storage. But here, on the first run of fingerprint cleanup, the entire fingerprint base would get migrated in one go.
5 replies
Would love some thoughts and opinions :)
@oleg-nenashev @afalko @mikecirioli
According to me second option looks better
stellargo
@stellargo
I was looking into dealing with passing a virtual file to a method which needs java.io.File (think SaveableListener). I found https://stackoverflow.com/questions/4317035/how-to-convert-inputstream-to-virtual-file which suggests basically creating a temp file on disk. One interesting answer is https://stackoverflow.com/a/17595282/8461040 (first comment on this) which suggests using an Apache Commons VFS RAM file as a way to have an in memory file. But it seems like it cannot be converted to a java.io.File as mentioned here : https://stackoverflow.com/questions/58140765/use-apache-commons-vfs-ram-file-to-avoid-using-file-system-with-api-requiring-a
I am not sure if creating a temporary file on disk is the right answer because that creates dependence on disk @oleg-nenashev @afalko @mikecirioli
Oleg Nenashev
@oleg-nenashev
@stellargo I doubt it is a right answer for 'SaveableListener'. We used temporary files to retain binary compatibility for Artifact and Log APIs, but in those cases this was a temporary solution. File-based APIs were deprecated. For SaveableListener such approach may work as well, but we never know whether the file would be modified by the listener. So such workaround may require you to also check tmo files, and propagate changes to the storage. It is doable, but a lot of overhead. IMHO it is better to start designing for non-File saveable objects
1 reply
I can take this topic for the next coding phase, IMHO it makes sense to design the new API together
VFS basically requires switching to another API as well. Path API from Java 7+ is more friendly to cirtualization, but also not ideal
2 replies
stellargo
@stellargo
Up for review: jenkinsci/jenkins#4817
Oleg Nenashev
@oleg-nenashev
@stellargo Looks like there are still issues with opengraph
stellargo
@stellargo
stellargo
@stellargo
Created jenkinsci/jenkins#4825 for migration, currently WIP
stellargo
@stellargo
Up for review: jenkinsci/jenkins#4817
Oleg Nenashev
@oleg-nenashev
@stellargo Hi. I assume that we do need a dry-run session for the demo, right? The previous one at the Cloud Native SIG was pretty good. Please let me know if you want to have another dry-run
stellargo
@stellargo
@oleg-nenashev as you would suggest I am fine with. I had incorporated the suggestions you had shared with me in the document after the cloud native sig demo. Rest will be pretty much the same from my side because that demo marked everything we did in phase 1. If you feel that was fine, then maybe we don't need a dry run
Oleg Nenashev
@oleg-nenashev
Yeah, I think it is fine. I am happy to do a dry run session if needed, but IMO we could use more time for coding and other stuff :_
stellargo
@stellargo
Yes, Awesome :)
Oleg Nenashev
@oleg-nenashev
stellargo
@stellargo
Woah!
I just went down the path of history of Redis after reading this :P
stellargo
@stellargo
stellargo
@stellargo
Also jenkinsci/redis-fingerprint-storage-plugin#23 (but this depends on 4817 to actually get merged)
Oleg Nenashev
@oleg-nenashev
Looks like both me and @afalko are quite exhausted after vacation :)
2 replies
stellargo
@stellargo
Created jenkinsci/jenkins#4834 for new way of configuring fingerprints
Oleg Nenashev
@oleg-nenashev
:+1: will try to review today
stellargo
@stellargo
So I am using hetero-radio (why? only one fingerprint storage should be allowed to be able to configured at a given time) to configure the fingerprint configuration using descriptors. Does this persist the data? because it seems like it doesn't
6 replies
stellargo
@stellargo
(what I am trying to do -> every storage now has a descriptor. Using the descriptor, we should be able to configure a storage)
Oleg Nenashev
@oleg-nenashev
I will have to drop after 30 min today. Got an unexpected meeting at CDF I need to attend. Sorry for the late notice
stellargo
@stellargo
No worries :)
michael cirioli
@mikecirioli
currently in another meeting, but i expect it to end in 10-15 minutes and will join then
stellargo
@stellargo
Is it possible to decouple the DescriptorImpl from the class it describes into another file? I don't want file sizes to become huge
Oleg Nenashev
@oleg-nenashev
@stellargo yes, it is possible
stellargo
@stellargo

Subject: Ideas Discussion: Tracing fingerprints (artifacts, credentials, docker images, etc.) across Jenkins Instances!

Hi Everyone!

I would like to start this thread to discuss some opinions and new ideas with the community regarding fingerprints and tracing.

We have been working on the Jenkins link:[fingerprint] engine, and we have made great progress so far. We have been able to make the Jenkins fingerprinting engine 'pluggable', which basically means that now the fingerprints created by Jenkins can be stored in the cloud, away from the local disk storage of Jenkins master. Currently we only have one link:[plugin] which can support this, which is powered by Redis.

One of the advantages we got out of this project was the ability to track fingerprints across instances. What this can help in achieving is tracking usage of artifacts, credentials, docker images, etc. across instances of Jenkins. We can leverage our current project to provide such functionalities. But doing so only makes sense if there is some need for it. Also, there can be other potential use cases that we may have missed. We would love to discuss ideas around this, which can benefit the Jenkins community.

More information about the project can be found on the link:[project page] and link:[blog].

We would deeply appreciate suggestions and comments on this topic from the developer community, and potential areas where you think this project can help you or other Jenkins users.

Best Regards,
Sumit

@oleg-nenashev @afalko @mikecirioli I will post this on the dev mailing list tomorrow evening to get more reach on a Monday. If there are any suggestions do let me know :)
Oleg Nenashev
@oleg-nenashev
@stellargo looks great!
Oleg Nenashev
@oleg-nenashev
@stellargo Hm? According to my calendar, we have 36 minutes before the meeting starts :)
stellargo
@stellargo
Oh! I am so sorry :P I got confused with Thursday timings
Updated :P
stellargo
@stellargo
In today's meeting we discussed the PostgreSQL plugin and it's implementation. There are two aspects that are important from a design perspective:
  • We need ORM to convert the fingerprints into JSON and not XML. Why? This will help us save the data as jsonb inside postgres and allow querying it directly from the database based on the use case. Example: searching for container ids from docker traceability plugin across jenkins instances. XStream seems to have a driver for JSON, but I am not aware about any other plugin that may have tried this so we can have a precedent.
  • Whilst postgres would allow query this jsonb documents, does it make sense to define a partial schema for the fingerprints? Whilst we can do that for some part of fingerprint data, we cannot do that for FingerprintFacet as facet is unstructured, schemaless and depends on the plugin that created it. So do we store the entire fingerprint as a blob of JSON, or we just store the facets as blobs. In the latter approach, we can index the other fields of fingerprint like md5, but I am not sure how much of an advantage that offers.
8 replies