Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jun 11 07:14
    uellue demilestoned #1047
  • Jun 11 07:14
    uellue milestoned #1047
  • Jun 11 07:05
    uellue commented #922
  • Jun 10 15:28
    sk1p commented #1045
  • Jun 10 14:43

    sk1p on gh-pages

    Deploy docs (compare)

  • Jun 10 14:43

    sk1p on continuous

    (compare)

  • Jun 10 14:43

    sk1p on continuous

    (compare)

  • Jun 10 14:16

    sk1p on master

    Post-release changelog update … Update release checklist (compare)

  • Jun 10 14:16
    sk1p closed #1047
  • Jun 10 14:16
    sk1p closed #985
  • Jun 10 14:16
    sk1p commented #1047
  • Jun 10 14:10
    sk1p edited #1047
  • Jun 10 14:09
    codecov[bot] commented #1047
  • Jun 10 14:09

    sk1p on gh-pages

    Deploy docs (compare)

  • Jun 10 14:06

    sk1p on continuous

    (compare)

  • Jun 10 14:06

    sk1p on continuous

    (compare)

  • Jun 10 14:06
    codecov[bot] commented #1047
  • Jun 10 14:04
    codecov[bot] commented #1047
  • Jun 10 14:02
    codecov[bot] commented #1047
  • Jun 10 14:01
    codecov[bot] commented #1047
Alexander Clausen
@sk1p
So from sender's perspective, it's just fire-and-forger
Anand Baburajan
@AnandBaburajan
So we don't need to take care of reliablity?
Alexander Clausen
@sk1p
Not end-to-end - the sending side should still be reliable insofar that if you have a "well behaved" receiver, it should be able to receive the full data stream (you can emulate that "well behaved receiver" with wireshark or other packet capture software)
Anand Baburajan
@AnandBaburajan
Ah, got it! But then how do we check "correctness: the replayed data is being sent exactly as it should be, without dropping packets" without having end-to-end reliability?
Alexander Clausen
@sk1p
@AnandBaburajan technically, it's possible that the OS could drop packets even from the send buffer - so the idea is to make sure that doesn't happen
Anand Baburajan
@AnandBaburajan
Okay, thanks!
Lizhen Chen
@LizhenChen10_twitter
hello. I have a .h5 files
how to open it and analyze the data
Dieter Weber
@uellue
Hi @LizhenChen10_twitter, welcome! :-) Is it 4D STEM data?
Lizhen Chen
@LizhenChen10_twitter
yes. \
is there any link to download the LiberTEM ?
Dieter Weber
@uellue
Ok, so we can use the GUI to get started. Did you already install LiberTEM?
Lizhen Chen
@LizhenChen10_twitter
not yet \
Dieter Weber
@uellue
I see. We have installation instructions here: https://libertem.github.io/LiberTEM/install.html
Operating system is Windows?
Lizhen Chen
@LizhenChen10_twitter
yes. I saw this website. But I cannot find the botton of download
Dieter Weber
@uellue
Ah I see. Since it is a Python-based software, it uses a Python package manager to install, so no download of a .exe
Do you already have Python on your computer?
Lizhen Chen
@LizhenChen10_twitter
I dont have. Maybe I need to install Python first
Dieter Weber
@uellue
Yes! :-) This one works well on Windows: https://docs.conda.io/en/latest/miniconda.html
Once Conda is installed, you can start the "Anaconda prompt", create a new virtual environment conda create -n libertem python=3.8, then activate with conda activate libertem, and finally pip install libertem
Dieter Weber
@uellue
To run it, open the Anaconda prompt, activate the LiberTEM environment conda activate libertem, and then just run libertem-server. That should open a browser window with the GUI. Here's more info on that: https://libertem.github.io/LiberTEM/usage.html
Lizhen Chen
@LizhenChen10_twitter
thank you so much. I am trying
Dieter Weber
@uellue
:thumbsup: :smile:
Dieter Weber
@uellue
Is it working so far? Let me know if I can help!
Anand Baburajan
@AnandBaburajan

Hi! I've some doubts regarding the K2 sim.

  1. In the pcap file, I can see that the packets are transmitted over IP multicast. I understand that multicast is used to send the data to multiple receivers, for redundancy, for monitoring, etc. So may I know know why multicast is used by the K2 camera? (So that I can think if the sim needs to be multicast.)

  2. Would the sim and the receiver be run on the same host or different hosts? If multicast isn't required and both the sim and receiver run on the same host, I think we can just use localhost as the MTU of the loopback network interface is 65536 bytes, which would help achieve fast speeds. If multicast is required or both the sim and receiver need to be run on different systems, I would like to know if the systems' MTU is large enough to support jumbo frames. (Because the sim would be more prone to packet loss if the packets need to be fragmented.)

  3. Should I consider using C to build the sim? Or would Python suffice if the performance, error, etc., requirements are met with it?

Please answer whenever you're free, thanks!

I also have a prototype to share (with Python using unicast on localhost):
https://gist.github.com/AnandBaburajan/ab1777a646b0a8567fd1d6ce4eb542db

Alexander Clausen
@sk1p
Hi Anand, for point 1, yes, multicast should be used, because this allows us to keep the receiver side 100% unchanged for "real" workload and simulated workload
For 2, we should support both, but usually it would run on the same host. MTU can be made large enough for jumbo frames - I'm using veth currently in testing, which works like the real hardware in respect to MTU (see make_veth.sh in the private repo for how a pair of veth is created). Not sure about fragmentation - did you test if this actually causes a problem?
For point 3, Python would be ok if it works out performance-wise. You can also think about using Cython if certain parts prove to be bottlenecks
Alexander Clausen
@sk1p
Thanks for sharing the prototype, I'll see if I can try it this or next week. Let me know if you need specific feedback or are blocked on something!
Anand Baburajan
@AnandBaburajan

about fragmentation - did you test if this actually causes a problem?

Not yet, but I read that it's best to have the packet size < MTU to avoid fragmentation, which can cause problems if there's packet loss. Thanks for replying, I'll read more about veth.

Alexander Clausen
@sk1p
Yeah, as far as I understood, the concern about fragmentation is mostly because in "real" multi-hop networks, you cannot be sure that fragmentation works well (as the individual hops can have different MTU) - in this case, we are directly connected to known-working hardware, and only have the switch between the data source and sink
Anand Baburajan
@AnandBaburajan

Hi Anand, for point 1, yes, multicast should be used, because this allows us to keep the receiver side 100% unchanged for "real" workload and simulated workload

Okay, can you also share why/how the real K2 camera uses multicast? :D

Alexander Clausen
@sk1p
My guess would be to simply allow more than one receiver - for example, for monitoring the data stream - this would be difficult with unicast, because you would need to use a special monitoring port/configuration on the switch etc. But as I didn't build the camera, I cannot say this with 100% certainty :)
Anand Baburajan
@AnandBaburajan
Got it, thanks! :D
Matthew Bryan
@matbryan52
Hi everyone, could I confirm that the minimum Python version is absolutely 3.6, and not '3.6 with some support for <3.6', and so syntax/library changes such as f-strings or using pathlib are OK ?
Dieter Weber
@uellue
Yes, strictly 3.6+. We use f-strings already. :-)
Matthew Bryan
@matbryan52
Great, thanks. By the way I was able to successfully run the live plotting notebook (once I found a suitable dataset), except for the gridded display which I am sure is just an environment issue.
Dieter Weber
@uellue
Great! :-) Did you encounter any issues where more or better documentation would have helped?
Matthew Bryan
@matbryan52

In the flow of the notebook, no, it is clear. I think the user difficulty will come from correctly applying custom plotting (particularly for the callable channel arg to LivePlot2D). The number of ways the plotskwarg to run_udf can be overloaded is also a potential source of user confusion, it's documented in the API reference but it's pretty deep in there!

I haven't thought long enough about it or checked out the backend for plotting yet, I will keep thinking and perhaps come up with a custom example.

The only issues I had were the mundane sort (obscure error messages when failing to load data), for which I might make a PR to improve,
FileNotFoundError: [Errno 2] No such file or directory: 'h'
had me stumped for a moment until I realised it was the first letter of home/... !!! Just need to add some checking before indexing
Dieter Weber
@uellue
Thx for the feedback! :-)
Alexander Clausen
@sk1p
FYI we just released LiberTEM version 0.7.0 - https://github.com/LiberTEM/LiberTEM/releases/tag/v0.7.0
Dieter Weber
@uellue
:party:
Where are the emojis? :-D
:tada:
Thx @sk1p :-)
Anand Baburajan
@AnandBaburajan
:D
Matthew Bryan
@matbryan52
Well done! :)