Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Adam Bouhenguel
@ajbouh
i think it might be overwhelming the disk. how can i avoid reading from too many different files at once?
Laurent Demailly
@ldemailly
if you only have 1 spinning disk 54MB/s sounds about right I guess and it’ll be your bottleneck if your network is gbit or faster. buy some ssd :)?
Adam Bouhenguel
@ajbouh
I can do 80MB/s when i use UDT
Same pair of machines
Same set of disks
Laurent Demailly
@ldemailly
try making a tgz first then send that?
what do you mean by 0 above btw
Adam Bouhenguel
@ajbouh
I'm trying to send ~100 files that are each 100GB
I mean that the instantaneous speed reported by wdt is sometimes 0, so it seems like some buffer or sending process is getting starved
Laurent Demailly
@ldemailly
100gb should be large enough/plenty to avoid overhead of small files seek/scan
you could try to throttle disk IOs I guess through cgroup or something but wdt just tries to read/write as fast as possible. maybe changing block size could help
Adam Bouhenguel
@ajbouh
Maybe something is throttling me only via TCP
Thanks for your help, I really appreciate it
CaptJak
@CaptJak
@ldemailly, looking to port this over to Windows (perhaps a dll?). Any suggestions on where to begin?
Laurent Demailly
@ldemailly
well I don’t know if there is a thin posix layer for windows, but you could probably run it in wsl2, otherwise you’d probably want a real port using native windows socket apis which I am afraid I don’t know anything about
CaptJak
@CaptJak
Right on WSL, it definitely works there. I'm looking to build something that's more deploy-able though, and unfortunately on multiple machines running different versions of Windows Server.
Laurent Demailly
@ldemailly
rewrite it in golang :smile: ?
CaptJak
@CaptJak
For sure is on my list of options ;)
I was hoping to maybe build it as a library, then write something in golang to use it.
But maybe it will be easier for me to rewrite it in golang, since I'm not familiar at all with C++.
Or Windows dev, for that matter, so rewriting portions of what you have using the MS socket API (whatever that is) might be more difficult.
My other options so far have been along the lines of what you suggested, WSL.
What I've gotten working so far on Windows (using you're project): Linux on Docker, Linux on Hyper-V, WSL, CentOS7/8, Ubuntu 18/20 and MacOS (High Sierra).
Gentoo is on my list for later, but so far so good.
CaptJak
@CaptJak
Maybe I'll Frankenstein it then. Reuse what I can from cpp and rewrite what I need to in golang and tie them together.
Laurent Demailly
@ldemailly
isn’t there a “redistributable” wsl2? or some way to make an installer that first install/requires wsl2 to be setup?
CaptJak
@CaptJak
Perhaps, but fwis typically the "feature" has to be enabled.
And most of the target machines in my case run OS as old as Windows Server 2008
which don't have that "feature"
But it looks like they do have a Posix subsystem that can be enabled...
WSU
I think I'll start by getting it to compile on Windows...
Besides, I think it would be awesome to have a WDT binary. Just in general.
Laurent Demailly
@ldemailly
👍
Morten V. Pedersen
@mortenvp
Hi all, seems WDT is archived on Github. Is there some alternative to use - or what is the status of it?
Laurent Demailly
@ldemailly
Archived doesn’t mean you can’t use it, this being said I am working with fb contacts to potentially move it to its own org
Davide Cavalca
@davide:cavalca.name
[m]
The WDT repo has been unarchived and is back at https://github.com/facebook/wdt
We have some housekeeping to do to resync it with our internal repo, but hopefully we'll be back in business sometime next week
Laurent Demailly
@ldemailly
awesome news!
uchihasharukh
@uchihasharukh
i have a windows machine and im a noob :P ... but i want to transfer huge sized files in windows. so is there a way to build this app in windows??
Laurent Demailly
@ldemailly
worst case you can build it under wsl but to natively use win32 api that would be quite a port
uchihasharukh
@uchihasharukh
image.png
any ideas!!
uchihasharukh
@uchihasharukh
hi how to fix the above error.??
uchihasharukh
@uchihasharukh
can anyone make a mac os build for me! :P.
uchihasharukh
@uchihasharukh
image.png
anyone knows how to solve this?? ran in ubuntu ec2
allowed ufw permissions on 46224 udp port