alibaba/Dragonfly

Dragonfly is an intelligent P2P based file distribution system

Allen Sun

@allencloud

@tirupatibg Could you attach your problem within GitHub issue? https://github.com/dragonflyoss/Dragonfly/issues

Actually we are using GitHub to communicate with each other. Thanks a lot.

tirupatibg

@tirupatibg

Sure @allencloud , thank you!

wangwj

@wangwj

各位，谁知道server，clinet的空间如何规划，找了一圈说是每个几分钟会清理文件，是清理哪的文件

xzy256

@xzy256

@wangwj server 目前是不会清理文件的，后续会支持定时清理， client端是默认3分钟清理。supernode默认使用的目录是/root/admin/supernode, 下载的piece放在repo目录下，dfclient默认使用/root/.small-dragonfly，清理的是data。目前版本0.4.3可以使用 --expiretime 来自定义时间

DMF Lists

@dmflists_twitter

👋was hoping to try out dragonfly but it won't work in our env since we use gcr for our images and 3rd party registries aren't supported. However, I noticed containerd does support mirrors for 3rd party registries. Has anyone tried dragonfly with containerd?

xiaoxiaoye

@xiaoxiaoye

问下，如果要做集群内所有主机的镜像预热功能，现在支持吗？

Ben Ye

@yeya24

@dmflists_twitter Hi, I think this is a good question. Can you open an issue in Github?

sunny-zyl

@sunny-zyl

请问下，这个安装server的方法，是否不适用最新的release？
文档需要更新么

https://d7y.io/zh-cn/docs/userguide/install_server.html

docker和physical的方法，都不好用

Inno Jia

@kobeHub

我有一个问题，在使用dfget下载镜像的时候，如何查看镜像来自于哪一个peer呢？

Ben Ye

@yeya24

@sunny-zyl 官网上面的文档应该都不是最新的，还是参考github上面的

ZhiQiang Gu

@yunfeiyanggzq

@kobeHub 你要进到dfget的日志里面查看

jessiezhou0424

@jessiezhou0424

想用harbor+k8s+dragonfly

但是harbor integration和kubernetes intergration文档都是空的。。。

Niki Chr

@NikiChr

Is there an option to disable back source downloading when using docker pull similar to the --notbs option when using the dfget command?
Maybe a line you can add to the dfget.yml?

#998

Иван Сердюк

@oceanfish81_twitter

Hi there

huddhudd

@huddhudd

我想用于，上万台网吧机器做游戏等分发。

适合这样的场景吗？

@tjjh89017

Date Huang

@tjjh89017

@huddhudd Hi
你需要的是什麼樣的分發?

huddhudd

@huddhudd

@tjjh89017 做游戏分发，没有端口映射，可能需要打洞

Date Huang

@tjjh89017

@huddhudd 你可以研究看看clonezilla能不能用
我之前做的是deploy OS用的，不是遊戲分發甚麼的hmm

Allen Sun

@allencloud

@huddhudd 要不你直接在github上提交issue沟通吧？我们最高优先级响应的是github issue。谢谢。https://github.com/dragonflyoss/Dragonfly/issues

Chinmay-Gurjar

@Chinmay-Gurjar

Hello, this is Chinmay, I would love to cotribute to dragonfly, are there any newcomers issues that I can look at ?

also is this the right channel to discuss this ?

Allen Sun

@allencloud

I am afraid that we have almost dropped the gitter chatting. If you wish to get realtime feedback, I think github issue or Dingtalk is the direct way you could use. @Chinmay-Gurjar Thanks.

CerebroWang

@cerebrowang

没时间弄这个了，先行撤退，谢谢诸位辛苦付出

babywaiting

@babywaiting

有钉钉群的加入方式吗？

主页上的钉钉群二维码失效了

babywaiting

@babywaiting

求助

Allen Sun

@allencloud

@babywaiting 钉钉群号：23304666

Cole Mickens

@colemickens

Hi. Can I join DingTalk from a browser or do I have to use an app?

shayubit

@shayubit

Hello everyone!

Allen Sun

@allencloud

Hi. Can I join DingTalk from a browser or do I have to use an app?

I think app is better

Hello everyone!

bmmadhuprasad

@bmmadhuprasad

HI Team

I have deployed Harbor v2.0 and dragonfly latest version trying to use harbor p2p image distribution to dragonfly but getting below when applied the preheat policy to preheat image
Check preheat progress: Task is failing and distribution is not happening

2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:93]: Preheating image 'dragonfly/test:0.1@sha256:54c9b521ae50156029ea48394041885f33414c1dad2a5deefcaa46ddf22079a3' to the target preheat provider: dragonfly newdragon:http://dragonflynew-supernode.dfn.svc.cluster.local:8002
2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:120]: Get preheat provider driver: dragonfly
2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:134]: Check health of preheat provider instance: Healthy
2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:146]: Sending preheat request is successfully done
2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:170]: Start to loop check the preheating status until it's success or timeout(30m)
2021-03-31T13:19:49Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z
2021-03-31T13:19:59Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z
2021-03-31T13:20:09Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z
2021-03-31T13:20:19Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z
2021-03-31T13:20:29Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z

Below is logs from dragonfly

2021-03-31 14:17:50.496 DEBU sign:9 : GET /_ping remote:192.168.71.183:40028 cost:2.901µs err:
2021-03-31 14:17:50.498 DEBU sign:9 : create preheat: Type[image] URL[https://harbor.madhu.mwpdemos.com/v2/test/mem/manifests/0.1] Filter[] Identifier[] Headers[map[Authorization:Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjdXTUE6S1ZYTzpLRjI1OjM1TkE6R0ozNTpaR0tZOlBHWVQ6STVVVDpUR1dZOlVVSTQ6UFQ3WjpYM0tXIiwidHlwIjoiSldUIn0.eyJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE2MTcyMDIwNjcsImp0aSI6Imc3TFhnUmJidmkzOFV4bFUiLCJpYXQiOjE2MTcyMDAyNjcsImlzcyI6ImhhcmJvci10b2tlbi1pc3N1ZXIiLCJuYmYiOjE2MTcyMDAyNjcsInN1YiI6ImRpc3RyaWJ1dG9yIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L21lbSIsImFjdGlvbnMiOlsicHVsbCJdfV19.DFeugff4V8VplCOGrW1AqAUhUTRo8F4pCwPAKa06zP8A67Euls4GC6MidfXpzAazU5QndtHBMkKdh-CBAUgzKLv73F3Kz1tdhXBAcyGXJHX7r8otQ4J8wNf0TwOHesmoW1sKc0IUBmlsA_a4fXR_7DAspzJmFiBchNin5qGCvhvn6449Tr_kF5HyoknOrw1NoK5_mG0mcBWLSC6E_16P1YIeAWaisfbuPurfnd-Zs2MLfSltWDjgibPXU02csiBmfGgy1mES0icDKkgeuqiLlFz90pgV61cZcQieqAeB5k3quMN20HWC3EaYTqPyspo0ZMjc_w1WItldMFDusncH8kqak7iKfQYSISf2MlD7WSmHO77_jgePWsEYKvdQIuLOcY-PZ-dswISb1DE72hJ6ZvNEnOna7LKe67VfFgfBvar4UEtwCzMCyJt0aAUzepgvFbdA1ajUQagpeAlco5o-oZmvmbfay_xiBRxEsBa0n9KIKrxQ3ceLhx5xCFau3OH9Poa9ch2QeMI7bYe5BAkL2xQKzZdk4LNkG6hpQtucLNlCzMTOZ_HO7E4lFXe2LQlYcNy6XSSv16g1R5qjM2W4UBf_pmGGo1E4wTC0WSVlgIXdlCkVo8P1h6xfVhmOfnZWiZgVmPynR21urj1pW-M0FEbarBefU-JTvQU4Qgt25Q0]]
2021-03-31 14:17:50.498 DEBU sign:9 : POST /preheats remote:192.168.71.183:40028 cost:193.662µs err:
2021-03-31 14:18:00.502 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:33.646µs err:
2021-03-31 14:18:10.499 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:24.602µs err:
2021-03-31 14:18:20.499 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:37.729µs err:
2021-03-31 14:18:30.500 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac44

even i have deployed dragonfly supernode in VM directly from source code without using docker as per document and tried to use that as end point in harbor but i see same error when i created p2p preheat policy and apply to initiate image preheat to dragonfly

bmmadhuprasad

@bmmadhuprasad

Below is the error i see from harbor

2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:93]: Preheating image 'test/mem:0.1@sha256:9871d56e046e38e9daaef6b592e67282c1e286a85e84797ed18b942a8263eadf' to the target preheat provider: dragonfly vm:http://54.215.243.61:8002
2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:120]: Get preheat provider driver: dragonfly
2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:134]: Check health of preheat provider instance: Healthy
2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:146]: Sending preheat request is successfully done
2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:170]: Start to loop check the preheating status until it's success or timeout(30m)
2021-04-05T13:49:10Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:49:20Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:49:30Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:49:40Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:49:50Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:50:00Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:50:10Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:50:20Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:50:30Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:50:40Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:50:50Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:51:00Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:51:10Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:51:20Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:51:30Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:51:40Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:51:50Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:52:00Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check p

below is the logs from dragonfly supernode

1ZYTzpLRjI1OjM1TkE6R0ozNTpaR0tZOlBHWVQ6STVVVDpUR1dZOlVVSTQ6UFQ3WjpYM0tXIiwidHlwIjoiSldUIn0.eyJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE2MTc2MzIzMzksImp0aSI6IkNobzlxcXFhMmFGTWJKeDMiLCJpYXQiOjE2MTc2MzA1MzksImlzcyI6ImhhcmJvci10b2tlbi1pc3N1ZXIiLCJuYmYiOjE2MTc2MzA1MzksInN1YiI6ImRpc3RyaWJ1dG9yIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L21lbSIsImFjdGlvbnMiOlsicHVsbCJdfV19.RXfGEpeMUpNYQA2v2F60IPSnSy4h_X4WKFEIlYAd0H363oVE5omj25faYccMBEU3XFPeFUWx2tGaHCdrbgnSa5SnQ5qJgerJ4FmGYCNgi1Pvr-eFjEjC8hqNAREoiZ25xsF4lg90t7-YQ_UZSxgmwMKQNRfaREseJw1emTXDLAulJSxzJVO6egEt2_jWAWsK58zyTBu40zTFGwD6fUr27vAXn2xf3KSPETf4Hf0HunDpzoZFTqMqAmHSp5w3loPuHvGF9pt9f4_blkT1JULmFFc3_zcSkuF-NDcqT2UAxN7ZQHaYJpG6uhuQmEc0bJ8fR69v1Ag_CKppOzUZbl0fGVMEBn0PDJLn67leEo5UwBn-knydMtPozfYC1fg20PxVAASo9i964ZS1jkbOoDfjpG5QKf-al5DgaCge8O6ULFo2PKPUZaip-6opnpcdG65tRAP64UZvCCYBZ8l1maecZHK2MLgQYiD5lGInfjcUWEYaENtw4FXrRltRteUiv1GYN8jBktuoAKZevLsia0ULMhayenlv3u999yAnLnAgjGaYijNWv6r2IMeO6mFnI6AZSzlKH2ig3N25gPbxh1UT7PC7jd-mJEuxVd_BBMFhS2BttRbLqyJf6Bkr5T4gKOYyVjAm0tBG0zdhulrjakAt97BW6v3IFbP2cI0aL1-vk"},"type":"image","url":"https://harbor.madhu.mwpdemos.com/v2/test/mem/manifests/0.1"}, res:{"body":{"id":"3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735"},"headers":{},"statusCode":"OK","statusCodeValue":200}
2021-04-05 13:49:00.958 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:c1c1ba809094891c97d8ca9f34817f54c72cfab868b6914d85f2a156a6ac26a9 parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735
2021-04-05 13:49:00.969 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:ff59de7222ecb2f8dd85ac954ca687d3b63b726d7a9631964f00535e65339584 parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735
2021-04-05 13:49:00.970 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:79f39971f65f5d879d2ed9e4402b5fba9a825803c4c76b8a09e4926ea58185c2 parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735
2021-04-05 13:49:00.973 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:e72295e4b7adb87d61408b4101c0e4f8ccc15c84ccbfdbfe756e410f83b0fcaf parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735
2021-04-05 13:49:00.974 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:e38d1a0197a41b27b23313a6ae0ec7f4219c31fdb55e5f5b9c43ee781237312f parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735
2021-04-05 13:49:00.980 INFO [preheat-12] c.d.d.s.s.p.PreheatServiceImpl - command: /usr/local/bin/dfget -u https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:ff59de7222ecb2f8dd85ac954ca687d3b63b726d7a9631964f00535e65339584 -o /home/admin/supernode/repo/preheat/87626317-ef0f-448a-8a6e-78e60f72cc6c --callsystem dragonfly_preheat --totallimit 90M -s 90M --notbs --header Authorization:Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjdXTUE6S1ZYTzpLRjI1OjM1TkE6R0ozNTpaR0tZOlBHWVQ6STVVVDpUR1dZOlVVSTQ6UFQ3WjpYM0tXIiwidHlwIjoiSldUIn0.eyJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE2MTc2MzIzMzksImp0aSI6IkNobzlxcXFhMmFGTWJKeDMiLCJpYXQiOjE2MTc2MzA1MzksImlzcyI6ImhhcmJvci10b2tlbi1pc3N1ZXIiLCJuYmYiOjE2MTc2MzA1MzksInN1YiI6ImRpc3RyaWJ1dG9yIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L21lbSIsImFjdGlvbnMiOlsicHVsbCJdfV19.RXfGEpeMUpNYQA2v2F60IPSnSy4h_X4WKFEIlYAd0H363oVE5omj25faYccMBEU3XFPeFUWx2tGaHCdrbgnSa5SnQ5qJgerJ4FmGYCNgi1Pvr-eFjEjC8hqNAREoiZ25xsF4lg90t7-YQ_UZSxgmwMKQNRfaREseJw1emTXDLAulJSxzJVO6egEt2_jWAWsK58zyTBu40zTFGwD6fUr27vAXn2xf3KSPETf4Hf0HunDpzoZFTqMqAmHSp5w3loPuHvGF9pt9f4_blkT1JULmFFc3_zcSkuF-NDcqT2UAxN7ZQHaYJpG6uhuQmEc0bJ8fR69v1Ag_CKppOzUZbl0fGVMEBn0PDJLn67leEo5UwBn-knydMtPozfYC1fg20PxVAASo9i964ZS1jkbOoDfjpG5QKf-al5DgaCge8O6ULFo2PKPUZaip-6opnpcdG65tRAP64UZvCCYBZ8l1maecZHK2MLgQYiD5lGInfjcUWEYaENtw4FXrRltRteUiv1GYN8jBktuoAKZevLsia0ULMhaye

Please help us to fix the issue and let us know anything i am missing

bmmadhuprasad

@bmmadhuprasad

can anyone please look into the below issue i have raised in dragonfly repo and help us to correct it

dragonflyoss/Dragonfly#1537

dbbhat

@dbbhat

Hi Team, I have raised the below issue yesterday-
dragonflyoss/Dragonfly#1539
We have a security requirement to have all communication between dfget client, dragonfly supernode and dfget server with TLS support. And I see that dfget server currently does not have support for this.

Can someone please help look at the issue and provide any thoughts/reasoning behind this? Thanks!

Where communities thrive

People

Repo info

Activity