Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Dec 07 10:25
    ghorofamike opened #45
  • Dec 03 22:04

    andris9 on hapi

    added routing for listing authl… (compare)

  • Dec 03 16:01

    andris9 on hapi

    added basic tests for domainali… added hapi routes for POST/DELE… (compare)

  • Dec 03 14:32
    andris9 commented #361
  • Dec 03 14:02
    IndeedNotJames commented #361
  • Dec 01 21:31

    andris9 on hapi

    Added Hapi routes for /domainal… (compare)

  • Dec 01 20:53

    andris9 on hapi

    Moved change stream endpoint to… (compare)

  • Nov 30 13:41

    andris9 on hapi

    added tests for certs/dkim endp… (compare)

  • Nov 30 13:07

    andris9 on hapi

    Added API endpoints for DKIM (compare)

  • Nov 28 13:55
    R4Ajeti opened #366
  • Nov 24 00:27
    snyk-bot opened #365
  • Nov 24 00:27

    andris9 on snyk-fix-a46d8c41e55a881121e0a6492550f04f

    fix: package.json to reduce vul… (compare)

  • Nov 24 00:27

    andris9 on snyk-fix-a46d8c41e55a881121e0a6492550f04f

    (compare)

  • Nov 23 13:00

    andris9 on hapi

    Started migrating from Restify … (compare)

  • Nov 21 15:30
    louis-lau commented #364
  • Nov 21 02:31
    louis-lau commented #364
  • Nov 21 02:20
    lexparsimonet closed #364
  • Nov 21 02:20
    lexparsimonet commented #364
  • Nov 21 02:12
    louis-lau commented #364
  • Nov 21 02:10
    lexparsimonet opened #364
Louis
@louis:laureys.me
[m]
My relatives always complain about the 25mb limit that's basically everywhere, but I know that I don't want a higher limit
Daviesmolly
@Daviesmolly
How to integrate text or simple basic captcha to wildduck-webmail??
Andris Reinman
@andris9
@Daviesmolly wildduck webmail supports reCaptcha but it is disabled by default, https://github.com/nodemailer/wildduck-webmail/blob/3371984a32a7942d7859c3fcde923cf62484e7fa/config/default.toml#L48-L51
Tiny product news. WIldduck Auditing System now generates verification hashes for email downloads (each download is logged and you can later download verification hash for the downloaded file to verify if the downloaded files has been changed or not)
Screenshot 2021-07-03 at 11.49.01.png
Screenshot 2021-07-03 at 11.49.22.png
Louis
@louis:laureys.me
[m]
Cool! What's the exact use case for this?
Andris Reinman
@andris9
Once an email is downloaded as an evidence it must be possible to later validate that the email has not been tampered with and is the same that was in the server
not every download is actually signed. Instead download hash is logged and once you try to download it then the file is put together and signed with server key
Louis
@louis:laureys.me
[m]
Ah, that's pretty cool
Andris Reinman
@andris9
Btw this does not hash actual emails but the container, eg the downloaded zip file. Every time you download emails, be it a single email file or a zipped selection, then that action is logged and you can later go and download signed verifying hash for that download. Would prefer to somehow include the hash with the initial download but the zip files are streamed (can be very large) and there is no way to know the hash of it before it has been actually downloaded
Audit system does not show email contents, only metadata (including subject and to/from addresses). To actually see the email you have to download it and that action is logged.
venusian
@venusian:matrix.org
[m]
Hi all, I doing research for a medium to large installation and was wondering what scale of installations Wildduck is used for at the moment, it sounds like it should scale really well but I have not found any references so far
Andris Reinman
@andris9
@venusian:matrix.org WildDuck is mainly developed for a single specific email system. That system currently stores about 70TB of emails (that's virtual size, actual db size with deduplication is 47TB) and has 100k+ registered accounts. I'm not 100% sure but I guess that there are about 10k-20k logged in IMAP users in peak hours. There are 7 mongodb shards. New shards are added whenever free space runs out.
Louis
@louis:laureys.me
[m]
Have you ever ran into ram or CPU limitations before space ran out on a shard? Or is that generally not a problem?
Andris Reinman
@andris9
CPU is usually not an issue. Real problem is memory size as MongoDB needs to keep indexes in memory
if there is not enough memory then Mongo loads only "hot" indexes to memory and keeps everything else on disk that makes irregular operatsions (eg. search) quite slow
this is also the main thing that limits shard size - too much data on a single shard means that there is no way that indexes fit into memory
another limit is backup - backing up regularly a lot of TBs is real pain
so if the shard is smaller then it is also easier to back it up as there is less data
Venusian
@venusian:matrix.org
[m]
We're looking at a small multiple of those stats and are interested in alternatives to the known dovecot setups however at this scale everything new is scary :)
Backing up would need to be 'smart' and not simply backup the mongodb files but contents based on actual change I think, as coping everything everytime would murder any viable setup I can think of
Louis
@louis:laureys.me
[m]
Afaik the only viable backup method without mongodb enterprise is filesystem snapshots:
https://docs.mongodb.com/manual/tutorial/backup-sharded-cluster-with-filesystem-snapshots/
Andris Reinman
@andris9
we use PerconaDB where you can create db snapshots. so on each shard there is one replica set member with an extra 10TB disk. once a day we run the command to create the snapshot to that disk.
but this approach is not 100% perfect. Percona has newer backup tools available that are better but we have not started using these yet
by perconaDb I mean the MongoDB version released by Percona. it is otherwise exact copy but has additional options
Louis
@louis:laureys.me
[m]
Oh interesting. I have looked at their backup tool (https://www.percona.com/software/mongodb/percona-backup-for-mongodb), but my test backup failed to restore without much additional info. So I gave up on that.
But that's something different I think
Andris Reinman
@andris9
Anyway so far most issues with scaling have been with MongoDB. Regarding WildDuck the issues so far have been usually been bugs, not scaling.
and the issues with MongoDB have not been related to MongoDB being bad but more of lack of experience in scaling a large cluster
each large cluster is different, so you can't really follow a tutorial or anything
Venusian
@venusian:matrix.org
[m]
Can I ask what the 16 nodes look like in specifications like hardware and cpu? Scaling, and backingup for that matter large mongodb databases is something that has been done, interesting..
Andris Reinman
@andris9

I can't say the specs as I don't have access to these machines. I remember that the first shard was 3 machines, each had 64GB RAM. I guess there were 32 cores, not 100% sure. There was no RAID (instead the system was relying on the MongoDB replication) but there were 2 disks:

  1. 1.9TB SSD that was mounted /var/lib/mongodb
  2. 10TB HDD mounted to /var/lib/mongodb/attachments/collection

This disk setup ensured that all the indexes, message metadata etc was stored on a fast SSD and all the attachment content was stored on that large, slow HDD

Daviesmolly
@Daviesmolly
i just really want a simple text captcha instead of google's recaptcha @andris9
Andris Reinman
@andris9
There is no ready made solution, you could probably edit the code and replace the reCaptcha thing with your own
Daviesmolly
@Daviesmolly
Thanks for your response
Although i got some other challenges @andris9
Daviesmolly
@Daviesmolly
How do i add and validate extra input field on the wildduck webmail registration form i.e <input type="text" class="form-control" name="TEXT" id="username"> and maybe include the expression >> <div class="form-group{{#if errors.text}} has-error{{/if}}"> ?? The end GOAL is adding customized CAPTCHA to the wildduck webmail registration FORM. HELP?? @andris9 @louis:laureys.me
Daviesmolly
@Daviesmolly
<input type="text" class="form-control" name="TEXT" id="text">
Andris Reinman
@andris9
I would actually suggest you to disable wildduck-webmail signup form entirely by setting allowJoin=false (here). Then create your own signup form and once you have validated user data, create the user with createUser API call (here)
Whit Huntley
@whit-colm
Hello! I was wondering if there were any plans to provide a Kubernetes Helm chart for Wildduck?
Louis
@louis:laureys.me
[m]
No plans right now
I'm not comfortable enough with k8s to be able to provide one
Whit Huntley
@whit-colm
ah ok, thanks
covalente
@covalente:matrix.org
[m]
Hello, I'm interested on using wildduck for the ability to manage it with its REST apis. I tried to start it through docker but I don't find the documentation on how can I correctly setup it and edit the config
Louis
@louis:laureys.me
[m]
covalente What exactly do you need to know? The setup instructions are in the readme, and they're fairly short.
Sonu Sharma
@echosonusharma
can i just change CMD_ARGS: --api.accessToken= value to set a AccessTokenAuth in the docker-compose.yml file so when i use my api , i can send that in header.
Louis
@louis:laureys.me
[m]
Sure. Or edit it in the config mounted to the docker container.
2 replies
Louis
@louis:laureys.me
[m]
Yes. You can set it in api.toml