Post

Log in
About
Code of conduct
Privacy
Users
Instances
About Bonfire

@algernon@come-from.mad-scientist.club · activity timestamp 3 hr. ago

So... the major thing missing from my milter (ryoms) is bayesian filtering. I originally planned to use linfa, because it has not only a bayes model, but a vectorizer too, and I thought that'd be a great help.

Unfortunately, while I can update the training of the model, I can't update the vectorizer, not without great trouble. Great trouble that pretty much amounts to writing my own vectorizer to begin with.

And in that case, I might aswell use smartcore, because it has a more convenient API.

Flag this post
Block

6 Replies

0 Boosts

Luddicus Mus

@algernon@come-from.mad-scientist.club · activity timestamp 3 hr. ago

So the idea is that I will clean up the body of every email: if it has a HTML part, I will prefer that over text (because the observation is that the text part is often inadequate in this case, like "text is not available" and similar bullshit). I'll remove the HTML markup, split the remaining stuff into unicode words, and count each.

The vocabulary will be append-only, and will only be updated when explicitly training. If classifying, any tokens not in the vocabularly will be ignored. When classifying, and if the vocab needs an update, I'll sort the new words by frequency, and shove the top and bottom 25% to the end of the vocabularly. This way indexes never change, and previous training remains valid.

(The 25% will be configurable, of course)

Flag this comment
Block

Luddicus Mus

@algernon@come-from.mad-scientist.club · activity timestamp 3 hr. ago

Hm. The problem with smartcore, otoh, is that I don't know how to update a model, a'la linfa's FitWith::fit_with. So maybe I'll stick to linfa, and use my own vectorization thingy.

Flag this comment
Block

Luddicus Mus

@algernon@come-from.mad-scientist.club · activity timestamp 3 hr. ago

I do not feel like messing with this right now, though. Brain feels too foggy to think about maths. But at least I made a step forward!

Flag this comment
Block

Luddicus Mus

@algernon@come-from.mad-scientist.club · activity timestamp 2 hr. ago

After a short sidequest about SMTP servers, it looks like I will have to adjust ryoms to work as an OpenSMTPD filter, rather than as a milter.

The two protocol share similarities, but... not quite. I wonder if I want to support both milter and opensmtpd-filter protocols?

Or perhaps... use my own protocol, and have frontends / adaptors to map the various protocols (milter, opensmtpd-filter, and possibly rspamd) to the internal one.

I think this latter option is the best, because that makes it easier to adapt to whatever smtpd, which in turn, makes it easier for others to use it too.

While ryoms is (currently) my own toy, and I don't want anyone to use it yet, the longer term plan is that, like iocaine, it will be usable by others too.

Flag this comment
Block

~5 more replies

Luddicus Mus

@algernon@come-from.mad-scientist.club · activity timestamp 2 hr. ago

Now, for what ryoms does, I only need a few phases:

connect
helo/ehlo
mail-from & rcpt-to
post-data

ryoms currently has checks in the mail, rcpt, header, and end_of_message milter phases, which map to mail-from, rcpt-to and post-data respectively.

I'm not sure it's worth having a header phase. In many cases, it's easier to let the entire message arrive, and process it as a complete email.

Flag this comment
Block

Luddicus Mus

@algernon@come-from.mad-scientist.club · activity timestamp 2 hr. ago

For my own protocol, I'll use a text-based one, because that's easier to manually fiddle with, and the overhead of processing it is negligible. If I were going for top performance, I'd use a binary protocol, but in this case, convenience is more important.

Flag this comment
Block