Bayesian Filter: Technology And Advantages
by: Julia Gulevich
Not a long time ago, most anti-spam products simply used a list of keywords to identify spam. A good set of keywords could catch much spam. However, a keyword-based anti-spam filter requires manual updating and can be easily fooled by tweaking the message a little. Spammers simply examine the latest anti-spam techniques and find ways to bypass them. At the result you’re left with a high number of false positives.
The need in a new effective technique to fight against spam stood up. The experience showed that this new method might adapt itself to the spammers' tactics that would change with time.
The Bayesian filtering is based on the principle that most events are dependent and that the probability of an event occurring in the future can be inferred from the occurrences of this event in the past. This approach is used to identify spam. If some piece of text occurred mostly in spam emails but not in legitimate mail, then it would be reasonable to suppose that this email is probably spam.
To filter mail using the Bayesian technology, you need to generate a database of words collected from spam and legitimate mail. Then a probability value is assigned to each word; the probability is based on the calculations that take into account how often that word occurs in spam as opposed to legitimate mail.
After the legitimate and spam databases are created during an initial training period, the word probabilities can be calculated and the Bayesian filter is ready for use. When a new mail arrives, it is broken into words and the most significant words are singled out. From these words, the Bayesian filter calculates the probability of a new message being spam or not. If the probability is greater than a spam threshold, say 0.9, the message is classified as spam.
Tip! G-Lock SpamCombat allows you assign the hot keys to the common operations. For example, you can assign F8 to Mark Message as SPAM function and F9 to Mark Message as Clean. Next time when you train the Bayesian filter you can simply use two keys on your keyboard F8 and F9.
It is important to note that the analysis of spam and legitimate mail is performed on the mail the particular user (organization, company, etc.) receives, and therefore the Bayesian filter is adjusted to this particular person, company, or organization. For example, a financial institution may receive a lot of emails with the "mortgage" word and would get a lot of false positives if using an outdated anti-spam filter. The Bayesian filter analyzes the entire message with the word "mortgage", and concludes whether this email is spam or legitimate basing NOT only on a single keyword "mortgage". The Bayesian approach to filter spam is highly effective - spam detection rates of over 99.7% can be achieved with a very low number of false positives!
Let’s summarize what benefits we get using the Bayesian filter to catch spam:
1) Much more intelligent approach because it examines all aspects of a message, as opposed to keyword checking that classifies a mail as spam on the basis of a single word.
2) Self-adapting - constantly learning from new spam and new valid inbound mails, the Bayesian filter evolves and adapts to new spam techniques.
3) Sensitive to the user – it learns the email habits of the company and understands that, for example, the emails with the "mortgage" word are not always spam.
4) Multi-lingual and international - being adaptive it can be used for any language. The Bayesian filter also takes into account certain languages deviations or the diverse usage of certain words in different areas, even if the same language is spoken.
5) Difficult to fool, as opposed to a keyword filter - an advanced spammer who wants to trick the Bayesian filter can either use fewer words that usually indicate spam, or more words that generally indicate valid mail (such as a valid contact name, etc). Doing the latter is impossible because the spammer would have to know the email profile of each recipient - and a spammer can never hope to gather this kind of information from every intended recipient.
About The Author
Julia Gulevich is a technical expert associated with development of computer software like AATools, Email Verifier, G-Lock EasyMail, Anti-Spam Software
http://www.glocksoft.com/sc/ More information can be found at Anti Spam Blocker Resources
http://www.glocksoft.net/sc/
