Eudora TM Search Contact Us
Support - Technical Support Resources and Knowledgebase User Forums - See users' online questions and answers about Eudora products, or post your own questions to the Eudora Community. Developers - Resources for Developers writing code to interact with Eudora Press - Press Releases and News Articles Testimonials - Read what others are saying about Eudora

Eudora in the News

Filtering Spam

Statistical analysis, 'honeypots' are among new tools to deal with junk e-mail
By Jonathan Sidener
May 24, 2004

E-mail users have some new weapons in the battle against spam. No one yet is declaring victory over unwanted commercial e-mail. However, some of the new technologies seem to be getting smarter and more effective at filtering out the junk. Some strategies use software that runs on users' computers. Others are server-side technologies that run on Internet service providers' network equipment.

It's often possible to combine the strategies, using software on the home computer to augment the ISPs' filters. One increasingly popular method, called Bayesian filtering, uses statistical analysis to separate spam. Bayesian spam filters are used in consumer software and in server-side applications. Other filtering approaches include so-called white lists of approved contacts, in which "challenge and response" systems require that unfamiliar senders respond to questions before their e-mail is delivered. A third approach uses "honeypots," which are e-mail accounts set up to attract spam. The information from the spam is used to customize filters.

Scoring spam
Bayesian filters are creating a lot of buzz in the fight against spam. Eudora e-mail software from San Diego-based Qualcomm, for example, recently added a feature called SpamWatch, which uses Bayesian filtering to customize spam screening for each user. America Online uses a Bayesian system to screen spam. And there are Bayesian programs for Windows and Mac users, including several plug-ins to work with Microsoft's dominant Outlook and Outlook Express programs. Bayesian filters sort e-mail to either an inbox or to a junk-mail folder. If a user finds legitimate e-mail in the junk folder and moves it to the inbox, the system re-reads the e-mail and adjusts its formula for detecting spam in an attempt to avoid a similar mistake. For example, for many Internet users, an e-mail containing the word "breasts" has a high probability of being spam. But for a physician, that might not be true, Eudora's Bill Ganon said. "For Dr. Fisher, who gets a lot of e-mail about breast cancer, the filter is going to adjust and send that mail to his inbox," Ganon said. In the past, many filters worked by scanning an e-mail message and its header, then assigned numerical values to certain words. "Viagra" and "Nigeria" would have high numbers because they have a greater probability of being words found in spam. The filter would add up all the numbers in one e-mail to produce a spam score.

There are a couple of shortcomings with the content-scoring method. The numbers assigned to a given word are, at best, educated guesses. Someone has to decide whether "Nigeria" gets 95 points or 97. Spammers can get around the content-scoring filters by misspelling words. "Nigeria" becomes "Nigeri@," for example. The misspelled word would not increase the spam score. Bayesian filters solve this problem by comparing the content of known spam with legitimate e-mail. For example, if "Nigeri@" appears only in a user's spam and never in legitimate e-mail, it would get a high score. The software, which is installed on the user's computer, creates and constantly updates two databases on the characteristics of spam and legitimate e-mail for that Internet user. It conducts an ongoing statistical analysis of the known databases for spam and legitimate e-mail, and uses the results to score incoming messages! . So, when a doctor moves an e-mail containing the word "breast" from the spam folder to the inbox, the filter recalculates the score, or probability, of that word appearing in the doctor's spam. Ganon said SpamWatch is better than 90 percent accurate "out of the box" and improves to 98 percent or 99 percent accuracy after it customizes its formula to an individual user's e-mail.

Network strategy
While Bayesian filters operate on individuals' computers, other filtering strategies work at the network, or server, level. In the white-list/blacklist method, mail is filtered using a list of approved contacts and domain names, called the white list. Known spam sources, whether from individuals or e-mail servers, go on the blacklist and the e-mail is blocked. Microsoft's free e-mail service, Hotmail, uses a version of the white-list approach. Only mail from approved contacts is delivered to a user's inbox.

One drawback to this system is that the user has to update the contact list frequently. If a friend from high school tracks down someone and tries to contact them via e-mail, the e-mail will be diverted to a spam folder. It is then up to the user to browse the spam folder to find legitimate e-mail.

"Challenge and response" systems take the white-list strategy a step further. If e-mail is sent from someone not on the contact list, the mail is not immediately delivered to the user. First, a reply, or challenge, is sent to the sender. This verifies that the e-mail came from a legitimate mail account. Spammers often falsify e-mail headers, the behind-the-scenes portion of an e-mail that routes the message through the Internet. The header generates the "from" line that the recipient sees and spammers often falsify, or spoof, the "from" line to cover their tracks. The challenge reply often contains the image of a word that cannot be read by the "bots," or automated systems spammers use to send out millions of pieces of e-mail. The original sender must type the word into the response form before the original e-mail can be delivered to the inbox.

One disadvantage to the "challenge and response" system is that legitimate information from banks and other businesses will be filtered out if it comes from an automated system that cannot respond. Both Yahoo! and Microsoft said recently they will soon be using new systems for making e-mail headers more secure and harder to falsify.

ISP choices
At EarthLink, one of the largest ISPs in the United States, the company provides subscribers with a choice of "white list" or "challenge and response" systems. EarthLink's SpamBlocker program offers two levels of filtering. The medium level uses a contact list to control what goes into the inbox. The high level uses the "challenge and response" system to require that unknown e-mailers answer three questions. "Can technology keep pace with spam on an individual consumer level? The short answer is absolutely," said EarthLink's Stephen Currie, who oversees the company's e-mail service. He said the company diverts about 250 million pieces of spam each week. "Technology is just one piece," he said. "The fight against spam has multiple fronts. There's also a need for legislation, education and litigation."

In April, Cox Communications launched a different approach, offering the services of San Francisco-based Brightmail to Cox Internet subscribers. As part of its strategies, Brightmail uses "honeypots," e-mail accounts that exist solely to attract spam. Brightmail uses the information from the spam it catches to update the formula in its filters every 10 minutes. The company has servers in the cable company's neighborhood equipment centers. In a recent report on spam-fighting technologies, market research firm IDC concluded that the new technologies provide additional tools to combat unwanted e-mail. However, because the volume and sophistication of spam, none of the technologies provides complete security. Most Internet providers and software companies use a layered approach that combines two or more technologies. At Cox, Dan Novak, the company's San Diego vice president for programming, agreed that gaining some ground doesn't mean the war is over. "Spam is a moving target," he said. "Some of these spammers are bright guys. What works today, won't necessarily work tomorrow."

Direct UT Link:


| Home | Online Support | Open Source Development | User Forums | Contact Webmaster |

| QUALCOMM | Section 508 | Privacy Statement | Terms of Use |

© 1999-2009 QUALCOMM Incorporated. All rights reserved. QUALCOMM and Eudora are registered trademarks of QUALCOMM Incorporated. All other trademarks are the property of their respective owners.