Java Programming Notes # 2154
- Preface
- Preview
- Operational Discussion
- Program Code
- Run the Program
- Summary
- What’s Next?
- Complete Program
Listing
Preface
This is the third lesson in a series designed to teach you how to
write
a Java program to remove SPAM from your Email server before you
download it into your primary Email client. The first lesson was
entitled Enlisting
Java in the War Against SPAM, Part 1, The Communications Module.
The previous lesson was entitled Enlisting
Java in the War Against SPAM, Part 2, The Screening Module.
The communications module
The first lesson explained the communications module used to
communicate with
your Email server, and to remove SPAM messages from the
server.
SPAM screening algorithm
The second lesson explained my SPAM screening algorithm. The
program is designed to allow you to use my SPAM screening
algorithm, or to invent your own. You can use my
algorithm as a starting point if you decide to invent your own.
Training the algorithm
My algorithm operates
separately on the Subject line, the From line,
and the body text of each Email message. This
lesson explains my program for training the algorithm to do a better
job of
identifying SPAM in the future based on the Subject line
of Email messages.
The next lesson will explain my program for training the
algorithm to do a better job of
identifying SPAM in the future based on the body text of
Email messages.
Viewing tip
You may find it useful to open another copy of this lesson in a
separate browser window. That will make it easier for you to
scroll back and forth among the different listings and figures while
you are reading about them.
Supplementary material
I recommend that you also study the other lessons in my extensive
collection of online Java tutorials. You will find those lessons
published at Gamelan.com.
However, as of the date of this writing, Gamelan doesn’t maintain a
consolidated index of my Java tutorial lessons, and sometimes
they are difficult to locate there. You will find a consolidated
index at www.DickBaldwin.com.
Preview
Can you write better SPAM screening
algorithms?
Did you ever think that you might be able to write better SPAM
screening algorithms than those available in the SPAM screening
software that you are now using? If so, this series of lessons is
for you.
Even if that is not the case, like most of us, you are probably
overwhelmed by SPAM
and therefore you may find this series of lessons interesting.
Remove SPAM from the server
In this series of lessons, I am showing you how to write a
Java program
that supplements the SPAM screening software that you are currently
using. The program is used to identify and remove SPAM from your
Email server before it is downloaded into your primary Email client.
Any SPAM that makes it past the SPAM screening program can be
further acted upon
by the SPAM screener that is built into your Email client.
Lessons in the series
This series consists of (at least) four lessons. The
first lesson in the
series explained the communications module used to communicate with
your Email server, and to remove SPAM messages from the
server.
As mentioned earlier, the program is designed to allow you to invent
and implement your own
SPAM screening algorithm in addition to, or as an alternative to my
algorithm.
The second lesson explained the inner
workings of my SPAM screening algorithm, which operates
separately on the Subject line, the From line,
and the body text of each Email message.
Algorithm training programs
The screening program is
most useful
when the algorithm has been well trained, and the training of the
algorithm is kept up to date. Although it is possible to perform
this training with a
text editor, you can be much more productive, and you are much more
likely to keep the training up to date using the programs that I will
explain in this lesson and the next.
This lesson explains a program named Pop302d,
which
uses historical data to train the algorithm to do a better job
of identifying SPAM in future messages based on the Subject
of the message.
The next lesson will explain another program named Pop302e,
which uses historical data to train the algorithm to do a
better job of identifying SPAM based on the body text of
the message, (which includes the From line but excludes the
Subject line).
Because of the need to train the algorithm and to keep it up to
date, and the ease with which
these programs make that possible, the training programs are
equally as important as the main program.
Operational sequence
Here is the typical operational sequence that I go through each
morning to remove SPAM from my Email server before downloading it into
my primary Email client, and to train the algorithm to recognize any
future SPAM messages that managed to evade the screening program on the
first pass that morning.
- Run the main program named Pop302 (explained in the
previous two lessons) to identify SPAM and remove it from the
server. This normally allows a few (typically about five
percent) SPAM messages (stragglers) to get
through. These straggler messages are stored in a history folder
on my local disk. - Run the program named Pop302d (explained in this
lesson) to train the algorithm to recognize the stragglers as SPAM
based on
information in the Subject line. (It is also
possible to recognize that a message is not SPAM and to exclude it from
further training activities.) - Run the program named Pop302e (explained in the next
lesson)
to train the algorithm to recognize the stragglers as SPAM based on
information
in the body text. - Go back and run the main program named Pop302 to remove
the SPAM stragglers from the server. - Run the primary Email client to download the remaining good
messages from the Email server into my local Email inbox.
When I am in a hurry …
However, it isn’t necessary to perform all of these steps every
day. On those mornings when I am in a hurry, I skip steps
2, 3, and 4, leaving the straggler messages in the local history folder
for use later.
(The straggler messages will, of course, end up in my
local Email inbox when I run my primary Email client without having
purposely
removed them from the server beforehand.)
Sometime later (perhaps the next day or several days later)
I will perform steps 2 and 3 to train the algorithm to recognize future
SPAM
messages represented by the characteristics of the SPAM messages that
have
been saved in the local
history folder.
Effectiveness of my algorithm
After several weeks of training, my
algorithm is reliably identifying about ninety-five percent of all SPAM
messages, allowing me to delete them from my Email server before
downloading them into my primary Email
client. By executing steps 2, 3, and 4 above, I am able to also
eliminate the remaining five percent of the SPAM messages before
downloading them into my primary Email client.
Most
of
the messages that make it past the screen are messages written in
a non-English language that I am unable to read. No one that I
know would send a message to me in a
language that I can’t read, so I consider such messages to be
SPAM. They constitute the bulk of the five-percent of the SPAM
messages that make it past the screen.
Operational
Discussion
Offensive words and phrases
My SPAM screening algorithm screens for SPAM on the basis of words
or
phrases in the From line, words or phrases in the Subject
line, and
words or phrases in the body text.
Friendly Email addresses and subjects
A list of friendly Email addresses and friendly subjects is used to
screen the From
line and the Subject line. Messages that are from
friendly Email addresses, and messages that have known good Subject
lines are preserved on the server and no information about those
messages is
saved on the local disk. They are simply ignored after determining that
they are friendly.
The friendly list is created and maintained using a simple text
editor. This is a relatively short list that rarely changes and
no special training program is needed to keep this list
up to date.
Different lists for Subject and body
text
Different lists of words and phrases are used for screening Subject
lines and body text for SPAM. This is important because
the same set of
words
and phrases aren’t always appropriate for use in both cases.
For example, the word ANTIVIRUS is appropriate for screening the
Subject line, but is not appropriate for screening the
body text. The
word ANTIVIRUS often appears legally in the header of Email messages
that have been scanned for viruses by the server, but also often
appears in the Subject line of SPAM messages.
Conversely, some of the most useful material in the list used to screen
the body text is a large set of URLs for known spammer web sites.
However, URLs have little value for screening the Subject line,
so there is no point in wasting computer time screening Subject lines
against URLs.
To summarize …
Although there are exceptions to every rule, the most useful material
for screening Subject lines consists of plain text
words and phrases that are commonly included in the Subject lines
of SPAM messages.
Such text is of very limited value when screening the body text
for reasons that I will explain later. The most useful material
for screening the body text is a list of known spammer
URLs, which are of very limited value when screening Subject lines.
Common spammer tricks are defeated
Several common spammer tricks are defeated by my SPAM screening
algorithm.
Insertion of extraneous characters and case
changes
For example, the common spammer trick of inserting a small number of
extra characters
between the
characters in an offending word or phrase is defeated. Also, the
common trick of mixing the case of the
characters in an offending word or phrase is defeated.
As a specific example, my algorithm will recommend deletion of any
message having
any of the following in its Subject line or its body
text if the word VIAGRA is included in the lists used to screen for
SPAM:
vIaGrA
V.IagRA
V.I.A.G.R.A
Therefore, when using this program to train the algorithm, if you find
a SPAM message containing V.IagRA in the subject line, you
should use the program to add the word VIagRA to the
list. (Simply edit out the extraneous period between the V
and the I.) The algorithm will deal with the other
variations of that word having extraneous characters inserted and
having random case changes.
Purposely misspelled words
As currently written, my algorithm has no way of dealing with another
common spammer trick of purposely misspelling words.
(Perhaps this is an opportunity for you to improve the
algorithm.)
Therefore, if you find a SPAM message where the Subject line
contains the word V.I@gRA, you should use the program to add
the word VI@gRA to the list to deal with the incorrect
spelling. (Once again, just edit out the extraneous period.)
Duplicates are not a problem
Don’t be concerned that you may forget and add a word or phrase to the
list more than once. The program will eliminate all duplicates,
thereby preventing the length of the list from increasing due to
duplicates.
Bogus HTML tags
One of the reasons that plain text words and phrases are not very
useful in screening body text is because spammers often
insert long bogus HTML tags in the middle of such words and
phrases. Because the tags are not recognized as valid HTML tags,
the HTML rendering engine simply ignores them when the SPAM message is
displayed in an HTML-compatible Email client.
My screening algorithm, as currently written, does not deal with this
situation. It wouldn’t be difficult to update the algorithm to
cause it to delete bogus HTML tags before screening the body
text. I will probably update the
algorithm to delete such tags at some point in the future, but the
current version doesn’t contain that capability.
The user interface
Figure 1 shows the GUI through which the user controls the training
program named Pop302d. Figure 1 also partially illustrates
the procedure that I use to train the algorithm to
do a better job of identifying SPAM in the Subject line
of future messages.
Figure 1 Graphical User Interface
(Note that this GUI was purposely made narrow to cause
it to fit into this narrow publication format. I
recommend that you increase the width of the Frame to at least 750
pixels, and increase the width of the TextField and TextArea objects to
at least 100 characters each.)
A message from the history folder
In Figure 1, a message previously stored in the history folder has been
loaded into the GUI. The complete raw text of that message is
available for viewing in the large text area if desired.
The From
line and the Subject line are displayed in the
top two text fields in the GUI. (I purposely deleted the
Email address of the sender.)
User instructions are displayed in the fourth text field.
Offensive text in the Subject line
In this case, the user has identified unique and offensive text (XA-N@X)
in
the Subject
line and has selected that text with the mouse. Having
first selected the text with the mouse, the user has pressed the
button labeled Copy Selected Text. That caused the
selected text to be automatically copied into the third text field.
Two additional steps
Two
additional steps are required to add a useful version of that text to
the word
list used to screen the Subject
line of future messages.
The first step is to delete the “-” character that separates the A
from the N, at which time the contents of the third text field
will read XAN@X.
The second step
is to press
the button labeled Post Text. This will cause the selected and (possibly)
edited text to be automatically added to the word list.
That is all that is required to cause the program to identify this
offensive text in the Subject lines of all future
messages.
Additional offensive material
At this point, the user may be able to identify other unique and
offensive words and phrases in the Subject line. (The
Subject line might contain a long list of drugs, for example.)
The user can select any number of additional words and phrases from the
Subject line and add them individually to the list using
the above
procedure.
Can type, or copy and paste also
In addition, the user can add other words and phrases to the list by
either typing them into the third text field, or by selecting and
copying words and phrases from the large text area and pasting them
into the third text field. Having done that, the user can cause
the word or phrase in the third text field to be added to the list
simply by pressing the Post Text button.
Process the next message
If the user then presses the Next button, the next message in
the history folder will be loaded into the GUI. The current
message will
not be deleted from the history folder.
(This is
what you would normally do if you are going to use the same message
later to train the algorithm to better identify SPAM on the basis of
the body text. I will discuss this procedure in the next lesson.)
If the user presses the Delete
Local File button, the current file will be deleted from the
history folder and the next message in the history folder will be
loaded into the GUI.
(This is
what you would normally do if you have determined that the message is
not SPAM, or should not be used for further training of the algorithm
for some other reason. Perhaps the message was received from a
friend whose Email address has not yet been added to the list of
friendly Email addresses discussed earlier in this lesson. Note
that deleting the message file from the local disk does not delete the
message from the server.)
A
very simple process
As you can
see, the process of training the algorithm on the Subject line
consists simply of selecting text with the mouse and pressing buttons
to cause the selected text to be added to the word list. This can
be accomplished very quickly with very little effort. Except for
the possible requirement to delete extra characters, no actual typing
is required.
Be aggressive
My experience using a moderate speed computer is that the length of the
list used to screen the Subject line has very little
impact on the time required to run the main program to identify SPAM
and remove it from the server. Therefore, you can afford to be
very aggressive when adding words and phrases to this list.
(As I will explain in the next lesson, that is not the
case for the list used to screen body text. For reasons
that I will explain then, you will need to be somewhat concerned about
the length of that list.)
If you identify a word or a phrase in the Subject line
of a SPAM message that you believe is unique to SPAM, and is likely to
occur in future SPAM messages, go ahead and add it to the list.
Be careful of false positives
Be careful, however, of adding words or phrases to the list that will
create false positives in the future (a false positive is a
non-SPAM message that is identified as SPAM by the main program).
For example, you probably shouldn’t add the word GET to the
list. This word is not unique, and it will be found in many
future messages, including those that are SPAM, and those that are
not SPAM.
False positives cause no permanent damage (you can always elect not
to delete messages from the server when running the main program,
even
if the program identifies them as SPAM).
The problem with false positives is that they force you to concentrate
more carefully when making the
decision to delete or not to delete. This can increase the time
required to process the messages on the server.
Program Code
Program Pop302d
This lesson explains the program named Pop302d,
which is used to train my screening algorithm to do a better job of
identifying future SPAM messages on the basis of the Subject
line.
Simple text files
All three word lists used by my screening algorithm are maintained in
local text files. These files can be
created and edited with an ordinary text editor if need be. Thus,
if one of the lists becomes corrupted, it is easy to correct the
situation using a text editor. However, it is easier to train the
algorithm using the programs that I will explain in this and the next
lesson.
File names
The following file names are hard-coded into the various programs that
use these text files. (You may
want to change these file names for your version of the programs.
If you do, make certain that you change them everywhere that they are
used.)
- Pop302a.txt – contains a word list used for screening the Subject
line for offensive words and phrases. This file is updated by the
program discussed in this lesson. - Pop302b.txt – contains a word list used for screening the
body
text
for offensive words and phrases. This file is updated by a
program named Pop302e that I will discuss in the next lesson. - Pop302c.txt – contains a list of friendly Email addresses
and
friendly subjects for
screening the From and Subject lines to
identify friendly messages. This file is easy to keep up to date
using a text editor. No special program is required for updating
this file.
Location of the text files
The programs require the three .txt files to
be in the same folder as the compiled .class files for
the programs named Pop302, Pop302d, and Pop302e.
However, you can easily modify the programs to change the location of
the .txt files if
you choose to do so. Just be sure to change the location in all
three programs.
Local copies of the messages are stored in two different
folders. Some of the
local copies are stored in a history folder while the remainder are
stored in an archive folder. The locations of these folders on
the disk are hard-coded into the three programs. You can change
the locations if you like, but be sure to make appropriate changes to
all three programs.
Testing
This program was tested using SDK 1.4.2 under WinXP.
Will discuss in fragments
I will discuss the program named Pop302d in fragments. A
complete listing of
the program is provided in Listing 30 near the end of the lesson.
You should be able to copy and paste that listing into your Java IDE to
compile and test the program on your system.
Purpose of the program
The purpose of this program is to process raw message text files
produced by the program named Pop302, using the information
contained in those files to update the word list stored in the file
named Pop302a.txt.
Designed for ease of use
The program is designed for extreme ease of use in order to encourage
the user to keep the word list up to date.
Beginning of the class definition
This program consists of one top-level class named Pop302d, and
six anonymous inner classes. The top-level class definition
begins in
Listing 1.
Note that Pop302d extends Frame. Therefore, an
object of type Pop302d is a GUI.
class Pop302d extends Frame{
|
Increase the size of the GUI
The code in Listing 1 declares numerous instance variables,
initializing many of them.
Note in particular the instantiation of objects of the classes TextArea
and TextField in Listing 1. As mentioned earlier, this
GUI was purposely made narrow to force it to fit into this narrow
publication format. The GUI is much more useful if it is
wider. I recommend that you modify the parameters passed to the TextArea
and TextField constructors in Listing 1 to make them at least
100 characters wide (instead of 50).
I also recommend that you modify the parameters passed to the setSize
method later in the program (see Listing 27) to cause the Frame
object to be at least 750 pixels wide.
The main method
The main method for the program is shown in Listing 2.
public static void main(String[] args){
|
The main method instantiates an object of the Pop302d
class, causing
the constructor to be executed.
(Much of the interesting code in
the program resides in the constructor.)
Then the main method invokes the makeSubjWordList
method on a reference to the object. I will discuss that method,
along with a couple of other utility methods before discussing the
constructor.
The makeSubjWordList method
The purpose of the makeSubjWordList method is to create a TreeSet
object
containing words used by the program named Pop302 to screen the Subject
lines of Email messages in an attempt to identify SPAM messages.
The makeSubjWordList method reads strings from a text file
named Pop302a.txt and creates the list as a TreeSet object
sorted in natural order with no duplicates.
The data read from the file is converted to upper case before being
added to the TreeSet object.
Also makes backup files
If that were all that the method did, it would be a relatively simple
method. However, after creating the TreeSet object, the
method writes the
data from the object into a backup file named Pop302a.bakN,
where
N is the value of the next available file name in the
directory.
A new backup file with a unique name is created each time the program
is run. Once the number of sequential backup files reaches 5, the
program automatically deletes the oldest file before creating a new
backup file.
Thus the program maintains a sequence of five backup files with
extensions bak0 through bak5 with one number missing.
Restoring data from a backup file
If you need to use one of the backup files to restore your primary
file, the age-order of the files should be determined by the
modification date and not by the name of the file.
The makeSubjWordList method begins in Listing 3.
void makeSubjWordList(){
|
Read text file and populate TreeSet object
The code in Listing 3 reads the words and phrases from the text file
named Pop302.txt (in the current directory) and
populates
the TreeSet object referred to by subjWordList with an
upper-case version of those words and phrases.
This code is straightforward and should not require detailed discussion.
When the code in Listing 3 finishes executing, the TreeSet
object is populated and ready for use.
Write a backup file
The code in Listing 4 begins the process of making a backup file before
returning to the calling method and allowing the data in the TreeSet
object to be modified.
int N = 0; |
Determine the next allowable backup file name
The code in Listing 4 determines the next allowable backup file name by
determining which one of the following set of six backup files does not
exist in the directory:
- Pop302a.bak0
- Pop302a.bak1
- Pop302a.bak2
- Pop302a.bak3
- Pop302a.bak4
- Pop302a.bak5
The number associated with the first backup file found to be missing is
saved in the local variable named N.
(Initially, of course, they are all missing so a value
of 0 would be saved in N, and the first backup file would be
named Pop302a.bak0.)
Cause N to rotate from 0 through 5
When the value of N reaches 5, the backup file named Pop302a.bak0
is deleted as shown in Listing 5. The backup file created by that
run will be named Pop302a.bak5 causing the file named Pop302a.bak0
to be the missing file during the next run.
if(N == 5){//del file 0 for use next time
|
For N not equal to 5
For values of N other than 5, the next backup file in the
sequence will be deleted (if it exists) making that file name
available for the next run. This is shown in Listing 6.
else{//delete the next file in sequence
|
Get output stream for the backup file
Having determined the value of N to be used in creating and
writing a backup file, the code in Listing 7 opens an output file
stream for a file using the name encapsulated in the File
object in Listing 4.
DataOutputStream dataOut = |
Write and close the backup file
The code in Listing 8 uses an Iterator to read the contents of
the TreeSet object and to write the data into the backup file.
Iterator iter = subjWordList.iterator(); |
Then the code in Listing 8 closes the backup file and the method
returns. At this point, the TreeSet object is populated
ready for use and a new backup file has been written containing the
contents of the TreeSet object.
Additional utility methods
The Pop302d class contains two additional utility methods,
which you can view in their entirety in Listing 30:
- writeSubjWordList
- removeStars
The writeSubjWordList method is completely straightforward and
shouldn’t require any discussion. This method is used to write
the output
file containing a modified word list when the program terminates.
The removeStars method is the same as a method having the same
name that I discussed in the previous lesson entitled Enlisting
Java in the War Against SPAM, Part 2, The Screening Module.
This method is used to remove asterisks that were purposely inserted
into the message files when they were created to defeat any virus code
that might be lurking there. I will simply refer you back to the
discussion in the previous lesson for this method.
The constructor
That brings us to the constructor for the class named Pop302d,
which begins in Listing 9.
The code in Listing 9 registers an anonymous WindowListener object
to service the close button on the Frame.
(The close button is typically the button with
the X in the upper-right corner of the Frame.)
Pop302d(){//constructor
|
The windowClosing method
This anonymous class defines the windowClosing method declared
in the WindowListener interface, by extending the WindowAdapter
class.
(If you don’t understand what I mean by this, see the
lessons on this topic at www.DickBaldwin.com.)
When the user presses the close button, the windowClosing
method is invoked. The windowClosing method invokes the writeSubjWordList
method to write the contents of the TreeSet object into the
output file named Pop302b.txt.
Then the windowClosing method terminates the program.
(As you will see later, the contents of the TreeSet
object are also written into the output file when the user clicks the Next
button and there are no more messages to be processed. I will
explain the rationale for that later.)
Set the layout for the GUI
The code in Listing 10 sets the layout for the GUI to FlowLayout.
This is a simple layout manager that works pretty well for this simple
GUI.
setLayout(new FlowLayout()); |
An ActionListener on the Next button
Listing 11 shows the beginning of code, which registers an anonymous ActionListener
object on the Next button in Figure 1.
nextButton.addActionListener( |
The definition of this anonymous class, which defines the actionPerformed
method of the ActionListener interface, is rather long and
complex. I will break it down into several fragments to discuss
it.
A String[] object containing file names
Later on, we will see code that populates an array object of type String[]
with references to String objects containing the names of all
the files in the directory that end
with the extension .txt. That object is referred to by a
reference variable named dirList, which is declared in Listing
1
and used in Listing 12.
For now suffice it to say that this String[] object will have
been populated before the user has an opportunity to press the Next
button, which causes the code in the actionPerformed method
that
begins in Listing 12 to be executed.
public void actionPerformed( |
As you may have surmised, the variable named fileCounter in
Listing 12 is
used to keep track of the number of files processed.
If not out of bounds
The actionPerformed method that starts in Listing 12 consists
of two major blocks of
code, delineated by an if/else statement.
The first block of code, which begins with the if statement in
Listing 12, is executed if the value of fileCounter is positive
and is less than the number of files whose names are stored in
the String[] object.
The second block of code will begin with the else clause that
in Listing 21 quite a bit later in this discussion.
If message is the last message
Having determined that the value of fileCounter is still within
bounds, the next step is to determine if the value of fileCounter
represents the last file in the directory. The code to determine
this and to take
the necessary action if true is shown in Listing 13.
if(fileCounter == |
No more message files
The body of the if statement in Listing 13 is executed when the
user clicks the Next button and the value of fileCounter
represents the last file in the directory. Among other things, we
need to disable the Next button so that the user cannot attempt
to read and process any more message files.
Write the output file
This code in Listing 13 begins by invoking the writeSubjWordList
method to cause the current version of the word list in the TreeSet
object to be written into
the output file.
Although the writeSubjWordList method will be invoked again
later
when the user presses the close button to terminate the
program, writing the file at this point ensures that all changes made
to the contents of the TreeSet object up to this point will be
reflected in the output file even if the user fails to properly
terminate the program by pressing the close button.
Construct message to the user
The code in Listing 13 constructs a message to the user that will be
displayed at the end of the message text in the large text area of
Figure 1.
This message indicates that while there are no more messages to be
processed, the user can post additional words and phrases to the list
before pressing the close button to terminate the
program.
Terminating the program
If the user terminates the program properly by pressing the close
button, the contents of the TreeSet object will be written
into the output file again when the program terminates. Those
contents will include any postings made while processing the last
message.
However, if the user terminates the program improperly by failing to
press the close button, only those final postings
associated with the last message will be lost. The contents of
the
TreeSet object have already been written into the output file
containing all postings made up to, but not including the postings
associated with the last message.
Disable the Next button
Finally, the code in Listing 13 disables the Next button to
ensure that the user cannot attempt to read and process additional
message files.
Standard processing for all message files
The code beginning in Listing 14 is executed for all message files,
including the last one.
textArea.setText("Processing " +
|
Messages for the user
The code in Listing 14 displays three messages for the user. One
message is displayed at the top of the large text area in Figure
1. The other two are displayed in the third and fourth text
fields in Figure 1.
Prepare to read the message file
The code in Listing 15 is executed in preparation for reading the
message file.
try{
|
The code in Listing 15 performs the following actions:
- Get an input stream that can be used to read the file whose name
is stored in the String[] object at index fileCounter.
Note that asterisks were written into the file on ten-character
intervals when it was written to defeat any executable virus code that
might be lurking there. - Declare a local variable named data that will be the
temporary repository for each message line as it is read from the file. - Set a mark at the beginning of the file that will allow for
rewinding the file to the beginning after some lines have been read. - Put text in the top two text fields of Figure 1 that will be
displayed in the unusual event that the message doesn’t have From
and Subject lines.
Read and display the Subject line
The code in Listing 16 reads far enough into the message to find the
Subject line and to display it in the second text field in Figure 1.
while((data = inData.readLine()) |
The code in Listing 16 performs the following actions:
- Read the next line of text from the file. If readLine returns
null, terminate the loop. - Invoke the removeStars method to remove the asterisks
purposely inserted into the file when it was written. - Test to see if the line starts with Subject: If
not, loop and read the next line. If so, convert the line to
upper case, write it into the second text field in Figure 1, and break
out of the loop. In that case, there is no point in reading
additional lines.
Get and display the From line
The code in Listing 17 rewinds the input stream from the file, and then
executes essentially the same code as Listing 16 to get and display the
From information in the first text field in Figure 1.
inData.reset(); |
Read and display entire message
The code in Listing 18 rewinds the input stream, and then reads and
displays the entire message, one line at a time, in the large text area
in Figure 1.
inData.reset(); |
This data is displayed to help the user decide if this is a SPAM
message, and if so, what to do in terms of updating the word list used
for screening the Subject line of future messages.
Display user message and close the message file
The code in Listing 19 displays a message to the user in the large text
area at the end of the email message in Figure 1.
textArea.append(msgToUser + "n"); |
The message will either be the message initialized into msgToUser
in Listing 1, or the value written into that object in Listing 13.
Then the code in Listing 19 closes the input file.
Increment the file counter
The code in Listing 20 increments the file counter so that the next
time the user presses the Next button, the program will be
ready to read and process the next message file.
fileCounter++; |
If fileCounter is out of bounds
Much earlier, in conjunction with the discussion of Listing 12, I told
you that the actionPerformed method consists of two major
blocks of code. One block, which I have been discussing up to
this point, is executed when the value of fileCounter is in
bounds insofar as the number of available files is concerned.
Consider, however, the case where the Next button is pressed
when the directory is empty of files. This case is handled by the
else clause in Listing 21.
else{//file counter is out of bounds
|
The code in the else clause in Listing 21 displays a message to
the user to the effect that there are no more files and disables the Next
button.
End of anonymous class definition for Next
button
Listing 21 also signals the end of the code, begun in Listing 11,
which registers an anonymous ActionListener object on the Next
button.
Instantiate ActionListener object from
anonymous class
The code that begins in Listing 22 registers a common ActionListener
on the Post Text button and also on the third text field in
Figure
1. This makes it possible to post a new word or phrase to the
list by pressing the Post Text button, or by pressing the Enter
key when the text field has the focus.
ActionListener postListener = |
The code in Listing 22 instantiates an object of an anonymous class
that implements the ActionListener interface, and saves that
object’s reference in the reference variable named postListener.
The actionPerformed method
The actionPerformed method defined in Listing 22 performs the
following actions:
- Get the word or phrase from the third text field in Figure 1 and
add it to the list in the TreeSet object. - Modify the text in the third text field to provide feedback to
the user indicating that the posting is complete.
Register ActionListener on Post Text
button and output field
The code in Listing 23 registers the ActionListener object on
both the Post Text button and the third text field in Figure 1.
postButton.addActionListener(postListener); |
Register ActionListener on Copy Selected
Text button
Recall from the earlier discussion of Figure 1 that the user can select
a block of text in the second text field and copy the selected text to
the third text field by pressing the Copy Selected Text button.
This is accomplished by an ActionListener object that is
registered on the Copy Selected Text button in Listing 24.
copyButton.addActionListener( |
The actionPerformed method in Listing 24 invokes the getSelectedText
method on the second text field in Figure 1 and writes that block of
text into the third text field by invoking the setText method
on the third text field.
Register ActionListener on Delete Local
File button
The code that begins in Listing 25 registers an ActionListener on
the Delete Local File button in Figure 1. This ActionListener
makes it possible for the user to remove a file from the local history
directory.
While running this program, the user would typically delete files from
the history directory that
are recognized as not being SPAM. Such files should not be used
for training the
screening algorithm.
deleteButton.addActionListener( |
Process for deleting a file
The process for deleting a file in Java is to instantiate a File
object that represents the file to be deleted, and then to invoke the delete
method on that File object.
A File object that represents the file to be
deleted
This is accomplished by the actionPerformed method defined in
Listing 25.
Recall that dataDir is a reference to a File object
that represents the directory where the message files are stored.
The code highlighted in red in Listing 25 constructs a String
object representing the path and file name of the file to be
deleted. This String is passed to the constructor for the
new File object.
(Note that when instantiating the new File
object, it is
necessary to subtract one from the value of the file counter to cause
it to reference the current file. This is because it has already
been
incremented by the event handler for the Next button in
preparation for
processing the next file.)
Then the code in Listing 25 confirms that the file actually exists, and
if true, the file is deleted.
Fire a synthetic ActionEvent on the Next
button
The code in Listing 26 fires a synthetic ActionEvent on the Next
button to cause the program to process the next file in the history
directory without a requirement for the user to press the Next
button.
Toolkit.getDefaultToolkit(). |
I have discussed the firing of synthetic event in previous lessons in
this series, so I won’t discuss that topic further in this lesson.
Configure the GUI
The code in Listing 27 configures the GUI by placing the various
components in it, setting the title, setting the size, and making the
whole thing visible.
add(copyButton); |
As mentioned earlier, you should increase the width
parameter to the setSize
method (highlighted in red) to at least 750 pixels. That
will cause your GUI to be much wider and much more useful.
(Don’t forget to also change the width of the text
fields and the text area in Listing 1 to at least 100 characters
instead of only 50 characters.)
Create the directory listing for the history
directory
The code in Listing 28 populates a String[] object referred to
by dirList with the names of the files in the history directory
having an extension of .txt.
This is an anonymous implementation of a class that implements the Filename
Filter
interface.
dirList = dataDir.list( |
What does Sun have to say about FilenameFilter?
The code in Listing 28 is a little cryptic. Here is
part of what Sun has to say about the FilenameFilter interface:
“Instances of classes that implement this interface are
used to filter filenames. These instances are used to filter directory
listings in the list method of class File
…“
The code in Listing 28 defines the accept method, which is
declared in the FilenameFilter
interface.
The list method of the File class
Recall that dataDir is a reference to an object of type File,
which represents the directory containing the message files. The
code in Listing 28 invokes the list method on the File
object, passing a reference to an object of type Filename Filter
as a parameter.
There are two overloaded versions of the list method in the File
class. One of the overloaded versions takes no parameters, and,
according to Sun,
“Returns an array of strings naming the files and
directories in the directory…”
That is not the version of list invoked in Listing 28.
Rather the version of the list method invoked in Listing 28
“Returns an array of strings naming the files and
directories in the directory … that satisfy the specified filter. The
behavior of this method is the same as that of the list() method,
except that the strings in the returned array must satisfy the filter.”
The filter
The characteristics of the filter referred to above are
determined by the behavior of the accept method defined in
Listing 28. The accept method returns true if
the name of the file, which is received as an incoming parameter, ends
with .txt. Otherwise, the accept method returns false.
In operation …
In operation, the code in the overloaded list method calls the accept
method once for each item in the directory, passing the directory and
the name of the item as parameters. If that item is a file, and
if the name of the file ends with .txt, the accept method
returns true, and that file name is included in the String[]
object returned by the list method. Otherwise, the accept
method returns false, and the name of that item is
not included n the returned String[] object.
Display a list of files
When the GUI first appears on the screen, the code in Listing 29 causes
a list of the text files in the history directory to be displayed in
the large text area in Figure 1.
this.textArea.append("Files to be processed"
|
The code in Listing 29 is straightforward and shouldn’t require further
explanation.
End constructor and class
The code in Listing 29 also signals the end of the constructor as well
as the end of the class. Once again, you will find a complete
listing of the program in Listing 30 near the end of the lesson.
Run the Program
I encourage you to copy the code from Listing 30 below, as well as
the program named Pop302 and the three
starter text files at the end of the previous lesson entitled Enlisting
Java in the War Against SPAM, Part 2, The Screening Module.
(The program named Pop302 is in Listing 34 in the previous
lesson.)
Compile and execute the
program named Pop302. This should put some message files
in your history directory that can be used to test the program named Pop302d.
Then compile and execute the program named Pop302d from
Listing 30 below. Experiment with it, making changes, and
observing the
results
of your
changes. For example, I recommend that you increase the width of
the GUI to at least 750 pixels, and increase the width of the text
fields and text area to at least 100 characters.
Before executing the two programs, you will need to create three text
files
having the following names and purposes and store them in the folder
containing your compiled Java class files.
- Pop302a.txt – contains offensive Subject line
words and phrases - Pop302b.txt – contains offensive body text words
and phrases - Pop302c.txt – contains friendly Email addresses and friendly Subject
line material
Eventually you will need to populate these files with words
and phrases that work well for you. The program named Pop302d
explained in this lesson will help you to update the file named Pop302a.txt
based on actual SPAM messages in your history directory.
In the meantime, I have provided sample text files in Listings 35,
36, and 37 in the previous lesson entitled Enlisting
Java in the War Against SPAM, Part 2, The Screening Module.
You can use these files as starter lists. If
you receive the same kinds of SPAM that I receive, the words in these
lists should make it possible for you to test the two programs and to
get a few
hits on SPAM messages.
These are simply text files so feel free to add other words and
phrases as appropriate.
(Let me caution you not to enable the
DELE
code in the program named Pop302 until you are certain
that you actually want to delete messages from the server. Once a
message is deleted from the server, there is no way to recover it from
the server.)
Summary
This lesson shows you how to train my SPAM
screening algorithm to do a better job of
identifying SPAM in the future based on the Subject line of SPAM
messages.
The lesson explains a program named Pop302d,
which
uses historical data to train the algorithm.
The GUI used to control the program is shown in Figure 1. The
program
is designed for extreme ease of use. Training the algorithm
consists
mainly of selecting text with the mouse and pressing buttons. This can
be accomplished very quickly with very little effort. Except for
the possible requirement to delete extra characters, no actual typing
is required.
After several weeks of training, my
algorithm is reliably identifying about ninety-five percent of all SPAM
messages, allowing me to delete them from my Email server before
downloading them into my primary Email
client.
Most
of
the messages that make it past the screen are messages written in
a non-English language that I am unable to read. No one that I
know would send a message to me in a
language that I can’t read, so I consider such messages to be
SPAM. They constitute the bulk of the five-percent of the SPAM
messages that make it past the screen.
What’s Next?
Long range forecast
There are a number of spammer tricks that haven’t been addressed so
far in this series of articles. I will be addressing many of them
in future articles. I am also working on programs that combine
spam screening with virus protection. I will be publishing those
programs as well.
HTML
Many SPAM messages are written in HTML. The use of HTML
provides a myriad of opportunities for the spammer to defeat SPAM
blocking software. For example, it is common practice for the
spammer to insert bogus HTML tags into offending words. Because
the tags are not recognized by the HTML rendering engine, they are
simply ignored, causing the offending word to be rendered intact.
This is fairly easy to handle, and I will describe the solution in a
future article.
Encoded Subject lines
The Email system allows for the Subject line to be
encoded in at least two different ways, making it appear to be
meaningless in its raw form. Apparently this is intended to make
it possible to reliably send characters other than those included in
the seven-bit ASCII character set across the Internet. Spammers
commonly use this as a way to disguise their subjects in raw text
form. I will show you how to deal with this.
Encoded body text
The Email system also allows for encoding of the body of the Email
message to make it possible to send graphic images and other types of
material across the Internet. Again, spammers find this to be a
convenient way to disguise their messages in raw text form.
Because Email messages can consist of multiple parts, some encoded
and some not encoded, this is somewhat more difficult to handle than
encoded Subject lines. I will show you how to
deal with this.
Writing the message as HTML entities
Sometimes I encounter SPAM messages where the spammer has
represented most of the characters in the message as HTML
entities. Sometimes this material is also encoded as described
above. I will show you how to deal with this.
Extraneous random material
Another big issue has to do with the inclusion of large numbers of
extraneous random words, random sentence fragments, or simply random
groups of letters in email messages. Apparently this is done to
defeat
statistical SPAM blocking programs such as the one described by Paul Graham.
Also, such material is sometimes inserted into the middle of offending
words and phrases.
I don’t know how effective this is in defeating statistical
programs, but it
is a nuisance to a human observer like myself trying to
identify offending words and phrases within the message. So far,
I
have identified three ways that spammers insert extraneous
material and make it almost invisible to the reader of the
message. (I expect to identify other ways as time goes on.)
Below the </HTML> tag
The most obvious way to hide extraneous material is to add the
extraneous material
below the </HTML> tag. Then it is simply ignored by the
HTML rendering engine. This is relatively easy to deal with
insofar as a human observer is concerned, and would probably be easy
for a statistical processor to deal with as well. I will show you
how I deal with this in my screening algorithm.
Color and size
A more subtle and effective way is to include the extraneous
material in the HTML body, but to cause the color of the extraneous
text to be the same as the background color. That causes it to
become
invisible, but it still occupies space.
A third way is to include the
extraneous text in the HTML body, but to reduce its size so that is
appears as a small irregular dot. This can be combined with
the color idea to cause the dot to become invisible. Then it
becomes hardly noticeable to the reader of the message.
I don’t have an easy solution to the color and size issues yet, but I
am working on them. If I develop a solution, I will explain it to
you.
Non-English messages
After a few weeks of training, my algorithm is currently
identifying about ninety-five percent of all SPAM messages. Most
of
the messages that make it past the screen are messages written in
a non-English language that I am unable to read.
Although I don’t know anything about the intentions of the author (SPAM
or otherwise), since I can’t read the message, I don’t want to see
it. Also, no one that I know would send a message to me in a
language that I can’t read, so I consider such messages to be
SPAM. They constitute the bulk of the five-percent of the SPAM
messages that make it past the screen.
I have some ideas about how to deal with this based on the use of
non-ASCII characters, but I haven’t tried them yet. If these
ideas bear fruit, I will tell you about them.
Other spammer tricks
As time goes on, I expect to identify other tricks used by spammers
to defeat SPAM blocking software. If I identify any that seem to
be significant, I will tell you about them as well.
The next lesson
The next lesson in this series will present and explain my
program named Pop302e, which provides an easy way to
train my screening algorithm to do a better job of identifying SPAM
based on
the body text of a message.
Complete Program Listing
A complete listing of the program is provided in Listing 30.
Disclaimer of responsibility: If you elect to use this
program
you use it at your own risk. Make absolutely certain that you
understand what you are doing before you execute the program.
Inappropriate use could result in the loss of Email messages. The
author of this program, Richard G. Baldwin, accepts no responsibility
for any losses that you may incur as a result of using this program.
/*File Pop302d.java Copyright 2003, R.G.Baldwin |
Copyright 2004, Richard G. Baldwin. Reproduction in whole or
in
part in any form or medium without express written permission from
Richard
Baldwin is prohibited.
About the author
Richard Baldwin
is a college professor (at Austin Community College in Austin, TX) and
private consultant whose primary focus is a combination of Java, C#,
and XML. In addition to the many platform and/or language independent
benefits of Java and C# applications, he believes that a combination of
Java, C#, and XML will become the primary driving force in the delivery
of structured information on the Web.
Richard has participated in numerous consulting projects, and he
frequently provides onsite training at the high-tech companies located
in and around Austin, Texas. He is the author of Baldwin’s
Programming Tutorials, which
has gained a worldwide following among experienced and aspiring
programmers. He has also published articles in JavaPro magazine.
Richard holds an MSEE degree from Southern Methodist University
and has many years of experience in the application of computer
technology to real-world problems.
-end-
