Java Introduction to Memory-Mapped IO in Java

Introduction to Memory-Mapped IO in Java

Java Programming Notes # 1794


Preface

New features in SDK Version 1.4.0

The recently released JavaTM 2 SDK, Standard Edition Version 1.4 contains a
large number of new features, including the concept
of IO channels.  The first lesson in this miniseries, entitled

FileChannel Objects in Java, Background Information
, introduced you to the concept of channels
from a read/write IO viewpoint. 

Mixed primitive types

The previous lesson, entitled
FileChannel
Objects in Java, Records with Mixed Types
, showed you
how to use the FileChannel class along
with the ByteBuffer class to:

  • Create records consisting of sequences of data values of mixed primitive
    types
  • Manipulate those records under program control
  • Transfer those records between the computer’s memory and a physical disk
    file using read/write operations on the channel

Memory-mapped IO

In this lesson, I’m
will teach you how to use the FileChannel class along with the
ByteBuffer
class to perform memory-mapped IO for data of type byte
This is an alternative to the read/write approach discussed in previous
lessons.

This lesson will teach you the basics of memory-mapped IO using data of type
byte

In the next lesson, I will teach you how to do memory-mapped IO for data
records containing mixed types of data.  I will also teach you how to do
memory-mapped IO for different data types using different views of the
data in the buffer.

Viewing tip

You may find it useful to open another copy of this lesson in a separate
browser window.  That will make it easier for you to scroll back and
forth among the different listings and figures while you are reading about
them.

Supplementary material

I recommend that you also study the other lessons in my extensive collection
of online Java tutorials.  You will find those lessons published at
Gamelan.com
However, as of the date of this writing, Gamelan doesn’t maintain a consolidated
index of my Java tutorial lessons, and sometimes they are difficult to
locate there.  You will find a consolidated index at
www.DickBaldwin.com.

What is a FileChannel?

Sun describes an object of the FileChannel class simply as “A
channel for reading, writing, mapping, and manipulating a file.”

Discussion
and Sample Code

Mapping files into memory

Previous lessons have explained the use of FileChannel objects
for reading and writing files.  In this lesson, I will show you how to use FileChannel objects for
mapping files into memory. 

Once you map a file into memory, changes that you make to the memory map are
automatically
propagated
to the file.

Will discuss sample program in fragments

I will illustrate memory-mapped IO for data of type byte using the sample program named
Channel04

As usual, I will discuss this program in fragments.  You will find a complete listing
of the program in Listing 17 near the end of the lesson.

Description of the program

This program,
which was tested using JDK 1.4.0 under Win2000, illustrates the use of FileChannel objects for
memory-mapped IO.

The program first creates and populates a disk file with data of type
byte, closes all connections between the program and the file, and makes
all references to the data eligible for garbage collection.

Then the program creates a memory map of the existing file,
closes the channel used to create the map, and displays the contents of the map.

Then the program modifies the map, which automatically modifies the contents of the
associated file.  (Note that there is no open channel connecting the
file to the program at this point.)

Then
the program reads and displays the modified file.

The above steps illustrate
the basic capability of memory-mapped IO.

Create a
partial read-only map

To illustrate some additional capabilities of
memory-mapped IO, the program then creates a read-only map of
a portion of the modified file and displays it.


The main method

For simplicity, this
program consists of a main method and a static convenience
method used to display the data in a ByteBuffer object.

The first fragment in Listing 1 shows the beginning of the main method.

  public static void main(
                        String[] args){
    byte[] array = {65,66,67,68,69,70};
    
    ByteBuffer buf = 
                ByteBuffer.wrap(array);

Listing 1

The wrap method

In an earlier lesson, you learned that one way to create and populate a
ByteBuffer
object is to invoke the static wrap method of the
ByteBuffer
class to wrap the buffer object around an existing array object
of type byte[](I will show you a different way to create a
ByteBuffer
object later in this lesson.)

The code in Listing 1 performs such a wrapping operation to create a new
ByteBuffer
object, which is populated with six sequential byte values
from 65 through 70 inclusive.  This will be the starting point for the
exercises that follow.

Display the data in the buffer

The code in Listing 2 invokes the showBufferData method to display the
contents of the buffer.

    showBufferData(buf,"buf-raw data");

Listing 2

The showBufferData method was explained in an earlier lesson, and
won’t be discussed further here.  You can view the entire method in Listing
17 near the end of the lesson.

The output

The output produced by Listing 2 is shown in Figure 1.

Data for buf-raw data
65 66 67 68 69 70

Figure 1

This output simply confirms that the contents of the ByteBuffer object are the same as
the contents of the original array object.

(You also learned in an earlier lesson that changes made to the array
object will be reflected in the buffer and vice versa.  In other words, I
can’t tell you how this process is handled in physical memory, but the effect is
as if the array object and the buffer object contain the same data.)

<

Getting a FileChannel object

You can get a FileChannel object that
is connected to an underlying file by invoking the getChannel method
on an existing FileInputStream, FileOutputStream, or RandomAccessFile
object.

A FileChannel object obtained from a FileInputStream object
will be open for reading, but not for writing.  A FileChannel
object obtained from a FileOutputStream object will be open for
writing, but not for reading.

A FileChannel object obtained from a RandomAccessFile
object will be open for reading if the RandomAccessFile object was created in the “r”
mode, and will be open for reading and writing if the object was created
in the “rw” mode.

(Note that creating a RandomAccessFile object in rw mode for
a particular file name does not delete an existing file having that name if such
a file already exists.)

Open channel for reading and writing

The code in Listing 3 begins by deleting a file named junk.txt if such
a file already exists.

Then the code in Listing 3 invokes the getChannel method on an
anonymous
RandomAccessFile
object’s reference to get a FileChannel
object that can read from and write to a new physical file named junk.txt.

      new File("junk.txt").delete();

      FileChannel rwCh = (
                  new RandomAccessFile(
                     "junk.txt","rw")).
                          getChannel();

Listing 3

The FileChannel object’s reference is stored in the reference variable
named rwCh.

(At this point, the file is empty, with a size of zero.)

Populate the physical disk file

The code in Listing 4 writes the contents of the ByteBuffer object to
the physical file.

      System.out.println(
                   "Bytes written = " 
                   + rwCh.write(buf));

Listing 4

As you learned in a previous lesson, the write method of the
FileChannel
class returns the number of bytes written to the file.  The
code in Listing 4 displays this value as shown in Figure 2.

Bytes written = 6

Figure 2

Disconnect from the file

At this point, for purposes of illustration, I elected to close the
FileChannel
object so that it can no longer be used to access the file.

I also elected to
make the original ByteBuffer object and the original array object eligible for garbage collection so that their data is no longer accessible.

Following the execution of the code in Listing 5, the data is available only via the file named
junk.txt.

      rwCh.close();
      buf = null;
      array = null;

Listing 5

Get a new FileChannel object

The code in Listing 6 gets a new FileChannel object for reading and
writing the existing file named junk.txt.

      rwCh = new RandomAccessFile(
                      "junk.txt","rw").
                          getChannel();

Listing 6

Map the file into memory

Now we’ve gotten to the interesting part.  The code in Listing 7 invokes
the map method of the FileChannel class to map the entire
file named junk.txt into a memory area as a ByteBuffer object. 
Then the code in Listing 7 closes the FileChannel object. 

      long fileSize = rwCh.size();

      ByteBuffer mapFile = rwCh.map(
        FileChannel.MapMode.READ_WRITE,
                          0, fileSize);
      rwCh.close();

Listing 7

(Despite the fact that the channel has been closed, the data in the file
continues to be available via the memory map, as will be demonstrated later.)

A more-detailed discussion

The code in Listing 7 is very important, and I don’t want to pass over it
lightly.

Listing 7 begins by invoking the size method on the FileChannel
object’s reference.  The size method returns the current size of the
channel’s file (junk.txt), measured in bytes and returned as type
long.

(At this point, the size of the file is six bytes.)

Invoking the map method

Then the code in Listing 7 uses this size value to invoke the map
method on the FileChannel object’s reference.

The map method deserves some additional discussion. 

The MappedByteBuffer class

To begin with, the map method returns a reference to an object of the MappedByteBuffer
class, which I haven’t discussed prior to this point in this series of tutorials.  Therefore, I will begin by
explaining the characteristics of a MappedByteBuffer object.

A MappedByteBuffer object
extends ByteBuffer, and is a direct byte buffer whose content is a memory-mapped region of a file.

Created via the map factory method

As you saw in Listing 7, mapped byte buffer objects are created via the map
method of the FileChannel class. 

The MappedByteBuffer class
extends the ByteBuffer class, providing operations
that are specific to memory-mapped file regions.  Those operations are:

  • force – Forces any changes made to this buffer’s content to be
    written to the storage device containing the mapped file.
  • isLoaded – Tells whether or not this buffer’s content is resident
    in physical memory.
  • load – Loads this buffer’s content into physical memory.

Valid until garbage collected

A MappedByteBuffer object and the file mapping that it represents remains valid until
the buffer itself is garbage-collected.

Contents can change or become inaccessible

The contents of a MappedByteBuffer object can change at any time. 
This can happen, for example, if the
content of the corresponding region of the mapped
file is changed.

(According to sun, whether or not such changes occur,
and when they occur, is operating-system dependent
and therefore unspecified.)

All or part of a mapped byte buffer may become inaccessible at any time. 
This can happen, for
example, if the mapped file is truncated.

An attempt to
access an inaccessible region of a mapped byte buffer will not change the
buffer’s contents and will cause an unspecified exception to
be thrown either at the time of the access or at some later time.

(According to Sun, it is

strongly recommended that appropriate precautions be
taken to avoid the manipulation of a mapped file by this program, or by a
concurrently running program, except to read or write the
file’s content.)

Behavior similar to ByteBuffer object

Otherwise, MappedByteBuffer objects behave no differently than ordinary direct byte
buffers.

Therefore, most of what you have previously learned about manipulating ByteBuffer
objects, (such as getting different views of a ByteBuffer object), also
applies to MappedByteBuffer objects.

The map method parameters

As you saw in Listing 7, the map method of the FileChannel
class requires three parameters:

  • mode
  • position
  • size

Using these parameters, the map method maps a region of the channel’s
file directly into memory.

Mapping modes

A region of a file may be mapped into memory in one of three modes:

  • read-only
  • read/write
  • private

If the file is mapped in read-only mode, any attempt to modify the contents
of the resulting buffer will cause an exception to be thrown.  In this
case, the channel must have been opened for reading.

In read/write mode, changes made to the contents of the buffer will be
propagated to the file.  (Note, however, that the changes may or may not be
visible to other programs that may have mapped the same file.)
  In this
case, the channel must have been opened for both reading and writing.

In private mode, changes made to the contents of the buffer will not be
propagated to the file.  Similarly, in this case, the channel must have
been opened for both reading and writing.

The position and size parameters of the map method

The position parameter of the map method defines the starting
point in the file that is mapped into the buffer. 

The size property defines the number of bytes that will be mapped.

(The combination of the position and size parameters can be used to define
a region of sequential bytes within the file that will be mapped into memory. 
This will be illustrated later in the sample program.)

Map the entire file into memory

The invocation of the map method in Listing 7 causes the entire file
named junk.txt to be mapped in read/write mode into the ByteBuffer
object referred to by the reference variable named mapFile.

(Since I had no plans to invoke any of the three methods defined in the
MappedByteBuffer class, it was appropriate for me to save the reference
as type ByteBuffer instead of as type MappedByteBuffer.)

The position, limit, and capacity properties

The MappedByteBuffer object returned by the map method will
have a position of zero.  Its limit and capacity will be equal to the
value of the size parameter passed to the map method.

Can close the FileChannel object

Once a mapping is established, it is not dependent on the FileChannel
object that was used to create it.  Closing the channel has no effect upon
the validity of the mapping.

Mapping files is expensive

According to Sun:

"For most operating systems, mapping a file into memory is more expensive
than reading or writing a few tens of kilobytes of data via the usual read and
write methods. From the standpoint of performance it is generally only worth
mapping relatively large files into memory."

Display the memory map

The code in Listing 8 displays the contents of the memory map.  Note
that since the memory map is simply a ByteBuffer object, the same method
used earlier to display the contents of a ByteBuffer object can be used
to display the contents of the memory map.

      showBufferData(
                    mapFile,"mapFile");

Listing 8

The output

The output produced by the code in Listing 8 is shown in Figure 3. 
Happily, the data in the memory map matches the original data indicating that no
data corruption or modification has occurred to this point in the program.

Data for mapFile
65 66 67 68 69 70

Figure 3

Modify values in the memory map

The code in Listing 9 purposely modifies the contents of the memory map,
which in turn, modifies the contents of the disk file represented by the memory
map (changes to the map propagate to the file). 

      mapFile.position(2);
      mapFile.put((byte)1);
      mapFile.put((byte)2);

Listing 9

(Note that the channel that was used to create the map was closed before the
modifications were made in Listing 9.  As mentioned earlier, once the map
is established, the validity of the map is not dependent on the channel used to
establish the map.)

The code in Listing 9 sets the position property of the map to a value of 2
and then executes two sequential relative put operations on the map.  This
causes the values in the third and fourth elements in the map to be changed from
the values of 67 and 68 to the values of 1 and 2.

Display the modified map

The code in Listing 10 displays the contents of the modified memory map.

      showBufferData(
                mapFile,"mod-mapFile");

Listing 10

This produces the output shown in Figure 4

Data for mod-mapFile
65 66 1 2 69 70

Figure 4

As you can see, the third and fourth values have been changed relative to
their original value (boldface added for emphasis).

(At the risk of being redundant, I am going to stress that making this change to
the values in the memory map causes the values in the associated disk file named
junk.txt to change accordingly.)

Confirm modifications to the file

Now it’s time to confirm that the modification of the values in the memory
map caused the values stored in the file to be changed accordingly.

The code in Listing 11 gets a FileChannel object that can be used to
read the contents of the file named junk.txt.

(Note that the channel was opened in read-only mode.)

      FileChannel newInCh = 
                  new RandomAccessFile(
                       "junk.txt","r").
                          getChannel();

Listing 11

Create a ByteBuffer object through allocation

The code in Listing 12 invokes the allocate method of the
ByteBuffer
class to create a new empty
ByteBuffer
object.

      ByteBuffer newBuf = 
                   ByteBuffer.allocate(
                        (int)fileSize);

Listing 12

(What is an empty ByteBuffer object?  Although I can’t say for
absolute certain, I suspect that each byte in the new ByteBuffer object
will have a value of zero.  I base this on the fact that the buffer will
have a backing array of type byte[].  Normally a new array object of
type byte[] will have each of its elements initialized to the default
value of zero (unless they are purposely initialized to some other value). 
Thus, an allocated ByteBuffer object wraps a new array object, which has
default initialization of its elements.  A reference to the backing array
can be obtained by invoking the array method on the ByteBuffer
object.)

An alternative approach

The allocate method provides an alternative to the wrap method
for creating a new ByteBuffer object.  In this case, the size of the
ByteBuffer object was set to match the size of the file, which was
determined earlier in Listing 7.

Populate the ByteBuffer object

The code in Listing 13 uses the read method of the FileChannel
class to read the file contents into the new ByteBuffer object.

      System.out.println(
               "Bytes read = " 
               + newInCh.read(newBuf));

      newInCh.close();

Listing 13

After reading the data from the file into the ByteBuffer object, the code in Listing 13 closes the
FileChannel
object.

Display the data read from the file

The code in Listing 14 invokes the showBufferData method again, this
time to display the data that was read from the disk file into the ByteBuffer
object.

      showBufferData(newBuf,"newBuf");

Listing 14

The output produced by the code in Listings 13 and 14 is shown in Figure 5
(boldface added for emphasis).

Bytes read = 6
Data for newBuf
65 66 1 2 69 70

Figure 5

File data was modified

As you can see, the third and fourth values in the disk file were modified
earlier when changes were made to those two values in the memory map.  This
confirms that changes made to the memory map propagate to the disk file
represented by the map.

Mapping a region of a file

You can map an entire file, or a region within a file into a memory map. 

The code in Listing 15 gets a new read-only FileChannel object, and
uses it to map a region of the same file named junk.txt into a new
read-only ByteBuffer object.

      FileChannel rCh = 
                  new RandomAccessFile(
                       "junk.txt","r").
                          getChannel();

      ByteBuffer roMapFile = rCh.map(
         FileChannel.MapMode.READ_ONLY,
                        1, fileSize-2);
      rCh.close();

Listing 15

The definition of the region

The selected region begins at a position value of 1 and includes a number of
bytes that is two less than the number of bytes in the file.

(Thus, the region includes all but the first and last bytes in the file.)

Close the channel

Having mapped the file into memory, the code in Listing 15 closes the
FileChannel
object used to perform the mapping operation.

Display partial memory map

The code in Listing 16 displays the contents of the partial memory map.

      showBufferData(roMapFile,
                          "roMapFile");

Listing 16

The output produced by the code in Listing 16 is shown in Figure 6.

Data for roMapFile
66 1 2 69

Figure 6

As you can see, only four of the six bytes in the file were mapped into the
memory map.  (The first and last bytes in the file were excluded.)

A read-only map

If this were a read/write memory map, it would be possible to modify any of
those four values in the map and have the changes propagate to the file.

However, this is a read-only map.  Therefore, any attempt to modify any
of the values in the map will cause a ReadOnlyBufferException to be
thrown.

That’s it for now

By now you should understand a quite a lot about the use of FileChannel
objects to create and manipulate memory maps of disk files.

Run the Program


If you haven’t already done so, I encourage you to copy the code from Listing
17 into your text editor, compile it, and execute it.  Experiment
with it, making changes, and observing the results of your changes.

Remember, however, that you must be running Java version 1.4.0 or later
to compile and execute this program.

Summary

In this lesson, I have taught you how to use the FileChannel class along with the
ByteBuffer
class to perform memory-mapped IO for data of type byte
This is an alternative to the read/write approach discussed in previous
lessons.

This lesson teaches the basics of memory-mapped IO using data of type
byte
.

What’s Next?

In the next lesson, I will teach you how to do memory-mapped IO for data
records containing mixed types of data.  I will also teach you how to do
memory-mapped IO for different data types using different views of the data in
a buffer.

Future plans

As time goes on, I plan to publish additional lessons that will help
you learn to use other new IO features including:

  • File locks
  • Character-set encoders and decoders
  • Pattern matching on files using regular expressions
  • Socket channels for both clients and servers
  • Non-blocking reads
  • Non-blocking servers

Complete Program Listing


A complete listing of the program discussed in this lesson is shown in
Listing 17 below.

 

/* File Channel04.java
Copyright 2002, R.G.Baldwin

Illustrates use of FileChannel objects
for mapped-memory IO.

First creates and populates a disk
file with data of type byte.

Then creates a memory map of the
existing file, and modifies the map, 
which modifies the contents of the 
file.

Then reads and displays the modified
file.

Then creates a read-only map of part of
the modified file and displays it.

Tested using JDK 1.4.0 under Win2000

The output is:

Data for buf-raw data
65 66 67 68 69 70
Bytes written = 6
Data for mapFile
65 66 67 68 69 70
Data for mod-mapFile
65 66 1 2 69 70
Read/display modified file
Bytes read = 6
Data for newBuf
65 66 1 2 69 70
Data for roMapFile
66 1 2 69
**************************************/

import java.io.*;
import java.nio.channels.*;
import java.nio.*;

class Channel04{  
  public static void main(
                        String[] args){
    //Create and populate an array obj
    byte[] array = {65,66,67,68,69,70};
    
    //Wrap array in a buffer to create
    // a ByteBuffer object.  Then 
    // display the contents of the
    // buffer.
    ByteBuffer buf = 
                ByteBuffer.wrap(array);
    showBufferData(buf,"buf-raw data");
        
    try{
      //Delete the file named junk.txt
      // if it already exists
      new File("junk.txt").delete();
      //Get a channel for writing a
      // random access file in rw
      // mode.
      FileChannel rwCh = (
                  new RandomAccessFile(
                     "junk.txt","rw")).
                          getChannel();
                          
      //Write buffer data to the file
      System.out.println(
                   "Bytes written = " 
                   + rwCh.write(buf));
                   
      //Close the channel so that it
      // can no longer be used to 
      // access the file.
      rwCh.close();
                   
      //Make original ByteBuffer object
      // and original array object
      // eligible for garbage 
      // collection so that their data
      // is no longer accessible.  The
      // data is now available only
      // via the file named junk.txt.
      buf = null;
      array = null;
      
      //Get a new FileChannel object
      // for reading and writing the 
      // existing file
      rwCh = new RandomAccessFile(
                      "junk.txt","rw").
                          getChannel();

      //Map entire file to memory and
      // close the channel
      long fileSize = rwCh.size();
      ByteBuffer mapFile = rwCh.map(
        FileChannel.MapMode.READ_WRITE,
                          0, fileSize);
      rwCh.close();
      
      //Display contents of memory map
      showBufferData(
                    mapFile,"mapFile");
      //Change contents of the map and
      // hence the file.  Note that
      // the channel is closed.
      mapFile.position(2);
      mapFile.put((byte)1);
      mapFile.put((byte)2);
      
      //Display new contents of memory
      // map
      showBufferData(
                mapFile,"mod-mapFile");

      System.out.println(
         "Read/display modified file");
      //Get new channel for read only
      FileChannel newInCh = 
                  new RandomAccessFile(
                       "junk.txt","r").
                          getChannel();
                          
      //Allocate (don't wrap) a new
      // ByteBuffer
      ByteBuffer newBuf = 
                   ByteBuffer.allocate(
                        (int)fileSize);

      //Read file data into the new
      // buffer, close the channel, and
      // display the data.
      System.out.println(
               "Bytes read = " 
               + newInCh.read(newBuf));
      newInCh.close();
      showBufferData(newBuf,"newBuf");
      
      //Get new read-only partial
      // mapping for file and display
      FileChannel rCh = 
                  new RandomAccessFile(
                       "junk.txt","r").
                          getChannel();
      ByteBuffer roMapFile = rCh.map(
         FileChannel.MapMode.READ_ONLY,
                        1, fileSize-2);
      rCh.close();
      showBufferData(roMapFile,
                          "roMapFile");
      
      //Following put throws java.nio.
      // ReadOnlyBufferException, so
      // it is disabled.
      roMapFile.position(2);
      //roMapFile.put((byte)88);
      
    }catch(Exception e){
      System.out.println(e);}
  }// end main
  //---------------------------------//
  
  static void showBufferData(
          ByteBuffer buf, String name){
    //Displays byte buffer contents
    
    //Save position
    int pos = buf.position();
    //Set position to zero
    buf.position(0);
    System.out.println(
            "Data for " + name);
    while(buf.hasRemaining()){
      System.out.print(
                      buf.get() + " ");
    }//end while loop
    System.out.println();//new line
    //Restore position and return
    buf.position(pos);
  }//end showBufferData
  //---------------------------------//
}//end class Channel04 definition

Listing 17

Copyright 2002, Richard G. Baldwin.  Reproduction in whole or in
part in any form or medium without express written permission from Richard
Baldwin is prohibited.

About the author

Richard Baldwin
is a college professor (at Austin Community College in Austin, TX) and
private consultant whose primary focus is a combination of Java, C#, and
XML. In addition to the many platform and/or language independent benefits
of Java and C# applications, he believes that a combination of Java, C#,
and XML will become the primary driving force in the delivery of structured
information on the Web.

Richard has participated in numerous consulting projects, and he
frequently provides onsite training at the high-tech companies located
in and around Austin, Texas.  He is the author of Baldwin’s Programming
Tutorials,
which has gained a worldwide following among experienced and aspiring programmers.
He has also published articles in JavaPro magazine.

Richard holds an MSEE degree from Southern Methodist University and
has many years of experience in the application of computer technology
to real-world problems.

[email protected]

-end-

 

Latest Posts

Related Stories