http://www.developer.com/

Back to article

Introduction to Memory-Mapped IO in Java


November 26, 2002

Java Programming Notes # 1794


Preface

New features in SDK Version 1.4.0

The recently released JavaTM 2 SDK, Standard Edition Version 1.4 contains a large number of new features, including the concept of IO channels.  The first lesson in this miniseries, entitled FileChannel Objects in Java, Background Information, introduced you to the concept of channels from a read/write IO viewpoint. 

Mixed primitive types

The previous lesson, entitled FileChannel Objects in Java, Records with Mixed Types, showed you how to use the FileChannel class along with the ByteBuffer class to:

  • Create records consisting of sequences of data values of mixed primitive types
  • Manipulate those records under program control
  • Transfer those records between the computer's memory and a physical disk file using read/write operations on the channel

Memory-mapped IO

In this lesson, I'm will teach you how to use the FileChannel class along with the ByteBuffer class to perform memory-mapped IO for data of type byte.  This is an alternative to the read/write approach discussed in previous lessons.

This lesson will teach you the basics of memory-mapped IO using data of type byte.  In the next lesson, I will teach you how to do memory-mapped IO for data records containing mixed types of data.  I will also teach you how to do memory-mapped IO for different data types using different views of the data in the buffer.

Viewing tip

You may find it useful to open another copy of this lesson in a separate browser window.  That will make it easier for you to scroll back and forth among the different listings and figures while you are reading about them.

Supplementary material

I recommend that you also study the other lessons in my extensive collection of online Java tutorials.  You will find those lessons published at Gamelan.com.  However, as of the date of this writing, Gamelan doesn't maintain a consolidated index of my Java tutorial lessons, and sometimes they are difficult to locate there.  You will find a consolidated index at www.DickBaldwin.com.

What is a FileChannel?

Sun describes an object of the FileChannel class simply as "A channel for reading, writing, mapping, and manipulating a file."

Discussion and Sample Code

Mapping files into memory

Previous lessons have explained the use of FileChannel objects for reading and writing files.  In this lesson, I will show you how to use FileChannel objects for mapping files into memory. 

Once you map a file into memory, changes that you make to the memory map are automatically propagated to the file.

Will discuss sample program in fragments

I will illustrate memory-mapped IO for data of type byte using the sample program named Channel04.  As usual, I will discuss this program in fragments.  You will find a complete listing of the program in Listing 17 near the end of the lesson.

Description of the program

This program, which was tested using JDK 1.4.0 under Win2000, illustrates the use of FileChannel objects for memory-mapped IO.

The program first creates and populates a disk file with data of type byte, closes all connections between the program and the file, and makes all references to the data eligible for garbage collection.

Then the program creates a memory map of the existing file, closes the channel used to create the map, and displays the contents of the map.

Then the program modifies the map, which automatically modifies the contents of the associated file.  (Note that there is no open channel connecting the file to the program at this point.)

Then the program reads and displays the modified file.

The above steps illustrate the basic capability of memory-mapped IO.

Create a partial read-only map

To illustrate some additional capabilities of memory-mapped IO, the program then creates a read-only map of a portion of the modified file and displays it.

The main method

For simplicity, this program consists of a main method and a static convenience method used to display the data in a ByteBuffer object.

The first fragment in Listing 1 shows the beginning of the main method.

  public static void main(
                        String[] args){
    byte[] array = {65,66,67,68,69,70};
    
    ByteBuffer buf = 
                ByteBuffer.wrap(array);

Listing 1

The wrap method

In an earlier lesson, you learned that one way to create and populate a ByteBuffer object is to invoke the static wrap method of the ByteBuffer class to wrap the buffer object around an existing array object of type byte[](I will show you a different way to create a ByteBuffer object later in this lesson.)

The code in Listing 1 performs such a wrapping operation to create a new ByteBuffer object, which is populated with six sequential byte values from 65 through 70 inclusive.  This will be the starting point for the exercises that follow.

Display the data in the buffer

The code in Listing 2 invokes the showBufferData method to display the contents of the buffer.

    showBufferData(buf,"buf-raw data");

Listing 2

The showBufferData method was explained in an earlier lesson, and won't be discussed further here.  You can view the entire method in Listing 17 near the end of the lesson.

The output

The output produced by Listing 2 is shown in Figure 1.

Data for buf-raw data
65 66 67 68 69 70

Figure 1

This output simply confirms that the contents of the ByteBuffer object are the same as the contents of the original array object.

(You also learned in an earlier lesson that changes made to the array object will be reflected in the buffer and vice versa.  In other words, I can't tell you how this process is handled in physical memory, but the effect is as if the array object and the buffer object contain the same data.)

<

Getting a FileChannel object

You can get a FileChannel object that is connected to an underlying file by invoking the getChannel method on an existing FileInputStream, FileOutputStream, or RandomAccessFile object.

A FileChannel object obtained from a FileInputStream object will be open for reading, but not for writing.  A FileChannel object obtained from a FileOutputStream object will be open for writing, but not for reading.

A FileChannel object obtained from a RandomAccessFile object will be open for reading if the RandomAccessFile object was created in the "r" mode, and will be open for reading and writing if the object was created in the "rw" mode.

(Note that creating a RandomAccessFile object in rw mode for a particular file name does not delete an existing file having that name if such a file already exists.)

Open channel for reading and writing

The code in Listing 3 begins by deleting a file named junk.txt if such a file already exists.

Then the code in Listing 3 invokes the getChannel method on an anonymous RandomAccessFile object's reference to get a FileChannel object that can read from and write to a new physical file named junk.txt.

      new File("junk.txt").delete();

      FileChannel rwCh = (
                  new RandomAccessFile(
                     "junk.txt","rw")).
                          getChannel();

Listing 3

The FileChannel object's reference is stored in the reference variable named rwCh.

(At this point, the file is empty, with a size of zero.)

Populate the physical disk file

The code in Listing 4 writes the contents of the ByteBuffer object to the physical file.

      System.out.println(
                   "Bytes written = " 
                   + rwCh.write(buf));

Listing 4

As you learned in a previous lesson, the write method of the FileChannel class returns the number of bytes written to the file.  The code in Listing 4 displays this value as shown in Figure 2.

Bytes written = 6

Figure 2

Disconnect from the file

At this point, for purposes of illustration, I elected to close the FileChannel object so that it can no longer be used to access the file.

I also elected to make the original ByteBuffer object and the original array object eligible for garbage collection so that their data is no longer accessible.

Following the execution of the code in Listing 5, the data is available only via the file named junk.txt.

      rwCh.close();
      buf = null;
      array = null;

Listing 5

Get a new FileChannel object

The code in Listing 6 gets a new FileChannel object for reading and writing the existing file named junk.txt.

      rwCh = new RandomAccessFile(
                      "junk.txt","rw").
                          getChannel();

Listing 6

Map the file into memory

Now we've gotten to the interesting part.  The code in Listing 7 invokes the map method of the FileChannel class to map the entire file named junk.txt into a memory area as a ByteBuffer object.  Then the code in Listing 7 closes the FileChannel object. 

      long fileSize = rwCh.size();

      ByteBuffer mapFile = rwCh.map(
        FileChannel.MapMode.READ_WRITE,
                          0, fileSize);
      rwCh.close();

Listing 7

(Despite the fact that the channel has been closed, the data in the file continues to be available via the memory map, as will be demonstrated later.)

A more-detailed discussion

The code in Listing 7 is very important, and I don't want to pass over it lightly.

Listing 7 begins by invoking the size method on the FileChannel object's reference.  The size method returns the current size of the channel's file (junk.txt), measured in bytes and returned as type long.

(At this point, the size of the file is six bytes.)

Invoking the map method

Then the code in Listing 7 uses this size value to invoke the map method on the FileChannel object's reference.

The map method deserves some additional discussion. 

The MappedByteBuffer class

To begin with, the map method returns a reference to an object of the MappedByteBuffer class, which I haven't discussed prior to this point in this series of tutorials.  Therefore, I will begin by explaining the characteristics of a MappedByteBuffer object.

A MappedByteBuffer object extends ByteBuffer, and is a direct byte buffer whose content is a memory-mapped region of a file.

Created via the map factory method

As you saw in Listing 7, mapped byte buffer objects are created via the map method of the FileChannel class. 

The MappedByteBuffer class extends the ByteBuffer class, providing operations that are specific to memory-mapped file regions.  Those operations are:

  • force - Forces any changes made to this buffer's content to be written to the storage device containing the mapped file.
  • isLoaded - Tells whether or not this buffer's content is resident in physical memory.
  • load - Loads this buffer's content into physical memory.

Valid until garbage collected

A MappedByteBuffer object and the file mapping that it represents remains valid until the buffer itself is garbage-collected.

Contents can change or become inaccessible

The contents of a MappedByteBuffer object can change at any time.  This can happen, for example, if the content of the corresponding region of the mapped file is changed.

(According to sun, whether or not such changes occur, and when they occur, is operating-system dependent and therefore unspecified.)

All or part of a mapped byte buffer may become inaccessible at any time.  This can happen, for example, if the mapped file is truncated.

An attempt to access an inaccessible region of a mapped byte buffer will not change the buffer's contents and will cause an unspecified exception to be thrown either at the time of the access or at some later time.

(According to Sun, it is strongly recommended that appropriate precautions be taken to avoid the manipulation of a mapped file by this program, or by a concurrently running program, except to read or write the file's content.)

Behavior similar to ByteBuffer object

Otherwise, MappedByteBuffer objects behave no differently than ordinary direct byte buffers.

Therefore, most of what you have previously learned about manipulating ByteBuffer objects, (such as getting different views of a ByteBuffer object), also applies to MappedByteBuffer objects.

The map method parameters

As you saw in Listing 7, the map method of the FileChannel class requires three parameters:

  • mode
  • position
  • size

Using these parameters, the map method maps a region of the channel's file directly into memory.

Mapping modes

A region of a file may be mapped into memory in one of three modes:

  • read-only
  • read/write
  • private

If the file is mapped in read-only mode, any attempt to modify the contents of the resulting buffer will cause an exception to be thrown.  In this case, the channel must have been opened for reading.

In read/write mode, changes made to the contents of the buffer will be propagated to the file.  (Note, however, that the changes may or may not be visible to other programs that may have mapped the same file.)  In this case, the channel must have been opened for both reading and writing.

In private mode, changes made to the contents of the buffer will not be propagated to the file.  Similarly, in this case, the channel must have been opened for both reading and writing.

The position and size parameters of the map method

The position parameter of the map method defines the starting point in the file that is mapped into the buffer. 

The size property defines the number of bytes that will be mapped.

(The combination of the position and size parameters can be used to define a region of sequential bytes within the file that will be mapped into memory.  This will be illustrated later in the sample program.)

Map the entire file into memory

The invocation of the map method in Listing 7 causes the entire file named junk.txt to be mapped in read/write mode into the ByteBuffer object referred to by the reference variable named mapFile.

(Since I had no plans to invoke any of the three methods defined in the MappedByteBuffer class, it was appropriate for me to save the reference as type ByteBuffer instead of as type MappedByteBuffer.)

The position, limit, and capacity properties

The MappedByteBuffer object returned by the map method will have a position of zero.  Its limit and capacity will be equal to the value of the size parameter passed to the map method.

Can close the FileChannel object

Once a mapping is established, it is not dependent on the FileChannel object that was used to create it.  Closing the channel has no effect upon the validity of the mapping.

Mapping files is expensive

According to Sun:

"For most operating systems, mapping a file into memory is more expensive than reading or writing a few tens of kilobytes of data via the usual read and write methods. From the standpoint of performance it is generally only worth mapping relatively large files into memory."

Display the memory map

The code in Listing 8 displays the contents of the memory map.  Note that since the memory map is simply a ByteBuffer object, the same method used earlier to display the contents of a ByteBuffer object can be used to display the contents of the memory map.

      showBufferData(
                    mapFile,"mapFile");

Listing 8

The output

The output produced by the code in Listing 8 is shown in Figure 3.  Happily, the data in the memory map matches the original data indicating that no data corruption or modification has occurred to this point in the program.

Data for mapFile
65 66 67 68 69 70

Figure 3

Modify values in the memory map

The code in Listing 9 purposely modifies the contents of the memory map, which in turn, modifies the contents of the disk file represented by the memory map (changes to the map propagate to the file). 

      mapFile.position(2);
      mapFile.put((byte)1);
      mapFile.put((byte)2);

Listing 9

(Note that the channel that was used to create the map was closed before the modifications were made in Listing 9.  As mentioned earlier, once the map is established, the validity of the map is not dependent on the channel used to establish the map.)

The code in Listing 9 sets the position property of the map to a value of 2 and then executes two sequential relative put operations on the map.  This causes the values in the third and fourth elements in the map to be changed from the values of 67 and 68 to the values of 1 and 2.

Display the modified map

The code in Listing 10 displays the contents of the modified memory map.

      showBufferData(
                mapFile,"mod-mapFile");

Listing 10

This produces the output shown in Figure 4

Data for mod-mapFile
65 66 1 2 69 70

Figure 4

As you can see, the third and fourth values have been changed relative to their original value (boldface added for emphasis).

(At the risk of being redundant, I am going to stress that making this change to the values in the memory map causes the values in the associated disk file named junk.txt to change accordingly.)

Confirm modifications to the file

Now it's time to confirm that the modification of the values in the memory map caused the values stored in the file to be changed accordingly.

The code in Listing 11 gets a FileChannel object that can be used to read the contents of the file named junk.txt.

(Note that the channel was opened in read-only mode.)

      FileChannel newInCh = 
                  new RandomAccessFile(
                       "junk.txt","r").
                          getChannel();

Listing 11

Create a ByteBuffer object through allocation

The code in Listing 12 invokes the allocate method of the ByteBuffer class to create a new empty ByteBuffer object.

      ByteBuffer newBuf = 
                   ByteBuffer.allocate(
                        (int)fileSize);

Listing 12

(What is an empty ByteBuffer object?  Although I can't say for absolute certain, I suspect that each byte in the new ByteBuffer object will have a value of zero.  I base this on the fact that the buffer will have a backing array of type byte[].  Normally a new array object of type byte[] will have each of its elements initialized to the default value of zero (unless they are purposely initialized to some other value).  Thus, an allocated ByteBuffer object wraps a new array object, which has default initialization of its elements.  A reference to the backing array can be obtained by invoking the array method on the ByteBuffer object.)

An alternative approach

The allocate method provides an alternative to the wrap method for creating a new ByteBuffer object.  In this case, the size of the ByteBuffer object was set to match the size of the file, which was determined earlier in Listing 7.

Populate the ByteBuffer object

The code in Listing 13 uses the read method of the FileChannel class to read the file contents into the new ByteBuffer object.

      System.out.println(
               "Bytes read = " 
               + newInCh.read(newBuf));

      newInCh.close();

Listing 13

After reading the data from the file into the ByteBuffer object, the code in Listing 13 closes the FileChannel object.

Display the data read from the file

The code in Listing 14 invokes the showBufferData method again, this time to display the data that was read from the disk file into the ByteBuffer object.

      showBufferData(newBuf,"newBuf");

Listing 14

The output produced by the code in Listings 13 and 14 is shown in Figure 5 (boldface added for emphasis).

Bytes read = 6
Data for newBuf
65 66 1 2 69 70

Figure 5

File data was modified

As you can see, the third and fourth values in the disk file were modified earlier when changes were made to those two values in the memory map.  This confirms that changes made to the memory map propagate to the disk file represented by the map.

Mapping a region of a file

You can map an entire file, or a region within a file into a memory map. 

The code in Listing 15 gets a new read-only FileChannel object, and uses it to map a region of the same file named junk.txt into a new read-only ByteBuffer object.

      FileChannel rCh = 
                  new RandomAccessFile(
                       "junk.txt","r").
                          getChannel();

      ByteBuffer roMapFile = rCh.map(
         FileChannel.MapMode.READ_ONLY,
                        1, fileSize-2);
      rCh.close();

Listing 15

The definition of the region

The selected region begins at a position value of 1 and includes a number of bytes that is two less than the number of bytes in the file.

(Thus, the region includes all but the first and last bytes in the file.)

Close the channel

Having mapped the file into memory, the code in Listing 15 closes the FileChannel object used to perform the mapping operation.

Display partial memory map

The code in Listing 16 displays the contents of the partial memory map.

      showBufferData(roMapFile,
                          "roMapFile");

Listing 16

The output produced by the code in Listing 16 is shown in Figure 6.

Data for roMapFile
66 1 2 69

Figure 6

As you can see, only four of the six bytes in the file were mapped into the memory map.  (The first and last bytes in the file were excluded.)

A read-only map

If this were a read/write memory map, it would be possible to modify any of those four values in the map and have the changes propagate to the file.

However, this is a read-only map.  Therefore, any attempt to modify any of the values in the map will cause a ReadOnlyBufferException to be thrown.

That's it for now

By now you should understand a quite a lot about the use of FileChannel objects to create and manipulate memory maps of disk files.

Run the Program

If you haven't already done so, I encourage you to copy the code from Listing 17 into your text editor, compile it, and execute it.  Experiment with it, making changes, and observing the results of your changes.

Remember, however, that you must be running Java version 1.4.0 or later to compile and execute this program.

Summary

In this lesson, I have taught you how to use the FileChannel class along with the ByteBuffer class to perform memory-mapped IO for data of type byte.  This is an alternative to the read/write approach discussed in previous lessons.

This lesson teaches the basics of memory-mapped IO using data of type byte.

What's Next?

In the next lesson, I will teach you how to do memory-mapped IO for data records containing mixed types of data.  I will also teach you how to do memory-mapped IO for different data types using different views of the data in a buffer.

Future plans

As time goes on, I plan to publish additional lessons that will help you learn to use other new IO features including:

  • File locks
  • Character-set encoders and decoders
  • Pattern matching on files using regular expressions
  • Socket channels for both clients and servers
  • Non-blocking reads
  • Non-blocking servers

Complete Program Listing

A complete listing of the program discussed in this lesson is shown in Listing 17 below.
 
/* File Channel04.java
Copyright 2002, R.G.Baldwin

Illustrates use of FileChannel objects
for mapped-memory IO.

First creates and populates a disk
file with data of type byte.

Then creates a memory map of the
existing file, and modifies the map, 
which modifies the contents of the 
file.

Then reads and displays the modified
file.

Then creates a read-only map of part of
the modified file and displays it.

Tested using JDK 1.4.0 under Win2000

The output is:

Data for buf-raw data
65 66 67 68 69 70
Bytes written = 6
Data for mapFile
65 66 67 68 69 70
Data for mod-mapFile
65 66 1 2 69 70
Read/display modified file
Bytes read = 6
Data for newBuf
65 66 1 2 69 70
Data for roMapFile
66 1 2 69
**************************************/

import java.io.*;
import java.nio.channels.*;
import java.nio.*;

class Channel04{  
  public static void main(
                        String[] args){
    //Create and populate an array obj
    byte[] array = {65,66,67,68,69,70};
    
    //Wrap array in a buffer to create
    // a ByteBuffer object.  Then 
    // display the contents of the
    // buffer.
    ByteBuffer buf = 
                ByteBuffer.wrap(array);
    showBufferData(buf,"buf-raw data");
        
    try{
      //Delete the file named junk.txt
      // if it already exists
      new File("junk.txt").delete();
      //Get a channel for writing a
      // random access file in rw
      // mode.
      FileChannel rwCh = (
                  new RandomAccessFile(
                     "junk.txt","rw")).
                          getChannel();
                          
      //Write buffer data to the file
      System.out.println(
                   "Bytes written = " 
                   + rwCh.write(buf));
                   
      //Close the channel so that it
      // can no longer be used to 
      // access the file.
      rwCh.close();
                   
      //Make original ByteBuffer object
      // and original array object
      // eligible for garbage 
      // collection so that their data
      // is no longer accessible.  The
      // data is now available only
      // via the file named junk.txt.
      buf = null;
      array = null;
      
      //Get a new FileChannel object
      // for reading and writing the 
      // existing file
      rwCh = new RandomAccessFile(
                      "junk.txt","rw").
                          getChannel();

      //Map entire file to memory and
      // close the channel
      long fileSize = rwCh.size();
      ByteBuffer mapFile = rwCh.map(
        FileChannel.MapMode.READ_WRITE,
                          0, fileSize);
      rwCh.close();
      
      //Display contents of memory map
      showBufferData(
                    mapFile,"mapFile");
      //Change contents of the map and
      // hence the file.  Note that
      // the channel is closed.
      mapFile.position(2);
      mapFile.put((byte)1);
      mapFile.put((byte)2);
      
      //Display new contents of memory
      // map
      showBufferData(
                mapFile,"mod-mapFile");

      System.out.println(
         "Read/display modified file");
      //Get new channel for read only
      FileChannel newInCh = 
                  new RandomAccessFile(
                       "junk.txt","r").
                          getChannel();
                          
      //Allocate (don't wrap) a new
      // ByteBuffer
      ByteBuffer newBuf = 
                   ByteBuffer.allocate(
                        (int)fileSize);

      //Read file data into the new
      // buffer, close the channel, and
      // display the data.
      System.out.println(
               "Bytes read = " 
               + newInCh.read(newBuf));
      newInCh.close();
      showBufferData(newBuf,"newBuf");
      
      //Get new read-only partial
      // mapping for file and display
      FileChannel rCh = 
                  new RandomAccessFile(
                       "junk.txt","r").
                          getChannel();
      ByteBuffer roMapFile = rCh.map(
         FileChannel.MapMode.READ_ONLY,
                        1, fileSize-2);
      rCh.close();
      showBufferData(roMapFile,
                          "roMapFile");
      
      //Following put throws java.nio.
      // ReadOnlyBufferException, so
      // it is disabled.
      roMapFile.position(2);
      //roMapFile.put((byte)88);
      
    }catch(Exception e){
      System.out.println(e);}
  }// end main
  //---------------------------------//
  
  static void showBufferData(
          ByteBuffer buf, String name){
    //Displays byte buffer contents
    
    //Save position
    int pos = buf.position();
    //Set position to zero
    buf.position(0);
    System.out.println(
            "Data for " + name);
    while(buf.hasRemaining()){
      System.out.print(
                      buf.get() + " ");
    }//end while loop
    System.out.println();//new line
    //Restore position and return
    buf.position(pos);
  }//end showBufferData
  //---------------------------------//
}//end class Channel04 definition

Listing 17


Copyright 2002, Richard G. Baldwin.  Reproduction in whole or in part in any form or medium without express written permission from Richard Baldwin is prohibited.

About the author

Richard Baldwin is a college professor (at Austin Community College in Austin, TX) and private consultant whose primary focus is a combination of Java, C#, and XML. In addition to the many platform and/or language independent benefits of Java and C# applications, he believes that a combination of Java, C#, and XML will become the primary driving force in the delivery of structured information on the Web.

Richard has participated in numerous consulting projects, and he frequently provides onsite training at the high-tech companies located in and around Austin, Texas.  He is the author of Baldwin's Programming Tutorials, which has gained a worldwide following among experienced and aspiring programmers. He has also published articles in JavaPro magazine.

Richard holds an MSEE degree from Southern Methodist University and has many years of experience in the application of computer technology to real-world problems.

Baldwin@DickBaldwin.com

-end-
 

Sitemap | Contact Us

Thanks for your registration, follow us on our social networks to keep up-to-date