JavaEnterprise JavaJava 2 security and stack inspection

Java 2 security and stack inspection content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

With the release of Java 2 (a.k.a., JDK 1.2), Sun Microsystems upped the ante on sophisticated security models for mobile code.

In early Java incarnations, untrusted code was constrained to a security sandbox. Code signing was added to the security toolbox in 1997 with the introduction of JDK 1.1. Together, sandboxing and signatures make for a powerful approach to securing untrusted code. (See Sandboxes and signatures: The future of executable content.) However, even with code signing, the JDK 1.1 trust model is black and white; that is, code is either completely trusted or completely untrusted. Java 2 changes all that by allowing fine-grained security policy and access control enforcement. The access control system in Java 2 is built around the concept of stack inspection.

Java 2 code running on the new VMs can be granted special permissions and have its access checked against policy as it runs.

What’s the Big Idea?

Everyone agrees that code signing makes the Java security model a lot more complicated, not to mention actually using the new system. Where security is concerned, complexity is bad, because it increases the odds of an error in the system’s design or implementation. If we’re going to add all of this complexity, what exactly is it that we are gaining? What’s the main goal?

The main goal is to gain better control over the security of mobile code. We can achieve this goal by winning the battle on three fronts. By adding code signing and expanding beyond a black-and-white trust model, we hope to gain:

  1. The ability to grant privileges when they’re needed.
  2. The ability to have code operate with the minimum necessary privileges.
  3. The ability to closely manage the system’s security configuration.

The Java 2 security model addresses these goals by providing a policy-based security enforcement mechanism based on stack inspection.

At its heart, the Java 2 security model has a simple idea: Make all code run under a security policy that grants different amounts of privilege to different programs. While the idea may be simple, in practice, creating a coherent policy is quite difficult.

Java 2 code running on the new Java VMs can be granted special permissions and have its access checked against policy as it runs. The cornerstone of the system is policy (something that will not surprise security practitioners in the least). Policy can be set by the user (usually a bad idea) or by the system administrator, and is represented in the class
. Herein rests the Achilles’ Heel of Java 2 security. Setting up a coherent policy at a fine-grained level takes experience and security expertise. Today’s harried system administrators are not likely to enjoy this added responsibility. On the other hand, if policy management is left up to users, mistakes are bound to be made. Users have a tendency to prefer “cool” to “secure.”

Code can be signed with multiple keys and can potentially match multiple policy entries.

Executable code is categorized based on its URL of origin and the private keys used to sign the code. The security policy maps a set of access permissions to code characterized by particular origin/signature information. Protection domains can be created on demand and are tied to code with particular


properties. If this paragraph confuses you, imagine trying to create and manage a coherent mobile code security policy!

Code can be signed with multiple keys and can potentially match multiple policy entries. In this case, permissions are granted in an additive fashion.

A simple example

An easy example of how this works in practice is helpful. First, imagine a policy representing the statement “code from ‘’ applet signed by ‘self’ is given permission to read and write files in the directory /applet/tmp and connect to any host in the domain.” Next, a class that is signed by “self” and that originates from “” applet arrives. As the code runs, access control decisions are made based on the permissions defined in the policy. The permissions are stored in permission objects tracked by the Java runtime system. Technically, access control decisions are made with reference to the runtime call stack associated with a thread of computation (more on this below).

Access Control and Stack Inspection

The main business of computer security is controlling access to protected resources. A common approach, taken for decades by security researchers and practitioners, is to set up groups of users and sets of permissions. The idea is to define a logical system in which entities known as principals (often corresponding one-to-one with code owned by users or groups of users) are authorized to access a number of particular protected objects (often system resources such as files).

A good analogy is the notion of user IDs and file permissions found in modern operating systems like Unix and Windows NT. In these systems, logical groupings of users are given particular privileges to read, write, and execute files. These groupings can be used to separate groups so that appropriate boundaries can be placed between them.

Java implements such a system by allowing security-checking code to examine the runtime stack for frames executing untrusted code.

Not surprisingly, Java’s language-based approach to security makes use of groupings and permissions. Sometimes a Java application (say, a Web browser) needs to run untrusted code within itself. In this case, Java system libraries need some way of distinguishing between calls originating in untrusted code and calls originating from the trusted application itself. Clearly, the calls originating in untrusted code need to be restricted to prevent hostile activities. By contrast, calls originating in the application itself should be allowed to proceed (as long as they follow any security rules that the operating system mandates). The question is, how can we implement a system that does this?

Java implements such a system by allowing security-checking code to examine the runtime stack for frames executing untrusted code. Each thread of execution has its own runtime stack (see Figure 1). Security decisions can be made with reference to this check. This is called stack inspection [Wallach and Felten, 1998]. All the major vendors have adopted stack inspection to meet the demand for more flexible security policies than those originally allowed under the old sandbox model. Stack inspection is used by Netscape Navigator, Microsoft Internet Explorer, and Sun Microsystems’ Java 2. (Interestingly, Java is thus the most widespread use of stack inspection for security ever. You can think of it as a very big security-critical experiment.)

Figure 1: A runtime stack tracks method calls.

Simple stack inspection

Netscape 3.0’s stack-inspection-based model (and every other black-and-white security model) is a simple access control system with two principals: system and untrusted. Just to keep things simple, the only privilege available is full.

In this model, every stack frame is labeled with a principal (system if the frame is executing code that is part of the VM or the built-in libraries and untrusted otherwise). Each stack frame also includes a flag that specifies whether privilege is full. A system class can set this flag, thus enabling its privilege. This need only be done when something dangerous must occur – something that not every piece of code should be allowed to do. Untrusted code is not allowed to set the flag. Whenever a stack frame completes its work, its flag (if it has one) disappears.

Every method about to do something potentially dangerous is forced to submit to a stack inspection. The stack inspection is used to decide whether the dangerous activity should be allowed. The stack inspection algorithm searches the frames on the caller’s stack in sequence from the newest to the oldest. If the search encounters an untrusted stack frame (which as we know can never get a privilege flag), the search terminates, access is forbidden, and an exception is thrown. The search also terminates if a system stack frame with a privilege flag is encountered. In this case, access is allowed (see Figure 2).

In the example shown in Figure 2, each stack is made of frames with three parts: a privilege flag (where full privilege is denoted by an X), a principal entry (untrusted or system), and a method. In STACK A, an untrusted applet is attempting to use the
method to access a file in the browser’s cache. The VM makes a decision regarding whether to set the privilege flag (which it does) by looking at the parameters in the actual method invocation. Since the file in this case is a cache file, access is allowed. In short, a system-level method is doing something potentially dangerous on behalf of the untrusted code. In STACK B, an untrusted applet is also attempting to use the
method; however, in this case, the file argument is not a browser cache file but a normal file in the filesystem. Untrusted code is not allowed to do this, so the privilege flag is not set by the VM and access is denied.

Figure 2: Two examples of simple stack inspection.

Real stack inspection

The simple example of stack inspection just given is only powerful enough to implement black-and-white trust models. Code is either fully trusted (and granted full permission at the same level as the application) or untrusted (and allowed no permission to carry out dangerous operations). However, what we want is the ability to create a shades-of-gray trust model. How can we do that?

It turns out that if we generalize the simple model, we get what we need. The first step is to add the ability to have multiple principals. Then we need to have many more specific permissions than full. These two capabilities allow us to have a complex system in which different principals can have different degrees of permission in (and hence, access to) the system.

Research into stack inspection shows that four basic primitives are all that are required to implement a real stack inspection system. In particular, see Dan Wallach’s Ph.D. thesis at Princeton [Wallach, 1998] and the paper Understanding Java Stack Inspection [Wallach and Felten, 1998]. Each of the major vendors uses different names for these primitives, but they all boil down to the same four essential operations (all explained more fully in the following discussions):

  • enablePrivilege()
  • disablePrivilege()
  • checkPrivilege()
  • revertPrivilege()

Some resources such as the file system or network sockets need to be protected from use (and possible abuse) by untrusted code. These resources are protected by permissions. Before code (trusted or otherwise) is allowed access to one of these resources, say, R, the system must make sure to call


All three major Java vendors implement a very similar (and simple) stack-inspection algorithm.

Java libraries are set up in such a way that dangerous operations must go through a Security Manager check before they can occur [McGraw and Felten, 1999]. The Java API provides all calls necessary to implement a virtual OS, thus making isolation of all required security checks possible within the API. When a dangerous call is made to the Java API, the Security Manager is queried by the code defining the base classes. The

method is used to help make behind-the-scenes access control decisions in a very similar fashion. To achieve backwards compatibility, the Security Manager can be implemented using the four stack inspection primitives.

When code wants to make use of some resource, R, it must first call

. When this method is invoked, a check of local policy occurs that determines whether the caller is permitted to use R. If the use is permitted, the current stack frame is annotated with an

mark. This allows the code to use the resource normally.

Permission to use the resource does not last forever; if it did, the system would not work. There are two ways in which the privilege annotation is discarded. One way is for the call to return. In this case, the annotation is discarded along with the stack frame. The second way is for the code to make an explicit call to


. The latter call creates a stack annotation that can hide an earlier enabled privilege. The former simply removes annotations from the current stack frame.

All three major Java vendors implement a very similar (and simple) stack-inspection algorithm. A generalization of this algorithm, after Wallach, is shown in the Listing [Wallach and Felten, 1998].

The question is, if and when will Java 2 security mechanisms be adopted in the field?

The algorithm searches stack frames on the caller’s stack, in order, from newest to oldest. If the search finds a stack frame with the appropriate enabled-privilege annotation, it terminates, allowing access. If the search finds a stack frame that is forbidden from accessing the target by local policy, or has explicitly disabled its privileges, the search terminates, forbidding access.

It may seem strange that the vendors take different actions when the search reaches the end of the stack without meeting any of the conditions (sometimes called falling off the end of the stack). Netscape denies permission, while both Microsoft and Sun allow permission. This difference has to do with backward compatibility. The Netscape choice causes legacy code to be treated like an old-fashioned applet, and confined to the sandbox. The Microsoft/Sun choice allows a signed Java application to use its privileges, even without explicitly marking its stack frames, thus making it easy to migrate existing applications. Since Netscape did not support applications, they felt no need to follow the Microsoft/Sun approach and, instead, chose the more conservative course of denying permission. For more implementation detail on the three vendors’ different code-signing schemes, see Appendix C of Securing Java [McGraw and Felten, 1999].

Listing 1. An algorithm for stack inspection.


// loop, newest to oldest stack frame

foreach stackFrame{

if (local policy forbids access to target

by class executing in stackFrame)

throw ForbiddenException;

if (stackFrame has enabled privilege for target)

return; // allow access

if (stackFrame has disabled privilege for target)

throw ForbiddenException;


// if we reach here, we have fallen off the end of the stack

if (Netscape)

throw ForbiddenException;

if (Microsoft IE || Sun JDK 1.2)

return; // allow access


Formalizing stack inspection

Members of Princeton University’s Secure Internet Programming team (in particular, Dan Wallach and Edward Felten) have created a formal model of Java’s stack inspection system in a belief logic known as ABPL (designed by Abadi, Burrows, Lampson, and Plotkin) [Abadi et al., 1993]. Using the model, the Princeton team demonstrates how Java’s access control decisions correspond to proving statements in ABPL. Besides putting Java’s stack inspection system on solid theoretical footing, the work demonstrates a very efficient way to implement stack inspection systems as pushdown automata using security-passing style. Interested readers should see [Wallach and Felten, 1998], which is available through the Princeton University Web site.

Will Stack Inspection Succeed?

Java 2, with its built-in stack inspection approach to access control, is a very sophisticated security architecture. The question is, if and when will Java 2 security mechanisms be adopted in the field?

Given the inadequate PKI, especially Internet-wide, I predict Java 2 adoption will unfold as follows. (I reserve the right to be completely wrong, of course.) Early Java 2 adopters will institute Java 2 policies inside their organizations using an in-house PKI. This will allow them to sidestep issues surrounding certificate validation chains to multiple issuing authorities, certificate revocation and aging, and other concerns that always crop up when using a PKI in the real world. Only after these organizations have seen some level of success will business-to-business PKIs begin to appear. Business-to-business connectivity will leverage lessons learned from using Virtual Private Networks (VPNs). Finally, Java 2 may find widespread acceptance on the Internet as a whole, but only after a PKI has been established for some other reason (say encrypted e-mail or IPv6).


  • Abadi, M., Burrows, M., Lampson, B., and Plotkin, G. (1993), A calculus for access control in distributed systems. ACM Transactions on Programming Languages and Systems, 15(4):706-734, September 1993.
  • Gary McGraw and Ed Felten (1999), Securing Java: Getting down to business with mobile code, John Wiley & Sons, NY.
  • Dan Wallach and Ed Felten (1998), “Understanding Java Stack Inspection.” In Proceedings of the IEEE Symposium on Security and Privacy, May 1998.
  • Wallach, D. (1998), A New Approach to Mobile Code Security. Ph.D. dissertation, Department of Computer Sciences, Princeton University.


Portions of this article are taken by permission from Securing Java: Getting down to business with mobile code (John Wiley & Sons, 1998).

Gary McGraw, Ph.D., is vice president of corporate technology at Reliable Software Technologies. Dr. McGraw is a noted authority on mobile code security and co-authored both Java Security: Hostile Applets, Holes, & Antidotes (John Wiley & Sons, 1996) and Securing Java: Getting down to business with mobile code (Wiley, 1999), with Prof. Ed Felten of Princeton University. Dr. McGraw has written over fifty peer-reviewed technical publications, consults with major e-commerce vendors, including Visa, and researches software assurance technologies.

Get the Free Newsletter!

Subscribe to Developer Insider for top news, trends & analysis

Latest Posts

Related Stories