JavaEnterprise JavaSecuring Virtual Private Networks (VPN)

Securing Virtual Private Networks (VPN) content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

This article introduces you to the Virtual Private Networks, the demand of today’s growing networks, and its endangered security. This article highlights the following topics:


The widespread range of networks and the less expensive availability of Web access to the public gives rise to different security hazards that become bottlenecks for some important corporate, business, and military transactions to be performed on public data networks. Therefore, some multimillionaire sectors can afford to buy the complete data link and dedicate it for their private and secret data transfers; but what will the others do? They cannot afford to have such a dedicated link. Here comes the advent of Virtual Private Networks to the rescue and safeguard the private and secret data of these organizations.

What Is VPN?

Data transference on a shared network such as public data networks, for example, ATM (Asynchronous Transfer Mode) networks, Frame Relay networks, and IP (Internet Protocol) networks on which data is delivered securely by applying some security measures on the data packets and the machines on the path, for example hosts (source computer and destination computer), routers (such as gateway routers and peer routers), and bridges.

Security Threats to Public Networks

The most vulnerable threats found in networks to the secured and stable packet delivery are as follows:

  1. Loss of Privacy
  2. Impersonation
  3. Loss of Integrity
  4. Denial of Service

The solutions of the above described security threats are possible and applied in Virtual Private Networks so that they are not vulnerable any more.

  1. To provide confidentiality for avoiding loss of privacy through the use of encryption or the art of cryptography.
  2. To provide a source of authenticity to avoid impersonation.
  3. To provide integrity.
  4. To provide satisfactory service at a reasonable cost.

How Intruders Break Into the Network System

Generally, an intruder follows these steps to break into a network or a computer:

Outside reconnaissance:

The intruder does this by finding public information or appearing as a normal user. At this stage, it is difficult to detect them. The intruder might walk through your DNS tables to find the names of your machines. The intruder might search news articles and press releases about your company.

Inside reconnaissance:

The intruder scans for information, but still doesn’t do anything harmful. If required, they walk through all your Web pages and look for CGI scripts (CGI scripts are often easily hacked).

Exploit the system:

The intruder starts exploiting possible holes in the target machines. The intruder might attempt to exploit well-known buffer-overrun holes by sending large amounts of data. The intruder may start checking for login accounts with easily guessable (or empty) passwords.

Get hold of the system:

At this stage, the hacker has successfully gained a foothold in your network by hacking into a machine. The intruder’s main goal is to hide evidence of the attacks (doctoring the audit trail and log files) and make sure they can get back in again. They may install ‘toolkits’ that give them access, replace existing services with their own Trojan horses that have backdoor passwords, or create their own user accounts.

Gain advantage of the attack:

The intruder takes advantage of their status to steal confidential data, misuse system resources (i.e. stage attacks at other sites from your site), or deface Web pages.


Encryption is a method by which the data is encoded through some encryption algorithm into some other (unreadable) form at the source computer before transmitting and then it is decoded at the destination by applying the reverse decryption algorithm.

There are two types of encryption:

  1. Symmetric encryption, also known as conventional encryption.
  2. Asymmetric encryption, also known as Public Key encryption.

Symmetric Encryption

The symmetric encryption scheme consists of the following components:

Component Description
Plain text Constitutes our data that has to be sent securely over the network.
Encryption algorithm Can be any technique of hiding the original data and making it unreadable to others.
Secret Key Used because the encryption algorithm, if known to anyone, can be decrypted easily on the network but if a key just like a hidden password is used to lock the data, the data cannot be unlocked by the intruder even he/she gets to know the algorithm by any means and only the recipient can unlock it.
Cipher text (Encoded) The encrypted text that is encoded in some other form and unreadable to others.
Decryption algorithm The reverse algorithm that can be applied to decrypt or decode the encrypted data to make it readable again. This is done at the destination side.

Asymmetric Encryption

Asymmetric Encryption, or public key encryption, depends on a pair of keys called public key and private key; hence the name. The keys are selected such that, if data is encrypted through key 1, it can be only decrypted through key 2 and vice versa. Of the two keys, we tell about one to everybody and call it a public key. The other is kept private for decrypting and called a private key. For example, our e-mail account has a public e-mail address that we give to everyone we want to but we won’t tell the password to anyone.

Typical Example of the Use of Encryption

Suppose a person named Linda is a broker and she gets a request mail by James Anderson for buying some stock shares for his company. She performs all the arrangements and sends a confirmation mail to James. In the end, she sends a bill to him for the payment; at this point, James completely denies that he has ever sent a mail to Linda for any stock shares. Now what should Linda do? She is in extreme trouble because there is no clue to prove that James was the actual e-mailer.

The solution is provided by the use of public key encryption; if Linda has encrypted the data by a public key, it can be decrypted only through Linda’s private key which should be told only to James, so when James replies to the confirmation mail for the shares, it is known for sure that the answering person is no other then James Anderson and he is caught. This is source authentication.

If we use the hashing scheme, such as MD5, on our data and generate a hash value for it at the source computer and send it along the data to the target, the destination computer will also compute its hash code for the received data. If the hash generated by the destination is same as the one received by the source, our data integrity is preserved; in other words, the data has reached its destination without any change or loss. This hash code is called a digital signature when sent with e-mail data.

IPSEC Security Services

  • Data Integrity
  • Data origin authentication
  • Replay prevention
  • Confidentiality
  • Limited traffic flow confidentiality

Replay prevention means that if somebody gets to know the keys by some means and resends your messages again or if someone gets to know the user name and password of your account, he or she can directly learn all your important business transactions and deals with others and can enjoy full authority to make other deals with them on your account using your name.

Internet Key Exchange (IKE)

IKE is a mechanism in IPSec where we exchange the key. It is a hybrid protocol that implements Oakley and Skeme key exchanges inside the ISAKMP framework. While IKE can be used with other protocols, its initial implementation is with the IPSec protocol. IKE provides authentication of the IPSec peers, negotiates IPSec keys, and negotiates IPSec security associations. The main features of IKE are as follows:

  • Negotiates policy to protect communication
  • Authenticated Diffie-Hellman key exchange
  • Negotiates (possibly multiple) security associations (SA) for IPSec.

Diffie-Hellman is a public-key cryptography protocol that allows two parties to establish a shared secret over an unsecured communication channel. Diffie-Hellman is used within IKE to establish session keys. 768-bit and 1024-bit Diffie-Hellman groups are supported.

Security Association (SA) combines the agreed upon principles for VPN communication. This is done by IKE. The secret key exchange is the main process so that the dependent data to be delivered is secured.

Isakmp + oakley is the IKE policy that we define to start the encryption process. The Internet Security Association and Key Management Protocol (isakmp) is a protocol framework that defines payload formats, the mechanics of implementing a key exchange protocol, and the negotiation of a security association. Oakley is a key exchange protocol that defines how to derive authenticated keying material. Skeme is a key exchange protocol that defines how to derive authenticated keying material, with rapid key refreshment.

MD5 (Message Digest 5) is a hash algorithm used to authenticate packet data. HMAC is a variant that provides an additional level of hashing. The Data Encryption Standard (DES) is used to encrypt packet data. IKE implements the 56-bit DES-CBC with Explicit IV standard. Authentication header is used for data integrity and source authentication whereas encapsulating security protocol is used for confidentiality.

Initiating a New VPN Connection

IKE SA is established in “main mode” where our own generated key is provided. IPSec SA is established in “quick mode” where self generating keys—for example, session keysÿare generated. The session keys are for a certain time limit and are generated automatically by the IPSec for more security. There are multiple quick modes for one main mode.

Pre-share means that authentication keys are known before the data transfer through different ways such as personally telling anyone by telephone, e-mail, or face-to-face.

Modes of Configuring Encryption on VPN Networks

There are basically two modes for setting up an encryption policy for VPN:

  1. Tunnel mode
  2. Transport mode

Tunnel mode is the default mode. In tunnel mode, the encryption process is done by routers or gateways, whereas in transport mode, source computers do the encryption; in other words, the data source encrypts the data itself.

In tunnel mode, the IP header is also encrypted by the router and the router puts its own header with the source address as the IP address of its own interface and the destination address as the IP address of its peer router on the traffic path so this mode is more secured than transport mode.

Making Secure VPNs Through IDS and Firewalls

IDS (Intrusion Detection System) is a device or application used to inspect all network traffic and alert the user or administrator when there have been unauthorized attempts or access. The two primary methods of monitoring are signature-based and anomaly-based. Depending on the device or application used, the IDS can either simply alert the user or administrator or it could be set up to block specific traffic or automatically respond in some way.

Signature-based detection relies on comparison of traffic to a database containing signatures of known attack methods. Anomaly-based detection compares current network traffic to a known-good baseline to look for anything out of the ordinary. The IDS can be placed strategically on the network as a NIDS (network-based intrusion detection) that will inspect all network traffic, or it can be installed on each individual system as a HIDS (host-based intrusion detection) that inspects traffic to and from that specific device only.

A firewall is simply a device that shuts off everything, and then turns back on only a few well-chosen items. The reason we have firewalls is precisely because security holes are left open accidentally. We are all dependent on firewalls and other perimeter protection systems to protect our sites. It is virtually impossible to secure all the systems in a facility and keep them secured, so we turn to perimeter defenses.

SSH, SSL, and PPTP Technology in VPN

SSH (Secure Shell) is a program for logging into a remote machine and for executing commands on a remote machine. It provides secure encrypted communications between two untrusted hosts over an insecure network. X11 connections and arbitrary TCP/IP ports also can be forwarded over the secure channel. It is intended as a replacement for rlogin, rsh, and rcp, and can be used to provide rdist and rsync with a secure communication channel.

SSL (Secure Socket Layer): The traditional VPN requires a special client installed on each computer, or can use a built-in VPN client in many operating systems. A VPN server/firewall terminates the connection on the enterprise side and allows access across the VPN encrypted tunnel to internal resources. The SSL/TLS VPN uses a browser as the front end, and delivers applications inside the browser terminating the connection behind a firewall at an SSL/TLS server which relays application data to appropriate internal resources.

VPN includes a a technology called PPTP (PPP over TCP) built into operating systems. This gives a machine two IP addresses—one on the Internet, and a virtual one on the corporate network. IPSec enhances the traditional IP protocol with security. While VPN vendors claim their product “enhance security,” the reality is that they decrease corporate security. While the pipe itself is secure (authenticated, encrypted), either ends of the pipe are wide open.

The fundamentals of tunneling with SSH, SSL, IPSec and PPTP are out of the scope of this article. This article gives you an understanding on what is and how VPN encryption and authentication work. I hope it’s helpful for the readers.

Get the Free Newsletter!

Subscribe to Developer Insider for top news, trends & analysis

Latest Posts

Related Stories