By Ram Lakshmanan
Recently, we ported a B2B Webservices application from a traditional hosting provider to an AWS cloud. The application’s traffic was originally load balanced using Apache Load Balancer. In the AWS cloud environment, we replaced Apache Load Balancer with AWS Elastic Load Balancer. During this migration, we ran into an interesting hiccup. In this article, we share the problem and solution because it may benefit others, too.
Problem Statement
The application was a B2B (Webservices) application. Its services were consumed by multiple clients, who are geographically distributed throughout the world. The application maintained the white-list of IP addresses of its clients. Only the clients on the white-list can invoke the services. Team had developed a Java ServletFilter in the application layer and, using this filter, it was authenticating the incoming requests’ IP address. (Even though it’s questionable whether it’s a secure solution, it is outside the focus of this article.)
The requests, which were serviced properly under Apache Load Balancer, started to fail when we moved to AWS Elastic Load Balancer. In fact, they were failing because of IP address authentication. How can the requests fail IP address authentication, when IP filtering was happening in the application layer code? The application layer code remained unchanged in both environments. Isn’t it puzzling?
Root Cause
Figure 1: Apache Load Balancer versus AWS Elastic Load Balancer
| Note: Before reading further, please review Figure 1 for a few seconds. It will help you understand the following content. |
Apache Load Balancer Preserved the Originator’s IP Address
Assume that a client from IP address 1.1.1.1 fires an HTTP request to the application. The request first hits the Apache Load Balancer, then Load Balancer sends the request to the Application Server. When the request finally hits the application server, the client’s IP address was preserved in the HTTP request. In other words, the server would see the originator’s IP address as 1.1.1.1; therefore, authentication worked correctly without any issues under the Apache Load Balancer.
AWS Elastic Load Balancer not Preserving Originator’s IP Address
Under AWS Elastic Load Balancer, when the client’s HTTP request hits the application server, the client’s IP address was NOT preserved. In other words, suppose the client sent the request from the IP: 1.1.1.1. When the request hits the server, the server sees the AWS Elastic Load Balancer’s IP address only (as in 20.20.20.20) and not the client’s IP address (1.1.1.1). Because AWS Elastic Load Balancer’s IP address isn’t white-listed, the application started to reject the requests.
Because of this discrepancy in the load balancer behaviour, the application started to reject all incoming requests.
Solution
There are couple of solutions to address this problem:
1. Web ACL
AWS provides a simple but yet powerful Web Access Control List (Web ACL) service. Using this service, one can do the IP filtering in the AWS Elastic Load Balancer. Thus, the team moved the IP Filtering logic from the application layer to the Load Balancer using the ‘Web ACL’ service.
The application layer also maintained a white-list of Load Balancer’s IP addresses only. Because of this, no one besides Load Balancers can shoot the request directly to the application server.
2. X-Forwarded-For
AWS Elastic balancer can be configured to pass the client’s IP address in the X-Forward-For header element. The X-Forwarded-For request header takes the following form:
X-Forwarded-For: clientIPAddress
Example: X-Forwarded-For: 203.0.113.7
The team went with solution #1, Web ACL, because it’s better to knock down the rogue request at the Load Balancer level itself instead of letting it come all the way to the Application server.
About the Author
Every single day, millions and millions of people in North America—bank, travel, and commerce—use the applications that Ram Lakshmanan has architected. Ram is an acclaimed speaker in major conferences on scalability, availability, and performance topics. Recently, he has founded a startup that specializes in troubleshooting performance problems.
*** This article was contributed. Views expressed are those of the author’s ©Developer.com, all rights reserved ***
