August 20, 2014
Hot Topics:
RSS RSS feed Download our iPhone app

Resource Management: An Introduction

  • September 18, 2007
  • By Art Sedighi
  • Send Email »
  • More Articles »

Resources' Information Database

This is not really a database in most cases because it is sustainable to a high degree of change. As resources come and go, this database must be updated. The info feeds directly into the scheduling engine used to schedule tasks to the available resources.

A resource needs to make itself available for it to be scheduled. When the resource logs on to the Grid, it sends its information to the resource manager. The resource manager uses this information to figure out what kinds of tasks can run on this specific resource. For example, if you are talking about a Windows-based server, perhaps you can run Java and Windows-complied applications. For Linux, the situation is a bit different. The same applies for the number of processors available, the amount of memory, the version of the OS, and many other system parameters that are used by the resource manager and later by the scheduler to determine the feasibility of that resource to the needed resource.

The difficulty arises when you try to find the best fit possible and the number of system parameters you use to determine how fit a resource really is. The number of parameters is directly proportional to the scheduling granularity and complexity. If the scheduler must take a number of parameters into consideration before making a final decision, this will add more overhead to the scheduler and increase system complexity. One of the ways this is mitigated is by the use of advanced hashtables and lookup tables that more information is found faster and more efficient updates are possible.

Clients' Information Database

This is a bit simpler because the number of clients is usually much smaller than the number of resources in a given Grid infrastructure. You may have a couple thousand resources, but only a handful of clients accessing and using these resources. From the perspective of the client manageability, the resource manager has a simpler task. From the point of view of high availability, however, it is not the case. Resource managers are or at least should follow the "fire and forget" methodology. This means that the clients send a task to the resource manager and from that moment on, the resource manager is in charge of making sure that the task gets done. This poses a number of challenges for the resource manager. The way this problem is solved varies from vendor to vendor, and as always the solution poses a performance vs. reliability tradeoff.

Scheduling Process

This is where the rubber meets the road. You have client demands; you have resources; what you need to do now is to match your supplies with your demands. Sounds easy enough, don't you think? It is, for the most part, easy. As mentioned before, the number of criteria that you use to choose the best decision directly impacts the complexity of your system. The more criteria, the more complex the scheduler will be. Generally speaking, you need to worry about the type of the target Operating System, CPU type, amount of memory, and available resources such as access to a file system or database. At the end of the day, schedulers are just one big finite state machine (FSM) with the input being the task and the final stage being the target resource that gets assigned that task.

A number of vendors give you the capability to optimize the scheduler. Things such as protocol used for communication between the components, amount of memory to consume whilst scheduling, and the ability to provide user-based filtering mechanisms are just some examples. Keep in mind that scheduling is and will be a difficult problem (NP-complete for those interested) and it can only be solved heuristically. This basically means that schedulers make some basic assumptions and then make a decision as to what request goes to what resource. When you choose a resource manager, you must make sure that the vendor allows you to change these assumptions used to solve this optimization problem that is scheduling.

Summary

I have started to talk about the components that make up a Grid. The focus was mainly on the resource manager at this point and, as you probably noticed, it is not an easy problem to tackle. Each implementation of a resource manager varies. Each vendor likes to add its own flavor to the mix and tackle the problem a little differently the next. You need to keep in mind that a solution that worked a few years back may not apply today. There are a number of advancements in the field of Grid and High Performance Computing and you are only beginning to scratch the surface. Keep your eyes open for new advancements and changes occurring in this field almost every day.

Endnote

1 I use the terms scheduler and resource manager interchangeably throughout this article. Although a scheduling (and the scheduler subsystem) is only one part of the resource manager, it is the most important part.





Page 2 of 2



Comment and Contribute

 


(Maximum characters: 1200). You have characters left.

 

 


Sitemap | Contact Us

Rocket Fuel