TeraGrid Architecture

From TeraGrid Wiki

Jump to: navigation, search

TeraGrid is an instance of cyber-infrastructure supporting TeraScale science. As such it connects a wide variety of resources, services and users for a wider variety of uses. To understand the overall architecture a diagram and description of the various structural elements would be nice to have. We have developed several elements of this in the Wiki organically. This section is intended to be a home for creating a top level organizing document.

This document is focused on describing the top level of TeraGrid's architecture. It describes "what we've built" (artifacts) rather than "what we do" (activities). It would be very nice to have a companion document that describes the many activities that the TeraGrid staff conducts around the system described in this document.


Design Goals

This section describes why TeraGrid has the architecture it has, as described in the rest of this document. It does this by describing the high-level goals that the TeraGrid community is attempting to enable through construction, operation, and use of the TeraGrid facility.

TeraGrid's Vision: Deep, Wide, Open

The TeraGrid vision requires us to work toward objectives in three major mission areas: deep, wide, and open. TeraGrid's "deep" mission is to improve and expand the capabilities available to world-class HPC researchers. TeraGrid's "wide" mission is to broaden the user population which it serves. TeraGrid's "open" mission is to provide a reliable infrastructure with cleary defined interfaces and service definitions on which the US research community (and its collaborators) can build their own tools and infrastructure extensions.

Diverse Resources, Flexible Degrees Of Integration

TeraGrid is not a single, hierarchical organization or a hegemony. There is no monopoly service provider for HPC capabilities. At all levels of the TeraGrid facility there are multiple resource and service provider organizations, working within a common framework to provide elements of a coherent facility. This is not the case for ideological reasons: the reality is that in the United States (and the world) there are many candidate resource providers, and each has unique strengths and competencies that they can bring to the community.

Our strategy is to construct an enabling framework that enables and encourages many diverse resource and service providers to integrate their offerings to the community. The framework allows for community-wide standards and best practices to be developed and promoted, without prohibiting the introduction of unique or non-standard offerings. In practice, we have found that although diversity is allowed, there are many areas where consensus can be reached, and in those areas, resource providers find little or no incentives to differentiate the interfaces they offer from those of other resource providers. In areas where concensus has not yet been reached, the framework allows for experimentation and the development of new best practices.

A key feature of this open framwork is that integration into the framework is flexible and can be done to greater or lesser degrees depending on the unique nature of the resource or service being offerered. The "TeraGrid Resources" section describes the four degrees of integration that we believe are the "typical" cases within this framework.

Interoperability and Standards

TeraGrid is an important element of NSF's cyberinfrastructure, but it is not the only one. Furthermore, for most TeraGrid users, TeraGrid is one of several IT systems that they make use of. (Nearly all users make use of personal workstations or laptops and campus or departmental systems. Many TeraGrid users also use other HPC systems, often non-NSF, and sometimes systems in other countries.) It is important that TeraGrid's systems and service interfaces are designed and operated in such a way that they can interoperate as smoothly as possible with the other elements of our users' overall IT landscape. Although there is a huge diversity in external systems--which we have little or no control over--we believe that standards, both formal and informal, offer a good likelihood of making our user's work easier as they integrate TeraGrid systems into their overall workflow.

Consequently, we have placed a high premium on compliance with community standards wherever that is possible. This plays out most clearly in our service interfaces. (See the section titled "TeraGrid's Common Interface Definitions.") Our use of Grid community standards, for example, makes it significantly easier for TeraGrid's science gateways, virtual organizations, and sophisticated individual scientists to add the use of TeraGrid resources to their existing suite of systems.

One area where we have identified an opportunity for significant improvements is in regard to user identity certification and credential management. Efforts in the fourth and fifth year of the current TeraGrid funding cycle are aimed at making greater use of existing systems at our user's home institutions to reduce the degree to which TeraGrid "stands alone" in regard to certifying user identities and issuing credentials.

System-wide Documentation

These are resources like our website, documentation series, and community exchange fora (coming soon) which do not require a TeraGrid allocation or account and which directly support the end user and/or the general public in interacting with the TeraGrid or supporting their collaborations.


The TeraGrid website provides a "front door" for TeraGrid. It accomplishes the following goals:

  1. Route the general public to a structure of information about TeraGrid starting at very high level down to the level of user registration/SGW redirect/contact info for query.
  2. Route users to the User Portal for processes targeting established users.
  3. Route interested stakeholders to information about technical arch/planning/performance.

Stakeholders include: sponsors, press, grid developers, … TeraGrid's website is available at http://www.teragrid.org/.

User Portal

The user portal is described in more detail in the section "TeraGrid's Core Components," but it plays an important role here also. The User Portal provides both TeraGrid users and potential TeraGrid users with information and services about how TeraGrid's allocation process works and the mechanisms for producing and submitting allocation requests, as well as managing their accounting information once they receive an allocation.

Knowledge Base

TeraGrid's Knowledge Base system provides an online, searchable help system for TeraGrid. It provides a wealth of information about the TeraGrid system and how to use it, packaged in short, question/answer chunks of information. This allows users to enter human-language questions and receive answers that are easy to read and apply. The Knowledge Base is primarily used for end-user support, but there is also a mechanism for staff-only entries that can be used to support TeraGrid staff.

Trouble Ticket System

The TeraGrid community uses a common ticketing system for problem reports and issue tracking. In addition to its primary function as a tool for tracking the status of open issues, it also stores and provides access to the record of current and past issues. The system is currently operated by one of the TeraGrid resource provider organizations.

Community Documents

The TeraGrid community publishes and maintains a series of documents that have been formally reviewed and/or approved by TeraGrid's governance structure. These documents describe the TeraGrid system and its governance. This series is available at TeraGrid Documents.

TeraGrid Resources

TeraGrid's open architecture allows independent resource and service providers to offer their capabilities to the TeraGrid community. Resource and service providers may use TeraGrid Core Components (see the following section, "TeraGrid's System Architecture" for details) to add increasing degrees of integration with the community's mechanisms, but the threshold for the simplest forms of integration is intentionally very low.

In practice, TeraGrid resources and services currently consist of the nationally allocated resources that most people think of when they think of TeraGrid. Use of these resources is governed by a peer review process at the national level proportional to the scale of the allocation request. These resources use most or all of the Core mechanisms to establish a high degree of integration with the TeraGrid community.

In addition to the nationally allocated resources, we see a clear need for more lightweight classes of participation in the TeraGrid community, and we are exploring options in that direction.

Base Model: Nationally Allocated Resources

Nationally Allocated Resources provided by the TeraGrid are managed services where usage is metered and allocations given (either individually or collectively) via a peer review process tied to the scale of the allocation request.

TeraGrid's Nationally Allocated resources have historically fallen into three categories, distinguished by their intended uses.

  • Compute resources
  • Storage resources
  • Visualization resources

Compute Resources

Compute resources are selected to support scientific applications based on many different criteria: interconnect latency, memory architecture, number of processors, etc. Ultimately the choice of which system best supports a given application is one made by the user and has a number of non-compute aspects (history

We categorize systems into 3 groups based on their policies on scheduling priority: high fractional use of specific service, fair share (based on equal access or allocation weighted), or service level agreements (e.g. co-scheduling or latency targets). Furthermore we have a threshold for allocations process of X TF gross capacity {and/or >50% job expansion ratios due to queue depth the previous quarter (too fast a turnaround cycle for allocations process) – are there small systems which are over-requested ? requested >50% ?}

High Fractional use systems give higher priority to requests which efficiently use large fractions of the machine. The rationale here is to give priority to jobs which require large core counts to run (the presumption being that smaller jobs are more agile, at least in principle. That may not be the case for example with viz jobs, etc. but is that only the case for specialty machines ?) The measure of success here is XXX.

Fair share systems balance the job priority based on previous history of the requesting party. In democratic scheduling, there weights are the same for each individual within a queue such that in a fully populated system usage distributions look the same for all authorized individuals. With allocation weighted scheduling in a fully subscribed system, the fractional use of a resource reflects their allocation fraction of that machine (within the pool of ready users – there’s getting to be too many caveats here. Need to strive for simplicity if have to sacrifice thoroughness..)

Service level agreement systems are prioritized based on some agreement made with individual groups within the user community. This includes systems which offer reservations (short or long term), preemptable queues (ala urgent computing), or other (standard) agreements.

(Do we need to define a common billing expectation for reservations ? Or just that valid reservations bill at clock time regardless of utilization. Is there a clear differentiation in standardization rigor needed for human service definitions and automata service definitions ?)

(What sort of detail from an architectural sense is needed in order to organize/define resource types from an allocation/accounting purpose ?)

Q: If these capabilities don’t change frequently on a machine, why are we investing in the framework ? A: Because the listings allow for (sub)versioning info which does change, because we have a ~5% turnover per quarter due to students (we should look to see whether that fraction persists if one looks at the charging users only)

Storage Resources

Visualization Resources

(no special capability yet defined. Does a Viz resource differ in a common significant way from a compute resource ? is it just another instance of coprocessor/software ?)

Extensions to the Base Model

Although the Nationally Allocated Resources class is currently the de facto model and the only one that has significant participation at the moment, we see a clear need for more lightweight classes of participation. Our base architecture allows for other classes, and we believe the following would be attractive targets for broadened participation in the "resource provider" area.

  • TeraGrid Accounted Resources: These resources are provided by TeraGrid Resource Providers in support of Terascale research. They are made available to all users with roaming allocations on TeraGrid. Accounting records (for use of the resource) are stored in the TeraGrid central accounting database. The national allocation process would not assign users to specific resources in this class. The primary use cases for this class would be opportunistic compute resources (e.g., cycle scavenging applications) and data storage allocations.
  • TeraGrid Authorized Resources: These resources are made available to persons registered with TeraGrid. Frequently these resources may have some access restriction (e.g. license limited to members of the TeraGrid community) that needs to be enforced. Usage of these resources is freely available to registered members of the TeraGrid community. TeraGrid's Authentication/Authorization services (Kerberos, MyProxy - see description in System Architecture section) provide the authorization mechanisms for limiting access to these resources.
  • TeraGrid Affiliated Services: The most loosely coupled and most flexible category of TeraGrid resources is the "TeraGrid affiliated" category. These are services provided by independent service providers to the TeraGrid user community. The TeraGrid community does not set the access policy, operations, or quality of the services provided, but the service providers utilize at least one of the Core mechanisms listed in the following section to integrate their services with the TeraGrid system.

TeraGrid System Architecture

TeraGrid's resources (particularly the nationally allocated resources) are key to achieving TeraGrid's "deep" mission: providing advanced HPC resources for leading-edge science. They are not sufficient to attain TeraGrid's "wide" or "open" missions. Progress in these areas relies on two additional architectural elements: (1) a set of core system components that provide system-wide services, and (2) a set of common interface definitions that resources or services may implement in order to provide users with familiar and consistent interfaces on which to build applications and infrastructure extensions.

TeraGrid's Core Components

The TeraGrid community maintains a set of core components that are used to integrate TeraGrid's many resources such that system-wide actions may be performed. The core components support the NSF's HPC allocation process and allow TeraGrid users to work with TeraGrid at the system-wide level.

These core components are not exclusive. Integration with TeraGrid's core components does not intentionally prohibit or inhibit integration with other systems (e.g., Open Science Grid). In fact, several TeraGrid resources do particicpate in multiple federations.


TeraGrid contracts with a network service provider to use a high bandwidth (currently 10 Gbps) wide area network. This network provides direct connectivity between each of the current TeraGrid resources. It also provides traffic routing to/from significant research and commercial networks.

User Portal

 Lee's note: This is too much information.  Replace most of this with pointers to details in other document(s).

The target audience for the User Portal is the enquiring, current and alumni user community. The assumption which is made is that the people coming to this site are primarily motivated by wanting to USE the TeraGrid and getting access to the information and processes needed in order to achieve their goals.

From that perspective, historical information, general announcements and explanatory text, job announcements, etc. are distractions and should be included by reference, if at all. The top objectives of the User Portal are:

  • Discovery of what resources are available (now and on a planned timeline)
  • Obtaining and managing access to these resources
  • Optimizing use of the resources
  • Introduction to available services and default uses of tools
  • Matching with troubleshooting/consulting/training opportunities
Proposal Management

The primary purpose here is for qualified users to be able to submit well-formed proposals for review. It needs to accomplish the following goals:

  • Provide overview/introduction to the process
  • Help user to create a valid proposal to submit to the appropriate review process
  • Provide status information about proposals in process.

An early (and recurring) interaction with aspiring and current users is the proposal request process. This should start smoothly from the user portal (starting from an origination of a person unkown to TG) guiding the user through the process, building from their history from previous interactions with TeraGrid (ie. Use registered data, etc.)

There are the following steps a user should be guided through:

  • Learn about the proposal process, timelines and requirements
  • Develop a proposal (This may include information/expectations from a previous proposal)
  • Submit the proposal for review
  • Respond to review comments
  • Appeal a decision and/or ask for a supplement/extension
Award/Account Management

The primary purpose here is for the user to be able to tend their continued access to the resources. It needs to accomplish:

  • Help user create and/or register the proper identity credentials to his/her account
  • Show and manage the current set of accounts and awards
  • Show the status or registered accounts
  • Show the status of recent award transactions.
  • Link into the extension/supplement proposal process

From this section, a user should be able to monitor their award/account status. There should be a listing of all accounts with summarized access information. There should be a summary of the account status and a “recent transactions” ability to retrieve detailed info about “charges”. This is particularly important as we move to a concept of a common “currency” for obtaining small scale services. A transaction granularity of monthly summary by resource is probably the proper default, but the user will need the ability to drill down to the level of daily or job level charges by resource if there is need to investigate anomalies. Links to resources should be live and cross-connected to go to descriptions of resources, contact information, policy and rate descriptions, etc.

Authentication/Authorization Services

  • Kerberos Realm Servers
  • MyProxy Servers
  • TGCDB (specifically, the part of TGCDB that stores data about allocated users)

The credential service provides an online method of obtaining proxies for grid transactions. Its primary support model is to support interactive use for the user to obtain credentials prior to initiating any grid transaction. The most frequent method of doing this is through the User Portal embedded methods.

Q: Do we have a method for automata (Portals, workflows, etc.) to interact with MyProxy to get their credentials ? A: They are assumed to have gotten and manage their own offline credentials and generate proxies from there (probably)

Accounting Database

  • TGCDB (specifically, the part of TGCDB that stores and manages account records)

Integrated Information Service

  • MDS Indices
  • Standardized Schema

An information service is really valuable in one of two cases:

where there are many (equivalent) services to choose from and some method of indexing the service is required in order to find the relevant subset for a given need (ties into ontologies (tarpit) and/or search capabilities).

Where there is need for a framework for managing information changes about a resource in a way that is transparent to the application (ala DNS).

Are there other aspects ? Which of these are we promoting to the users for which kinds of applications ?

TeraGrid's Common Interface Definitions

The TeraGrid community has defined a set of standard interface definitions that may be implemented by resource providers in order to provide useful capabilities to TeraGrid users in a way that will be familiar to the users because they are consistent with how other resource providers provide comparable services.

No resource provider is required to implement any of these interfaces, and resource providers are free to provide similar capabilities without observing the standard interfaces. In practice, it is the case that current resource providers are using these definitions whenever they decide to offer a capability that looks like any of these. (In other words, it is unusual for a resource provider to offer a "similar but different" capability.)

The standardized interface definitions are listed below. Details about the capabilities offered by these interfaces and how they are implemented is available at CTSS Version 4.

  • Remote Login
  • Remote Computation
  • Data Movement
  • Data Management
  • Wide Area Parallel Filesystem
  • Application Development and Runtime Support
  • Parallel Application Development and Runtime Support
  • Distributed Parallel Application Development and Runtime Support
  • Science Workflow Support
  • Scientific Visualization Support

Common interface definitions are established within the community via an open process described at The CTSS Process. The final step in this process is ratification by the TeraGrid Resource Provider Forum (RPF) as described in TeraGrid's governance processes. Successful definitions are added to the collection of TeraGrid Documents. As of the end of 2007, none of the definitions above have completed this final step, though all of them have been implemented and are currently available on TeraGrid resources.

Personal tools