Advance Reservation/Co-Scheduling Evaluation Criteria

1. General Information
   a. What is the name of the tool?
   b. Where is the web site for the tool?
   c. Cost/Licensing
      1. Is the tool free to use? Is support free? 
      2. How is the tool licensed? (GPL, Commercial,etc.) 
   e. Code Availability
      1. Is the code open-source? 
      2. Is there a mechanism for the developer to accept code changes
         from TeraGrid?
   f. Support/Documentation
       1. What support does the scheduler developer provide? (24/7,
         user forums, faqs)
       2. What is the quality of the documentation?
       3. Is the web site for the tool helpful and informative?
   g. Product Maturity
       1. How long has the product been available? 
       2. What is the production status of the code?
          (prototype/alpha/beta/production)?
       5. How many other production grids use this software today? 
       6. Approximately how many users of this software are there?
       7. Approximately how many developers support this product? 

2. Functionality. Does the tool support the following functionality at
   this time? (also indicate if the functionality is planned in the
   future and a timeline, if known)
   a. Advanced reservations of individual systems?
   b. Co-scheduling/co-allocation across more than one system?
   c. Request a reservation/co-allocation starting at a specific time?
   d. Request a reservation/co-allocation starting within a window of
      time?
   e. Request a reservation/co-allocation with an exact number of
      nodes/resources?
   f. Request a reservation/co-allocation with a range of number of
      nodes/resources?
   g. Request a reservation/co-allocation on an exact set of systems?
   h. Request a reservation/co-allocation on a subset of a candidate
      set of systems?
   i. Query the status of a previously made reservation/co-allocation?
   k. Query for when a specific reservation/co-allocation can be made?
   l. Provide useful information about why a reservation/co-allocation
      cannot be made so that a user can adjust their request? What are
      examples of reasons the tool provides?
   m. Submission of reservation requests directly to the local
      scheduler (i.e. without going through any global services)?
   n. Please list other relevant functionality not described above.

3. Installation
   a. Were the installation instructions clear?
   b. What Resource Managers does the tool interface with?  (e.g. LSF,
      PBS, LoadLeveler, SGE)
   c. Does the scheduler require any modifications to local resource
      manager? Are these modifications straightforward?
   d. How long did installation take in hours of work?
   e. What additional software is required in order to support the
      tool and where must it be installed? For each software
      dependency, is that software already in CTSS?
        1. each Teragrid resource (for example, GridFTP) 
        2. somewhere on the TeraGrid (for example, MDS, Myproxy) 
        3. on the same machine as the metascheduler (for example, OS, MySQL) 
   f. Did you ask any questions of the developers?
      If so, were the developers responsive?
   g. What customization was necessary to get the software to work?
      Was this customization easy or difficult?
   h. Are there installation problems that you expect would occur
      on many installations?
   i. For the software components that would be installed by RPs,
      are there any barriers to installing these components automatically
      as part of a CTSS kit?

4. Operation
   a. How reliable is the software (failures/week)?
   b. What failures were encountered?
   c. Does the software provide logging?
      Can the amount of logging be adjusted?
   d. What amount of resources are typically used by the software?
      On what systems? (e.g. central server, login node)
      (e.g. disk space, physical/virtual memory, CPU time)

5. User Experience
   a. What is the quality of the user documentation?
   b. What client interfaces are provided (GUI, command line, web
      interface, etc.)
   c. For each user interface evaluated (e.g. GUI, API, command line):
      Provide the interface name:
      Is it well documented?
      Is it easy to understand and use?
      Are there any changes to the interface that would improve it?
   d. Where any problems encountered? (e.g. documentation not matching
      interface, unimplemented features)
   e. Are the error messages clear and helpful for debugging problems?
   f. What is the average response time of the software?
   g. How does the software perform under load?
      At what amount of load does the software begin to respond slowly?
      (e.g. twice as slow as unloaded response time)
   h. For each TeraGrid user helping evaluate:
      Does this software meet your needs?
      Is this your preferred software for performing advance reservation
      and/or co-scheduling?

6. Any other evaluator comments?