Introduction:

Benchmarking is a method for comparing the performance of different computing platforms. In this project, you will design a benchmark to compare the performance of some aspect of the Java Virtual Machines (JVMs) embedded in a variety of web browsers. Which aspect you choose to benchmark is up to you. Some options for what to benchmark include, integer operations, string operations, floating point operations, method calling, object creation, or graphics operations. This list is not exhaustive and you are free to select some other aspect of the JVMs to benchmark. You will design your benchmark by combining the results from existing benchmarks. See the Assignment section below for the specific requirements for this project.

I am open to allowing variants of this project which benchmark some aspect of real hardware platforms. Though, if you want to go this route you need to discuss your plans with me before proceeding.

Assignment:

The first thing you will want to do is to decide on which aspect of the JVM you wish to benchmark. Looking at the Resources below may give you some ideas in addition to the list provided above. I suggest that you play around with these benchmarks for a little while to get a feel for what they are doing and how the results are reported.

Once you have decided on what aspect you wish to benchmark, you will need to select at least three benchmarks that measure that aspect of performance. You will ultimately combine the results from these benchmarks into your own benchmark. Note that you may use benchmarks that measure just your chosen aspect. Or you may use individual results from within a benchmark that measures many different aspects of performance. For example, the CaffeineMark benchmark measures a wide variety of aspects of JVM performance. If you choose to measure graphics performance then you would just want to use the individual results from within the CaffeineMark that deal with graphics, ignoring the others.

After you have identified the aspect of performance that your benchmark will measure and the existing benchmarks that you will be using, you can collect your data. You need to benchmark the JVMs in at least two web browsers. In addition, you also need to benchmark the performance of the Applet Viewer included with the Java JDK. Keep in mind that you will want to run each benchmark multiple times on each JVM to obtain an average score. When reporting your results you must report the average scores for each of the benchmarks that you ran. You also need to report the scores normalized using the Applet Viewer as the base machine. Finally, you also need to combine the individual benchmark scores into a single score for your benchmark.

Once you have collected and analyzed the data from your benchmark, you will write a scientific paper describing your benchmark and presenting the results. See the section on The Paper below for more information about the content and format of the paper.

Using Applet Viewer:

The Applet Viewer is installed on the machines in the CS labs. To use the Applet Viewer to load and execute an applet you will need to enter a command with the following format on the command line:

AppletViewer <URL of Web Page Containing Applet>

For example, to load and execute the CaffeineMark benchmark applet use the following command line:

AppletViewer http://www.benchmarkhq.ru/cm30/runtest.html

Running this command will bring the CaffeineMark applet up in its own window and execute it using the JVM within Java's JDK.

The Paper:

You will present your benchmark and the results that you obtained in the form of scientific paper. As a guideline for the length of the paper, I expect that a good job could be done in 7 to 10 pages. In addition to a title your paper should have the following sections:

• Abstract
  Your abstract should provide a complete, but very concise, summary of the entire paper. It should state what you did, how you did it and the significant results that you found. Your abstract for this paper should probably be no more than 5 or 6 sentences.

• Introduction
  Your introduction should clearly state what you set out to do and provide some motivation for why you are doing it. In the case of this paper, you will want to briefly discuss the purpose of benchmarking and describe at a high level how you are applying it. The introduction should contain only enough details to give the reader an idea of what you are doing. So you probably will not need to discuss the particular benchmarks you are using or the specific JVMs that you are applying them to.

• Background
  In this section you will want to identify the specific benchmarks that you are using and clearly explain what each one is doing. In order to do this you will need to research the benchmarks you have chosen and provide citations for the information that you use. You will probably find web sites describing the benchmarks very useful. In some cases, the benchmarks may not be very well documented and you may have to examine the source code to figure out what exactly is going on. You may also find that there are published papers about some of the benchmarks. When possible, you should cite these published papers as opposed to web sites that contain the same information. This is because the published paper provides a more permanent reference that does a web site, which may vanish at any time. Finally, this section will describe the mathematics that you use to compute your results. This section will quite likely be the longest part of the paper. I would suggest that using a sub-section for each benchmark and another for discussing how the results were calculated might make good sense.

• Methods and Results
  Here you need to discuss how you collected your data and present the results that were obtained using the calculations described in the previous section. As with any scientific paper, your primary concern in presenting your methods and results is repeatability. You must include enough detail so that someone could reasonably expect to repeat your experiments and obtain similar results.

• Conclusions
  In your conclusions you should very briefly recap what you have done. You should explicitly state and discuss the answer to the question you set out to investigate as well as any implications of the answer that you found. You also need to be sure to identify any limitations of your results and suggest techniques for addressing those limitations.

• References
  Include a reference to each citation that you made in the paper. See the Honors Thesis Style Guide for the required format for your citations and references.

Finally, your paper must be formatted according to the guidelines provided in class.

Resources:

The following are links to some of the Java Benchmarking applets that are available on-line. This list is by no means exhaustive and you are encouraged to discover and use any other benchmarks that are applicable to your project.