7 <title>Testing the JDK</title>
8 <style type="text/css">code{white-space: pre;}</style>
9 <link rel="stylesheet" href="../make/data/docs-resources/resources/jdk-default.css">
10 <!--[if lt IE 9]>
11 <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
12 <![endif]-->
13 <style type="text/css">pre, code, tt { color: #1d6ae5; }</style>
14 </head>
15 <body>
16 <header>
17 <h1 class="title">Testing the JDK</h1>
18 </header>
19 <nav id="TOC">
20 <ul>
21 <li><a href="#using-make-test-the-run-test-framework">Using "make test" (the run-test framework)</a><ul>
22 <li><a href="#configuration">Configuration</a></li>
23 </ul></li>
24 <li><a href="#test-selection">Test selection</a><ul>
25 <li><a href="#jtreg">JTReg</a></li>
26 <li><a href="#gtest">Gtest</a></li>
27 <li><a href="#special-tests">Special tests</a></li>
28 </ul></li>
29 <li><a href="#test-results-and-summary">Test results and summary</a></li>
30 <li><a href="#test-suite-control">Test suite control</a><ul>
31 <li><a href="#jtreg-keywords">JTReg keywords</a></li>
32 <li><a href="#gtest-keywords">Gtest keywords</a></li>
33 </ul></li>
34 </ul>
35 </nav>
36 <h2 id="using-make-test-the-run-test-framework">Using "make test" (the run-test framework)</h2>
37 <p>This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p>
38 <p>The main target <code>test</code> uses the jdk-image as the tested product. There is also an alternate target <code>exploded-test</code> that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
39 <p>Previously, <code>make test</code> was used invoke an old system for running test, and <code>make run-test</code> was used for the new test framework. For backward compatibility with scripts and muscle memory, <code>run-test</code> (and variants like <code>exploded-run-test</code> or <code>run-test-tier1</code>) are kept as aliases. The old system can still be accessed for some time using <code>cd test && make</code>.</p>
40 <p>Some example command-lines:</p>
41 <pre><code>$ make test-tier1
42 $ make test-jdk_lang JTREG="JOBS=8"
43 $ make test TEST=jdk_lang
44 $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
45 $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
46 $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
47 $ make exploded-test TEST=tier2</code></pre>
48 <h3 id="configuration">Configuration</h3>
49 <p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=<path to jtreg home></code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p>
50 <h2 id="test-selection">Test selection</h2>
51 <p>All functionality is available using the <code>test</code> make target. In this use case, the test or tests to be executed is controlled using the <code>TEST</code> variable. To speed up subsequent test runs with no source code changes, <code>test-only</code> can be used instead, which do not depend on the source and test image build.</p>
52 <p>For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that <code>make test-tier1</code> is equivalent to <code>make test TEST="tier1"</code>, but the latter is more tab-completion friendly. For more complex test runs, the <code>test TEST="x"</code> solution needs to be used.</p>
53 <p>The test specifications given in <code>TEST</code> is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an example, <code>:tier1</code> will expand to <code>jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1</code>. You can always submit a list of fully qualified test descriptors in the <code>TEST</code> variable if you want to shortcut the parser.</p>
54 <h3 id="jtreg">JTReg</h3>
55 <p>JTReg tests can be selected either by picking a JTReg test group, or a selection of files or directories containing JTReg tests.</p>
56 <p>JTReg test groups can be specified either without a test root, e.g. <code>:tier1</code> (or <code>tier1</code>, the initial colon is optional), or with, e.g. <code>hotspot:tier1</code>, <code>test/jdk:jdk_util</code> or <code>$(TOPDIR)/test/hotspot/jtreg:hotspot_all</code>. The test root can be specified either as an absolute path, or a path relative to the JDK top directory, or the <code>test</code> directory. For simplicity, the hotspot JTReg test root, which really is <code>hotspot/jtreg</code> can be abbreviated as just <code>hotspot</code>.</p>
57 <p>When specified without a test root, all matching groups from all test roots will be added. Otherwise, only the group from the specified test root will be added.</p>
58 <p>Individual JTReg tests or directories containing JTReg tests can also be specified, like <code>test/hotspot/jtreg/native_sanity/JniVersion.java</code> or <code>hotspot/jtreg/native_sanity</code>. Just like for test root selection, you can either specify an absolute path (which can even point to JTReg tests outside the source tree), or a path relative to either the JDK top directory or the <code>test</code> directory. <code>hotspot</code> can be used as an alias for <code>hotspot/jtreg</code> here as well.</p>
59 <p>As long as the test groups or test paths can be uniquely resolved, you do not need to enter the <code>jtreg:</code> prefix. If this is not possible, or if you want to use a fully qualified test descriptor, add <code>jtreg:</code>, e.g. <code>jtreg:test/hotspot/jtreg/native_sanity</code>.</p>
60 <h3 id="gtest">Gtest</h3>
61 <p>Since the Hotspot Gtest suite is so quick, the default is to run all tests. This is specified by just <code>gtest</code>, or as a fully qualified test descriptor <code>gtest:all</code>.</p>
62 <p>If you want, you can single out an individual test or a group of tests, for instance <code>gtest:LogDecorations</code> or <code>gtest:LogDecorations.level_test_vm</code>. This can be particularly useful if you want to run a shaky test repeatedly.</p>
63 <p>For Gtest, there is a separate test suite for each JVM variant. The JVM variant is defined by adding <code>/<variant></code> to the test descriptor, e.g. <code>gtest:Log/client</code>. If you specify no variant, gtest will run once for each JVM variant present (e.g. server, client). So if you only have the server JVM present, then <code>gtest:all</code> will be equivalent to <code>gtest:all/server</code>.</p>
64 <h3 id="special-tests">Special tests</h3>
65 <p>A handful of odd tests that are not covered by any other testing framework are accessible using the <code>special:</code> test descriptor. Currently, this includes <code>failure-handler</code> and <code>make</code>.</p>
66 <ul>
67 <li><p>Failure handler testing is run using <code>special:failure-handler</code> or just <code>failure-handler</code> as test descriptor.</p></li>
68 <li><p>Tests for the build system, including both makefiles and related functionality, is run using <code>special:make</code> or just <code>make</code> as test descriptor. This is equivalent to <code>special:make:all</code>.</p>
69 <p>A specific make test can be run by supplying it as argument, e.g. <code>special:make:idea</code>. As a special syntax, this can also be expressed as <code>make-idea</code>, which allows for command lines as <code>make test-make-idea</code>.</p></li>
70 </ul>
71 <h2 id="test-results-and-summary">Test results and summary</h2>
72 <p>At the end of the test run, a summary of all tests run will be presented. This will have a consistent look, regardless of what test suites were used. This is a sample summary:</p>
73 <pre><code>==============================
74 Test summary
75 ==============================
76 TEST TOTAL PASS FAIL ERROR
77 >> jtreg:jdk/test:tier1 1867 1865 2 0 <<
78 jtreg:langtools/test:tier1 4711 4711 0 0
79 jtreg:nashorn/test:tier1 133 133 0 0
80 ==============================
81 TEST FAILURE</code></pre>
82 <p>Tests where the number of TOTAL tests does not equal the number of PASSed tests will be considered a test failure. These are marked with the <code>>> ... <<</code> marker for easy identification.</p>
83 <p>The classification of non-passed tests differs a bit between test suites. In the summary, ERROR is used as a catch-all for tests that neither passed nor are classified as failed by the framework. This might indicate test framework error, timeout or other problems.</p>
110 <p>What test data to retain (<code>-retain</code>).</p>
111 <p>Defaults to <code>fail,error</code>.</p>
112 <h4 id="max_mem">MAX_MEM</h4>
113 <p>Limit memory consumption (<code>-Xmx</code> and <code>-vmoption:-Xmx</code>, or none).</p>
114 <p>Limit memory consumption for JTReg test framework and VM under test. Set to 0 to disable the limits.</p>
115 <p>Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).</p>
116 <h4 id="options">OPTIONS</h4>
117 <p>Additional options to the JTReg test framework.</p>
118 <p>Use <code>JTREG="OPTIONS=--help all"</code> to see all available JTReg options.</p>
119 <h4 id="java_options">JAVA_OPTIONS</h4>
120 <p>Additional Java options to JTReg (<code>-javaoption</code>).</p>
121 <h4 id="vm_options">VM_OPTIONS</h4>
122 <p>Additional VM options to JTReg (<code>-vmoption</code>).</p>
123 <h3 id="gtest-keywords">Gtest keywords</h3>
124 <h4 id="repeat">REPEAT</h4>
125 <p>The number of times to repeat the tests (<code>--gtest_repeat</code>).</p>
126 <p>Default is 1. Set to -1 to repeat indefinitely. This can be especially useful combined with <code>OPTIONS=--gtest_break_on_failure</code> to reproduce an intermittent problem.</p>
127 <h4 id="options-1">OPTIONS</h4>
128 <p>Additional options to the Gtest test framework.</p>
129 <p>Use <code>GTEST="OPTIONS=--help"</code> to see all available Gtest options.</p>
130 </body>
131 </html>
|
7 <title>Testing the JDK</title>
8 <style type="text/css">code{white-space: pre;}</style>
9 <link rel="stylesheet" href="../make/data/docs-resources/resources/jdk-default.css">
10 <!--[if lt IE 9]>
11 <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
12 <![endif]-->
13 <style type="text/css">pre, code, tt { color: #1d6ae5; }</style>
14 </head>
15 <body>
16 <header>
17 <h1 class="title">Testing the JDK</h1>
18 </header>
19 <nav id="TOC">
20 <ul>
21 <li><a href="#using-make-test-the-run-test-framework">Using "make test" (the run-test framework)</a><ul>
22 <li><a href="#configuration">Configuration</a></li>
23 </ul></li>
24 <li><a href="#test-selection">Test selection</a><ul>
25 <li><a href="#jtreg">JTReg</a></li>
26 <li><a href="#gtest">Gtest</a></li>
27 <li><a href="#microbenchmarks">Microbenchmarks</a></li>
28 <li><a href="#special-tests">Special tests</a></li>
29 </ul></li>
30 <li><a href="#test-results-and-summary">Test results and summary</a></li>
31 <li><a href="#test-suite-control">Test suite control</a><ul>
32 <li><a href="#jtreg-keywords">JTReg keywords</a></li>
33 <li><a href="#gtest-keywords">Gtest keywords</a></li>
34 <li><a href="#microbenchmark-keywords">Microbenchmark keywords</a></li>
35 </ul></li>
36 </ul>
37 </nav>
38 <h2 id="using-make-test-the-run-test-framework">Using "make test" (the run-test framework)</h2>
39 <p>This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p>
40 <p>The main target <code>test</code> uses the jdk-image as the tested product. There is also an alternate target <code>exploded-test</code> that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
41 <p>Previously, <code>make test</code> was used invoke an old system for running test, and <code>make run-test</code> was used for the new test framework. For backward compatibility with scripts and muscle memory, <code>run-test</code> (and variants like <code>exploded-run-test</code> or <code>run-test-tier1</code>) are kept as aliases. The old system can still be accessed for some time using <code>cd test && make</code>.</p>
42 <p>Some example command-lines:</p>
43 <pre><code>$ make test-tier1
44 $ make test-jdk_lang JTREG="JOBS=8"
45 $ make test TEST=jdk_lang
46 $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
47 $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
48 $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
49 $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
50 $ make exploded-test TEST=tier2</code></pre>
51 <h3 id="configuration">Configuration</h3>
52 <p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=<path to jtreg home></code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p>
53 <p>To be able to run microbenchmarks, <code>configure</code> needs to know where to find the JMH dependency. Use <code>--with-jmh=<path to JMH jars></code> to point to a directory containing the core JMH and transitive dependencies. The recommended dependencies can be retrieved by running <code>sh make/devkit/createJMHBundle.sh</code>, after which <code>--with-jmh=build/jmh/jars</code> should work.</p>
54 <h2 id="test-selection">Test selection</h2>
55 <p>All functionality is available using the <code>test</code> make target. In this use case, the test or tests to be executed is controlled using the <code>TEST</code> variable. To speed up subsequent test runs with no source code changes, <code>test-only</code> can be used instead, which do not depend on the source and test image build.</p>
56 <p>For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that <code>make test-tier1</code> is equivalent to <code>make test TEST="tier1"</code>, but the latter is more tab-completion friendly. For more complex test runs, the <code>test TEST="x"</code> solution needs to be used.</p>
57 <p>The test specifications given in <code>TEST</code> is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an example, <code>:tier1</code> will expand to <code>jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1</code>. You can always submit a list of fully qualified test descriptors in the <code>TEST</code> variable if you want to shortcut the parser.</p>
58 <h3 id="jtreg">JTReg</h3>
59 <p>JTReg tests can be selected either by picking a JTReg test group, or a selection of files or directories containing JTReg tests.</p>
60 <p>JTReg test groups can be specified either without a test root, e.g. <code>:tier1</code> (or <code>tier1</code>, the initial colon is optional), or with, e.g. <code>hotspot:tier1</code>, <code>test/jdk:jdk_util</code> or <code>$(TOPDIR)/test/hotspot/jtreg:hotspot_all</code>. The test root can be specified either as an absolute path, or a path relative to the JDK top directory, or the <code>test</code> directory. For simplicity, the hotspot JTReg test root, which really is <code>hotspot/jtreg</code> can be abbreviated as just <code>hotspot</code>.</p>
61 <p>When specified without a test root, all matching groups from all test roots will be added. Otherwise, only the group from the specified test root will be added.</p>
62 <p>Individual JTReg tests or directories containing JTReg tests can also be specified, like <code>test/hotspot/jtreg/native_sanity/JniVersion.java</code> or <code>hotspot/jtreg/native_sanity</code>. Just like for test root selection, you can either specify an absolute path (which can even point to JTReg tests outside the source tree), or a path relative to either the JDK top directory or the <code>test</code> directory. <code>hotspot</code> can be used as an alias for <code>hotspot/jtreg</code> here as well.</p>
63 <p>As long as the test groups or test paths can be uniquely resolved, you do not need to enter the <code>jtreg:</code> prefix. If this is not possible, or if you want to use a fully qualified test descriptor, add <code>jtreg:</code>, e.g. <code>jtreg:test/hotspot/jtreg/native_sanity</code>.</p>
64 <h3 id="gtest">Gtest</h3>
65 <p>Since the Hotspot Gtest suite is so quick, the default is to run all tests. This is specified by just <code>gtest</code>, or as a fully qualified test descriptor <code>gtest:all</code>.</p>
66 <p>If you want, you can single out an individual test or a group of tests, for instance <code>gtest:LogDecorations</code> or <code>gtest:LogDecorations.level_test_vm</code>. This can be particularly useful if you want to run a shaky test repeatedly.</p>
67 <p>For Gtest, there is a separate test suite for each JVM variant. The JVM variant is defined by adding <code>/<variant></code> to the test descriptor, e.g. <code>gtest:Log/client</code>. If you specify no variant, gtest will run once for each JVM variant present (e.g. server, client). So if you only have the server JVM present, then <code>gtest:all</code> will be equivalent to <code>gtest:all/server</code>.</p>
68 <h3 id="microbenchmarks">Microbenchmarks</h3>
69 <p>Which microbenchmarks to run is selected using a regular expression following the <code>micro:</code> test descriptor, e.g., <code>micro:java.lang.reflect</code>. This delegates the test selection to JMH, meaning package name, class name and even benchmark method names can be used to select tests.</p>
70 <p>Using special characters like <code>|</code> in the regular expression is possible, but needs to be escaped multiple times: <code>micro:ArrayCopy\\\\\|reflect</code>.</p>
71 <h3 id="special-tests">Special tests</h3>
72 <p>A handful of odd tests that are not covered by any other testing framework are accessible using the <code>special:</code> test descriptor. Currently, this includes <code>failure-handler</code> and <code>make</code>.</p>
73 <ul>
74 <li><p>Failure handler testing is run using <code>special:failure-handler</code> or just <code>failure-handler</code> as test descriptor.</p></li>
75 <li><p>Tests for the build system, including both makefiles and related functionality, is run using <code>special:make</code> or just <code>make</code> as test descriptor. This is equivalent to <code>special:make:all</code>.</p>
76 <p>A specific make test can be run by supplying it as argument, e.g. <code>special:make:idea</code>. As a special syntax, this can also be expressed as <code>make-idea</code>, which allows for command lines as <code>make test-make-idea</code>.</p></li>
77 </ul>
78 <h2 id="test-results-and-summary">Test results and summary</h2>
79 <p>At the end of the test run, a summary of all tests run will be presented. This will have a consistent look, regardless of what test suites were used. This is a sample summary:</p>
80 <pre><code>==============================
81 Test summary
82 ==============================
83 TEST TOTAL PASS FAIL ERROR
84 >> jtreg:jdk/test:tier1 1867 1865 2 0 <<
85 jtreg:langtools/test:tier1 4711 4711 0 0
86 jtreg:nashorn/test:tier1 133 133 0 0
87 ==============================
88 TEST FAILURE</code></pre>
89 <p>Tests where the number of TOTAL tests does not equal the number of PASSed tests will be considered a test failure. These are marked with the <code>>> ... <<</code> marker for easy identification.</p>
90 <p>The classification of non-passed tests differs a bit between test suites. In the summary, ERROR is used as a catch-all for tests that neither passed nor are classified as failed by the framework. This might indicate test framework error, timeout or other problems.</p>
117 <p>What test data to retain (<code>-retain</code>).</p>
118 <p>Defaults to <code>fail,error</code>.</p>
119 <h4 id="max_mem">MAX_MEM</h4>
120 <p>Limit memory consumption (<code>-Xmx</code> and <code>-vmoption:-Xmx</code>, or none).</p>
121 <p>Limit memory consumption for JTReg test framework and VM under test. Set to 0 to disable the limits.</p>
122 <p>Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).</p>
123 <h4 id="options">OPTIONS</h4>
124 <p>Additional options to the JTReg test framework.</p>
125 <p>Use <code>JTREG="OPTIONS=--help all"</code> to see all available JTReg options.</p>
126 <h4 id="java_options">JAVA_OPTIONS</h4>
127 <p>Additional Java options to JTReg (<code>-javaoption</code>).</p>
128 <h4 id="vm_options">VM_OPTIONS</h4>
129 <p>Additional VM options to JTReg (<code>-vmoption</code>).</p>
130 <h3 id="gtest-keywords">Gtest keywords</h3>
131 <h4 id="repeat">REPEAT</h4>
132 <p>The number of times to repeat the tests (<code>--gtest_repeat</code>).</p>
133 <p>Default is 1. Set to -1 to repeat indefinitely. This can be especially useful combined with <code>OPTIONS=--gtest_break_on_failure</code> to reproduce an intermittent problem.</p>
134 <h4 id="options-1">OPTIONS</h4>
135 <p>Additional options to the Gtest test framework.</p>
136 <p>Use <code>GTEST="OPTIONS=--help"</code> to see all available Gtest options.</p>
137 <h3 id="microbenchmark-keywords">Microbenchmark keywords</h3>
138 <h4 id="fork">FORK</h4>
139 <p>Override the number of benchmark forks to spawn. Same as specifying <code>-f <num></code>.</p>
140 <h4 id="iter">ITER</h4>
141 <p>Number of measurement iterations per fork. Same as specifying <code>-i <num></code>.</p>
142 <h4 id="time">TIME</h4>
143 <p>Amount of time to spend in each measurement iteration, in seconds. Same as specifying <code>-r <num></code></p>
144 <h4 id="warmup_iter">WARMUP_ITER</h4>
145 <p>Number of warmup iterations to run before the measurement phase in each fork. Same as specifying <code>-wi <num></code>.</p>
146 <h4 id="warmup_time">WARMUP_TIME</h4>
147 <p>Amount of time to spend in each warmup iteration. Same as specifying <code>-w <num></code>.</p>
148 <h4 id="results_format">RESULTS_FORMAT</h4>
149 <p>Specify to have the test run save a log of the values. Accepts the same values as <code>-rff</code>, i.e., <code>text</code>, <code>csv</code>, <code>scsv</code>, <code>json</code>, or <code>latex</code>.</p>
150 <h4 id="vm_options-1">VM_OPTIONS</h4>
151 <p>Additional VM arguments to provide to forked off VMs. Same as <code>-jvmArgs <args></code></p>
152 <h4 id="options-2">OPTIONS</h4>
153 <p>Additional arguments to send to JMH.</p>
154 </body>
155 </html>
|