Very interesting mail list post by Dalibor Topic in why he didn't seek the (Test Compliance Kit) TCK for Kaffe/Classpath

Contractual disputes between Apache and Sun notwithstanding, if you do get access to a JCP TCK, and the TCK is only available under NDA, then only a restricted subset of developers can see the tests. Those who have an access, have to keep it a secret. Although they can see the tests, they can't file bugreps that disclose public details of the test. That is, they could enter a bugrep that says "test 1032" failing, but not "test 1032 fails because https.proxyHost isnt set before java.net.URL ctor is called", or worse include a code snippet.

Secret tests go against the ethos of an OSS project, but they also go against the process, one in which you want every code submission to be regression tested before you commit the code. If you don't have the tests, you can only commit the code and hope it works.

The other sublety is that the TCK becomes the real specification, whether it is formal or not. If a test fails you either have a bug in your code, or a bug in the test. Unless you have the time to validate the test and the ability to effect changes, you are going to assume the TCK is correct, your code is wrong, and fix your code so the test passes. A TCK is a very formal specification: one which can be executed to see if an implementation follows it. By keeping the TCK secret, the true specification is effectively secret.

Open TCKs would (a) allow for broader code review of the TCK itself (and so improve its quality), and (b) make it easier for OSS projects to use the test kits.

updated fixed link