Screwing around with GJ/VGJ.

C. Scott Ananian

I've been playing with the various prototype compilers that Sun has been releasing as part of its "Extending the Java Programming Language with Generic Types" project (JSR-14). I've been programming with JSR-14 for about two years now, since the early 1.0 prototypes. In May Sun released version 2.0 of the prototype compiler, with support for variance and JSR-201 (enumerations, autoboxing, for loop enhancements, and import of static members). This page has been updated accordingly.

This page is a place to collect my various work on GJ-related things, including a more-readable syntax for variance and foreach, SinjP, a javap tool that works on v2.0 JSR-14 class files, SinjDoc, a javadoc tool that works on JSR-14 source files, a LALR(1) grammar for JSR-14 (see also Eric Blake's work), and a a list of bugs I've found in the 2.0 prototype. The current prototypes are *almost* but still-not-quite able to compile FLEX; I'll probably keep hacking on things until the GJ compiler is bug-free and usable enough to let me get Real Work Done on FLEX.

It's worth noting that, in addition to JSR-14 and JSR-201, also proposed for Java 1.5 are JSR-133 (revisions to the java memory model and threads) and JSR-181 (a metadata facility). There is also an official page for VGJ, the variance-enabled compiler.

I (and others) have also written a package of parameterized collections classes which greatly extend the power of the standard java.util library. This library was originally developed for the FLEX project.

When will the next GJ prototype be released?

SinjDoc: a Javadoc-like tool for JSR-14

This is still in active development. You can download the current version 0.3 sources; caveat emptor. API documentation (no frames version) generated by the tool is also available, as is a list of changes made since the last release. Version 0.3 of SinjDoc is not able to parse variance declarations or JSR-201 syntax (introduced in prototype 2.0 of the JSR-14 compiler).

Sign up on the SinjDoc-Announce mailing list to be notified of new releases of SinjDoc.

For a good example of what this tool can do, look at the generated documentation for the FLEX compiler infrastructure, which is generated using it.

SinjP: a Javap-like tool for JSR-14

I've written a no-frills javap-like tool that a) doesn't choke on bytecode emitted by the JSR-14 compiler, as does the standard Sun javap, and b) actually parses the JSR-14 Signature attribute to emit the correct parameterized signature for the methods in the class. This was invaluable to be in looking through the jar files supplied with the prototype compilers to look for what had changed. Maybe it will be helpful to you, too. I've updated it to support the new signature format of the 2.0 prototype.

To use:

• Download Harpoon.jar which contains the code for the tool. This is actually a full FLEX distribution, which is why the download is rather porky.
• Use it like so:

  % java -cp Harpoon.jar harpoon.Main.Javap java.util.AbstractSet
  

  (with the class you are interested in replacing java.util.AbstractSet, of course.)
  [see sample output]
• Make sure the prototype compiler's collect.jar file is also on your classpath, or you're likely to get the (mostly uninteresting) prototypes of the pre-GJ Collections classes; if the output of the command above doesn't give you GJ types on AbstractSet, this is the most likely reason why.
• (Note that there are odd historical reasons why the name "Harpoon" is associated with the FLEX compiler, which explains the package and jar file names above.)

Updated javadoc for standard libraries

Not yet updated for prototype 2.0.

The 1.2 prototype compiler updated the "generic"-ified Collections (and other?) APIs, but did not come with updated javadoc. As a result, your existing code which extends Collections tends to mysteriously fail because your type signatures no longer match properly. (It didn't help that the prototype compiler used to crash hard for some of these mismatches.) So --- to ease your pain, I've gone through the javadoc with the help of my javap-like tool below and manually tried to fixup the javadoc to be accurate. Note that this is accurate to what's shipping now with the 1.3 prototype; note my list below for details on what I think may be "broken" in this release and thus slated for change in future releases.

So, without further ado, here are some links to the goodies:

• A tgz archive just like you downloaded from Sun. Only, this one has my fixes applied. The legality of redistributing this might be somewhat sketchy, although (because?) there is no license information enclosed.
• A patch against the official Sun distributed files. I should be okay in redistributing this, but it'll take some work on your part to apply.
• For your convenience, I've put the updated Collections API javadoc and Reflection API javadoc here on the web, so you can just browse directly instead of downloading.
• I also have the signature-only javadoc created for the "latest Sun stubs" (as of 10-April-2003) by my sinjdoc tool. This includes some other Java 1.5 goodies which aren't yet in the official JSR-14 prototype compiler.

Bug list for 2.0 prototype compiler

Old bug list for 1.3 prototype compiler.

Old bug list for 1.2 prototype compiler.

Here are the bugs I know about or suspect are in the 2.0 prototype compiler. Note that these may or may not have been confirmed by the good folk at Sun, so some of these might actually be "features" in disguise.

Items in red are those I consider "showstoppers". There are seven of them so far; I've been very conservative marking things showstoppers, not marking them if there seems to be a manageable workaround, the bug might be construed as simply a "missing feature", or (as in the -novariance bugs) there is any doubt as to whether the bug will be relevant for the "final" JSR-14 spec. Your opinions as to seriousness may well vary from mine.

• Errata in section 7 of the specification document. The ParameterSignature and MethodTypeSignature productions are wrong. There are also some additional details which are not covered by the spec. The ClassTypeSignature production in the spec is also inconsistent with what is actually emitted by the compiler for nested parameterized types. Finally, the 2.0 prototype compiler introduced new signatures to represent variance. I've posted a corrected signature grammar which addresses all of these problems.
• I've proposed a new syntax for variance and foreach which adds readability without adding new keywords. Comments in the forum are requested!

  Neil Gafter suggested List<? extends X> for List<+X>, which I like a lot. My refinement of the idea would be List<? instanceof X>, List<? superclassof X>, List<?>, and List<X>. This involves a pseudo-keyword "superclassof", unfortunately — but this seems rare enough that even making it a new keyword shouldn't hurt much.

  Extending this to arrays is an open problem.

• Incorrect GJ prototypes for:
  • java.util.Map.get()
    The signature should be:

    public V get(Object o);
    

    but the signature in the distributed compiler is actually:

    public V get(K o);
    

    I have been assured that this is an oversight, not intentional.
  • java.util.Arrays/java.util.Collections
    The 2.0 prototype has the following signatures in the java.util.Arrays classfile:

        public static <T> void sort(T[=]);
        public static <T> void sort(T[=], int, int);
        public static <T, K> int binarySearch(T[+], K);
    

    and the following signatures in java.util.Collections:

        public static <T> void sort(java.util.List<T>);
    

    We seem to have lost the Comparable bounds to ensure that the elements of these collections can be sorted without a Comparator. Look at this (proper) signature from java.util.Collections:

        public static <T extends Object & Comparable<-T>> int binarySearch(java.util.List<+T>, T);
    

    Similarly, I believe the proper signatures for the methods in java.util.Arrays ought to be:

        public static <T extends Object & Comparable<-T>> void sort(T[=]);
        public static <T extends Object & Comparable<-T>> void sort(T[=], int, int);
        public static <T extends Object & Comparable<-T>gt; int binarySearch(T[+], T);
    

    And the proper signature for Collections.sort() ought to be:

        public static <T extends Object & Comparable<-T>> void sort(java.util.List<T>);
    

  • java.util.Collections
    The Collections.unmodifiableCollection() method, broken in 1.3, has been fixed in 2.0 so that it has the signature:

    public static <T> java.util.Collection<T> unmodifiableCollection(java.util.Collection<+T>);
    

    But not all of the "unmodifiableFOO" methods have been fixed-up. In particular, the 2.0 prototype has non-ideal signatures for unmodifiableMap().

    In addition, the emptySet() and emptyList() methods should probably be made private.

    CORRECTION 19-Jun-2003: Neal Gafter pointed out that the Comparator<-K> accessor method in SortedSet/SortedMap means that unmodifiableSortedSet() and unmodifiableSortedMap() cannot be made covariant. I'd previously suggested in this item that those two methods required signature fixups as well.

  • java.util.HashMap
    In the 1.3 prototype (and present earlier as well), the constructor to HashMap has the signature:

    public <K1 extends K, V1 extends V> HashMap(Map<K1,V1> m);
    

    In the new, variance-enabled compiler, the signature should probably be:

    public HashMap(Map<+K, +V> m);
    

  Although only the first of these incorrect prototypes has caused problems which keep FLEX from compiling, since they are all changes to the API which could cause problems with backwards or forwards compatibility, they should probably all be considered showstoppers.

• The variance presentation included in the 2.0 compiler as variance-overview.pdf discusses the following possible signature for Collection.toArray():

  Collection<E> {
    <T extends E[-]> T toArray(T a);
  }
  

  This example does not compile with the 2.0 prototype; it appears that bounds of array type are not supported by the compiler.
• Variance on multi-dimensional arrays seems to be all screwed up in prototype 2.0. (confirmed by Sun)
• Unavoidable casts in bridge methods. It is impossible to create a JSR-14 implementation of java.util.Map where the entrySet() method behaves according to the specification, which says that entrySet().add(x) should throw an UnsupportedOperationException. Instead, you will often get a ClassCastException instead.

  I think this is less a problem with the JSR-14 implementation than with the specification for the Map.entrySet() method. Note that in prototype 2.0 the signature will not let you do an entrySet().add() operation in any case, so this is only an interoperability concern.

• Cyclic inheritance problem. This is now Sun bug 4862621. This is most relevant to the current GJ developer because code that works now will become erroneous when this bug is fixed, and you can't "fix" your working code now to work in the future because the compiler will not accept the "correct" code now. Discovered in v1.3; still not fixed in v2.0.
• Extra parentheses cause VerifyError at runtime. This bug was discovered by "agentsmooth" on the GJ forum, and seems to exist in every prototype from 1.0 to 2.0. Removing the extraneous parentheses in the example fixes the VerifyError problem, and thus is an acceptable workaround.
• Unboxing appears not to work in the 2.0 prototype. Auto-unboxing was specifically left as optional in the draft spec, but manual unboxing was specified and not implemented. This is now Sun bug 4872709.
• Autoboxing only implemented for method invocation conversion. In particular, a return statement does not seem to be treated as an assignment conversion, as it ought to be. Found by "veenurs" in the GJ forum.
• Enum.valueOf() broken in 2.0. This bug was discovered by "hhkratz" in the GJ forums. The upshot is that the signature and method descriptor for the generated SomeEnum.valueOf() method are wrong (although the generated code is correct). More details are in the full report. This is now Sun bug 4872708.
• Bogus "ambiguous method call" error. This is Sun bug 4872698. (An additional sighting of this bug has since appeared on the GJ forums.)
• NullPointerException w/ foreach in static context. Found by "uncle_alice" in the GJ forums.
• Backwards incompatibilities with implicit conversions. The following code (from "yoseph_phillips" in the GJ forums) compiles with the non-generic compiler, but fails to typecheck with the generic compiler.

  public class Test {
    public static void main(String[] args) {
      Class c = true ? Boolean.class : Object.class;
    }
  }
  

  An implicit conversion to the raw type is wanted here. Similarly, if a constructor:

    C(List l, Comparator c)
  

  is genericized to:

    C(List<Integer> l, Comparator<Integer> c)
  

  then existing invocations of the form:

    Comparator cc = ...;
    new C(Arrays.asList(new Integer[0]), cc)
  

  fail to compile.

  Neither problem occurred with the 1.3 prototype compiler.

• Bug in array signature parsing in -novariance compiler. When reading in precompiled .class files, the "novariance" compiler seems to interpret a signature "X[]" as "X[=]". These types are not equivalent, and so typechecking fails.
• "Map.Entry does not take parameters". The prototype compiler seems to "forget" to use the proper parameterized versions of library functions from time to time, causing error messages like the one quoted above. This is my #1 usability problem with the prototype compiler. Full details in the link.
• Instantiating parameterized types with arrays is broken. Found by Hans Kratz in the GJ forum.
• new E[] creates Object[]. Thanks for "krausest1" on the GJ forum for bringing this to my attention. This is a pretty fundamental problem. Long discussion at the link above; I think the best short-term solution is to disallow new E[] (where E is a type variable) and to change the signature of 'toArray()' to always return an Object[].
• New "for each" syntax over Iterable doesn't work in the -novariance compiler. The stubs/implementation of java.lang.Iterable, java.lang.SimpleIterator, and java.lang.Iterator (extended version) are missing. These work in the VGJ version of the compiler.
• Distributed javac/javac-novariance scripts don't work on Mac OS X. The linked thread in the GJ forums contains code to work around this.
• Compiler crash. Reported by "alexpotanin" in the GJ forums.

C. Scott Ananian

cscottcscott.net