vm.ajr on IBM jre

[rvjansen]

While looking into the nested comment issue I was sidetracked again by
an IBM patch moving the location of vm.jar - again; as the machines are
managed by the Unix department there are surprises as to the timing of
the changes. Also, this seems to impact nothing else than NetRexx.

Now to be found at
/opt/mqm/java/jre64/jre/lib/amd64/compressedrefs/jclSC160/vm.jar , I
wonder how it is that we need to locate something that essential; also,
it shows again that putting stuff in the 'blessed' directories never
really is going to work; in that case I would have had something that
did not fit the jre release, without knowing it.

Kermit, did your context class loader change have any relation to this
or shed any light?

best regards,

René.

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Kermit Kiser]

Hi René --

I take it that vm.jar is part of the IBM Java. We do have some related
hard coded values in the RxClasser module:

       -- add standard location for java6 on zLinux so it does not need
to be put on classpath anymore
path=path||psep||'/opt/ibm/java-s390x-60/lib/tools.jar:/opt/ibm/java-s390x-60/jre/lib/s390x/default/jclSC160/vm.jar'

That is for s390 rather than *ix but it illustrates the extent of the
problem NetRexx is having with these paths.

The classloader issue is separate but related: Java classloaders are
hierarchical, so the higher the NetRexx processor is loaded in the
hierarchy, the less user classes it is able to see and load. Your issue
is probably caused by the way that NetRexx "pre-reads" all imported
classes rather than loading them with a classloader. That forces NetRexx
to physically locate all needed jar files via the classpath or JAVA_HOME
environment pointer, but unfortunately the system jars and other special
classes don't seem to be found that way in some environments.

However, Mike and I had an interesting discussion on this list about a
month ago titled "Reflection" which I cannot attach, so I will paste
part of it here between the lines:
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

> Thanks for the pointer, Mike.
>
> RxClassImage looks at two attributes from the byte-code:
> Deprecated and InnerClasses
>
> The "Deprecated" attribute was replaced in Java 1.5 with the
> @Deprecated annotation which is accessible via reflection.
> The InnerClasses data appears to be used to make invisible
> the passing of the instance value to dependent classes. That
> data seems to be available through reflection also.

In that case, I suspect it may be possible to skip the 'physical' class reading
and use reflection calls instead.  Might be worth doing a performance comparison
first, however.
 

> Question: Is the reading and analysis of all imported classes
> prior to starting the translation done for efficiency? (I am
> assuming that fully qualified references that are not
> imported are handled more adhoc.)

I don't really remember .. suspect it may have been more for simplicity.  Might
be something in the commentary about it; I generally document reasons for
decisions like that (if I made a conscious decision -- there may have been
something else that forced that approach).

Mike

---------------------------------------------------------------------------------------------------------------------------------------------------------------------
As you can see, Mike indicated that it might be possible to skip the
pre-read of the jar files now. And while I agree that we need to check
the performance implications, that does not seem to be an issue for
javac and other compilers and I am sure that they do not "pre-read"
classes. I suggest that we create a new experimental branch to check the
feasibility of removing the pre-reading of classes and to test the
performance impacts.

Although this would be a major refactor of NetRexx, in addition to
eliminating 90% of our classpath/installation issues, the change would
greatly simplify things like Jason's NetRexx on Android project since
most of the effort there is to handle the pre-read requirements of
NetRexx. We would probably lose the current safety check warning that
NetRexx produces about no classes found for an erroneous import
statement, but that seems a minor loss to me for the benefits.

I lack time to take this on at the moment but might be able to look at
it later.

-- Kermit


On 11/27/2012 6:34 AM, rvjansen wrote:

> While looking into the nested comment issue I was sidetracked again by
> an IBM patch moving the location of vm.jar - again; as the machines
> are managed by the Unix department there are surprises as to the
> timing of the changes. Also, this seems to impact nothing else than
> NetRexx.
>
> Now to be found at
> /opt/mqm/java/jre64/jre/lib/amd64/compressedrefs/jclSC160/vm.jar , I
> wonder how it is that we need to locate something that essential;
> also, it shows again that putting stuff in the 'blessed' directories
> never really is going to work; in that case I would have had something
> that did not fit the jre release, without knowing it.
>
> Kermit, did your context class loader change have any relation to this
> or shed any light?
>
> best regards,
>
> René.
>
> _______________________________________________
> Ibm-netrexx mailing list
> [hidden email]
> Online Archive : http://ibm-netrexx.215625.n3.nabble.com/
>

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[ThSITC]

++++++1!

Thanks for
*following this* up, Kermit, Mike and ALL! ... ???Thomas ... Going his
own way, nowadays, *BUT*
Listening, alöl yof you, be the way, *my friends* ..., ;-)

Thomas.
=======================================================================

Am 28.11.2012 02:20, schrieb Kermit Kiser:

> Hi René --
>
> I take it that vm.jar is part of the IBM Java. We do have some related
> hard coded values in the RxClasser module:
>
>       -- add standard location for java6 on zLinux so it does not need
> to be put on classpath anymore
> path=path||psep||'/opt/ibm/java-s390x-60/lib/tools.jar:/opt/ibm/java-s390x-60/jre/lib/s390x/default/jclSC160/vm.jar'
>
>
> That is for s390 rather than *ix but it illustrates the extent of the
> problem NetRexx is having with these paths.
>
> The classloader issue is separate but related: Java classloaders are
> hierarchical, so the higher the NetRexx processor is loaded in the
> hierarchy, the less user classes it is able to see and load. Your
> issue is probably caused by the way that NetRexx "pre-reads" all
> imported classes rather than loading them with a classloader. That
> forces NetRexx to physically locate all needed jar files via the
> classpath or JAVA_HOME environment pointer, but unfortunately the
> system jars and other special classes don't seem to be found that way
> in some environments.
>
> However, Mike and I had an interesting discussion on this list about a
> month ago titled "Reflection" which I cannot attach, so I will paste
> part of it here between the lines:
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>> Thanks for the pointer, Mike.
>>
>> RxClassImage looks at two attributes from the byte-code:
>> Deprecated and InnerClasses
>>
>> The "Deprecated" attribute was replaced in Java 1.5 with the
>> @Deprecated annotation which is accessible via reflection.
>> The InnerClasses data appears to be used to make invisible
>> the passing of the instance value to dependent classes. That
>> data seems to be available through reflection also.
>
> In that case, I suspect it may be possible to skip the 'physical'
> class reading
> and use reflection calls instead.  Might be worth doing a performance
> comparison
> first, however.
>
>
>> Question: Is the reading and analysis of all imported classes
>> prior to starting the translation done for efficiency? (I am
>> assuming that fully qualified references that are not
>> imported are handled more adhoc.)
>
> I don't really remember .. suspect it may have been more for
> simplicity.  Might
> be something in the commentary about it; I generally document reasons for
> decisions like that (if I made a conscious decision -- there may have
> been
> something else that forced that approach).
>
> Mike
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> As you can see, Mike indicated that it might be possible to skip the
> pre-read of the jar files now. And while I agree that we need to check
> the performance implications, that does not seem to be an issue for
> javac and other compilers and I am sure that they do not "pre-read"
> classes. I suggest that we create a new experimental branch to check
> the feasibility of removing the pre-reading of classes and to test the
> performance impacts.
>
> Although this would be a major refactor of NetRexx, in addition to
> eliminating 90% of our classpath/installation issues, the change would
> greatly simplify things like Jason's NetRexx on Android project since
> most of the effort there is to handle the pre-read requirements of
> NetRexx. We would probably lose the current safety check warning that
> NetRexx produces about no classes found for an erroneous import
> statement, but that seems a minor loss to me for the benefits.
>
> I lack time to take this on at the moment but might be able to look at
> it later.
>
> -- Kermit
>
>
> On 11/27/2012 6:34 AM, rvjansen wrote:
>> While looking into the nested comment issue I was sidetracked again
>> by an IBM patch moving the location of vm.jar - again; as the
>> machines are managed by the Unix department there are surprises as to
>> the timing of the changes. Also, this seems to impact nothing else
>> than NetRexx.
>>
>> Now to be found at
>> /opt/mqm/java/jre64/jre/lib/amd64/compressedrefs/jclSC160/vm.jar , I
>> wonder how it is that we need to locate something that essential;
>> also, it shows again that putting stuff in the 'blessed' directories
>> never really is going to work; in that case I would have had
>> something that did not fit the jre release, without knowing it.
>>
>> Kermit, did your context class loader change have any relation to
>> this or shed any light?
>>
>> best regards,
>>
>> René.
>>
>> _______________________________________________
>> Ibm-netrexx mailing list
>> [hidden email]
>> Online Archive : http://ibm-netrexx.215625.n3.nabble.com/
>>
>
> _______________________________________________
> Ibm-netrexx mailing list
> [hidden email]
> Online Archive : http://ibm-netrexx.215625.n3.nabble.com/
>

--
http://www.thsitc.com Austria, Europe Skype ID: Thomas.Schneider.Wien
Yahoo ID: [hidden email] FaceBook ID: Thomas.Schneider.Wien Member
of the Rexx Languge Asscociation (www.rexxla.org) Member of the NetRexx
Developer's Team (www.netrexx.org)

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[rvjansen]

Hi Kermit,

yes it is, and yes, I know that line, I put it there. Now that
packaging keeps changing, and we have more dissimilar JVM's to support,
like Dalvikvm - I wish Sun had made this part of the standard and stuck
to it - not that they (Oracle now) will have any influence over Google
after the trial bombed - instead they are also at the same game since
'97, not to mention Apple that has first done everything their own way
and now dropped the ball entirely and forked their UI work over to
OpenJDK.

I saw that discussion. The pre-reading must be the main cause of all
the mischief here, because it is not imaginable that a running JVM
instance cannot locate java.lang.Object. If we can do a proof of concept
with reflectively validating the interfaces we should be ok; maybe this
should be a goal for 3.03 - I would like to get 3.02 out of the way in
December if at all possible. It also will finally make it possible to
compile from my home directory itself - which is very unwise now and
leads to an enormous disk scan for all jars that it can find.

If this turns out to be viable we should redirect work on the installer
to this - most traditional NetRexx installation problems would be
solved. We need to squeeze in one more thing if we attempt this, which
is multi-language dependencies. This was an idea by Patrick based on
what some other languages do: set up a proxy class to link to when the
dependency is not resolvable by the compiler, e.g. when the original
source is Java, Scala or JRuby. A variant of this is to enable
compilation of other languages under control of the NetRexx translator,
one by one. Here the compiler API work comes into scope. I would like to
see this solved as it was the number one headache in a project some
years ago, I would spend hours tuning the makefile to be able to build
from scratch and caught flak for NetRexx among the nonbelievers.

Ok. Something to think about


On 2012-11-28 02:20, Kermit Kiser wrote:

> Hi René --
>
> I take it that vm.jar is part of the IBM Java. We do have some
> related hard coded values in the RxClasser module:
>
>       -- add standard location for java6 on zLinux so it does not
> need to be put on classpath anymore
> path=path||psep||'/opt/ibm/java-s390x-60/lib/tools.jar:/opt/ibm/java-s390x-60/jre/lib/s390x/default/jclSC160/vm.jar'
>
> That is for s390 rather than *ix but it illustrates the extent of the
> problem NetRexx is having with these paths.
>
> The classloader issue is separate but related: Java classloaders are
> hierarchical, so the higher the NetRexx processor is loaded in the
> hierarchy, the less user classes it is able to see and load. Your
> issue is probably caused by the way that NetRexx "pre-reads" all
> imported classes rather than loading them with a classloader. That
> forces NetRexx to physically locate all needed jar files via the
> classpath or JAVA_HOME environment pointer, but unfortunately the
> system jars and other special classes don't seem to be found that way
> in some environments.
>
> However, Mike and I had an interesting discussion on this list about
> a month ago titled "Reflection" which I cannot attach, so I will paste
> part of it here between the lines:
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
>> Thanks for the pointer, Mike.
>>
>> RxClassImage looks at two attributes from the byte-code:
>> Deprecated and InnerClasses
>>
>> The "Deprecated" attribute was replaced in Java 1.5 with the
>> @Deprecated annotation which is accessible via reflection.
>> The InnerClasses data appears to be used to make invisible
>> the passing of the instance value to dependent classes. That
>> data seems to be available through reflection also.
>
> In that case, I suspect it may be possible to skip the 'physical'
> class reading
> and use reflection calls instead.  Might be worth doing a performance
> comparison
> first, however.
>
>
>> Question: Is the reading and analysis of all imported classes
>> prior to starting the translation done for efficiency? (I am
>> assuming that fully qualified references that are not
>> imported are handled more adhoc.)
>
> I don't really remember .. suspect it may have been more for
> simplicity.  Might
> be something in the commentary about it; I generally document reasons
> for
> decisions like that (if I made a conscious decision -- there may have
> been
> something else that forced that approach).
>
> Mike
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
> As you can see, Mike indicated that it might be possible to skip the
> pre-read of the jar files now. And while I agree that we need to check
> the performance implications, that does not seem to be an issue for
> javac and other compilers and I am sure that they do not "pre-read"
> classes. I suggest that we create a new experimental branch to check
> the feasibility of removing the pre-reading of classes and to test the
> performance impacts.
>
> Although this would be a major refactor of NetRexx, in addition to
> eliminating 90% of our classpath/installation issues, the change would
> greatly simplify things like Jason's NetRexx on Android project since
> most of the effort there is to handle the pre-read requirements of
> NetRexx. We would probably lose the current safety check warning that
> NetRexx produces about no classes found for an erroneous import
> statement, but that seems a minor loss to me for the benefits.
>
> I lack time to take this on at the moment but might be able to look at
> it later.
>
> -- Kermit
>
>
> On 11/27/2012 6:34 AM, rvjansen wrote:
>> While looking into the nested comment issue I was sidetracked again
>> by an IBM patch moving the location of vm.jar - again; as the machines
>> are managed by the Unix department there are surprises as to the
>> timing of the changes. Also, this seems to impact nothing else than
>> NetRexx.
>>
>> Now to be found at
>> /opt/mqm/java/jre64/jre/lib/amd64/compressedrefs/jclSC160/vm.jar , I
>> wonder how it is that we need to locate something that essential;
>> also, it shows again that putting stuff in the 'blessed' directories
>> never really is going to work; in that case I would have had something
>> that did not fit the jre release, without knowing it.
>>
>> Kermit, did your context class loader change have any relation to
>> this or shed any light?
>>
>> best regards,
>>
>> René.
>>
>> _______________________________________________
>> Ibm-netrexx mailing list
>> [hidden email]
>> Online Archive : http://ibm-netrexx.215625.n3.nabble.com/
>>
>
> _______________________________________________
> Ibm-netrexx mailing list
> [hidden email]
> Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[rvjansen]

And darn that webmail interface to heck. The mail escaped.

I wanted to add the following question:

Do we need fullscale dependency analysis with mathematical modelling
before we attempt this, or would a lazy, just in time approach of
catching the exception and firing off a compile, depending on what we
find on the source path, suffice?

best regards,

René.


On 2012-11-28 11:18, rvjansen wrote:

> Hi Kermit,
>
> yes it is, and yes, I know that line, I put it there. Now that
> packaging keeps changing, and we have more dissimilar JVM's to
> support, like Dalvikvm - I wish Sun had made this part of the standard
> and stuck to it - not that they (Oracle now) will have any influence
> over Google after the trial bombed - instead they are also at the same
> game since '97, not to mention Apple that has first done everything
> their own way and now dropped the ball entirely and forked their UI
> work over to OpenJDK.
>
> I saw that discussion. The pre-reading must be the main cause of all
> the mischief here, because it is not imaginable that a running JVM
> instance cannot locate java.lang.Object. If we can do a proof of
> concept with reflectively validating the interfaces we should be ok;
> maybe this should be a goal for 3.03 - I would like to get 3.02 out of
> the way in December if at all possible. It also will finally make it
> possible to compile from my home directory itself - which is very
> unwise now and leads to an enormous disk scan for all jars that it can
> find.
>
> If this turns out to be viable we should redirect work on the
> installer to this - most traditional NetRexx installation problems
> would be solved. We need to squeeze in one more thing if we attempt
> this, which is multi-language dependencies. This was an idea by
> Patrick based on what some other languages do: set up a proxy class to
> link to when the dependency is not resolvable by the compiler, e.g.
> when the original source is Java, Scala or JRuby. A variant of this is
> to enable compilation of other languages under control of the NetRexx
> translator, one by one. Here the compiler API work comes into scope. I
> would like to see this solved as it was the number one headache in a
> project some years ago, I would spend hours tuning the makefile to be
> able to build from scratch and caught flak for NetRexx among the
> nonbelievers.
>
> Ok. Something to think about
>

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/