java.lang.NumberFormatException from java.io.Writer.write()

[ThSITC]

Hello Alan, and all,
   I'm following this discussion.
   Haven't currently engough knowledge about Mike's COSTING algorithm to really comment.

May I, nevertheless, suggest a *new option* COST, NOCOST *or* OLDCOST, NEWCOST
to have a flag just in case anybody doeshave a problem with the switch of the algorithm.

My personal feeling is that the new algorithm should be as KEMIT decsibed. Thank's Kermit.

Thomas.
======================================================================


Am 06.10.2011 23:28, schrieb Alan Sampson:

Rony,

While I agree with many of your thoughts I do take issue with the idea that NetRexx should coerce (as happened in my case) a java.lang.String object into a lang.netrexx.Rexx object and then apply that object's toInt() method to the contents just so it could force a call to java.io.BufferedWriter.write(int), particularly as this was inside a "binary" method in a "binary" class.  Perhaps that's where the distinction should be drawn; if the translator is working with a Rexx object and/or "nobinary" is in effect then the old algorithm could be employed, if casting and/or "binary" is specified then it should employ the new scheme.  Having gone to the trouble of casting the datatype and specifying that I wanted more control over the code, I expected the compiler to respect my request.  At some point the compiler should respect the programmer's wishes.

Alan.

On 6 October 2011 12:05, Rony G. Flatscher <[hidden email]> wrote:

On 05.10.2011 17:51, Kermit Kiser wrote:

The current method selection logic is a recursive algorithm:

1) If any methods in the current class can be used, select the best fit (least cost) and use it.
2) If any methods in the superclass can be used, select the best fit using this algorithm and use it.
3) If any methods in interfaces can be used, select the best fit and use it.

The proposed logic is:

1) If any methods in the current class can be used, save the best fit and it's cost.
2) If any superclass methods can be used, save the best fit using this algorithm and it's cost.
3) If any interface methods can be used, save the best fit and it's cost.
4) Select the best fit (least cost) of the above three and use it.

Thus the new algorithm examines all possible method calls and then selects the best fit (least cost in conversions) whereas the old algorithm preferred local methods over any others, regardless of fit. Note that local methods are still preferred, but ONLY if they fit as well as inherited methods.

Thank you for this description!

This change would replace the defined method resolution order (in NetRexx 2, p. 54) in the post NetRexx 3.01 implementation.

As the new algorithms change the current method resolution order one question would be, how it will affect existing (old) NetRexx projects that get changed and recompiled with the new algorithms. It is likely, that some of these recompilations might pick a different method than is currently the case, if a method in a superclass is "cheaper" in terms of coercion. The behaviour of such projects may as a result change (maybe slightly, which would be even more "dangerous" as one might overlook this).

The original problem Alan found with the current method resolution should be fixable by informing NetRexx to use a method in the super class of BufferedWriter instead, e.g. like

(Writer filebuff).write(rec)

which will create the following (expected) Java code:

(java.io.Writer)filebuff).write(rec);
  

Having read the current method resolution rules makes it clear that the above is indeed the solution. Alan's surprise about the method resolution has probably to do with him not being acquainted in the method resolution rules at that particular point in time and his observation, that a hand-written Java program would indeed pick the "write(java.lang.String)" method in the java.io.Writer class: this is to be expected, as Java is not behaving the same as NetRexx as it follows different lookup-rules. Or with other words: because the NetRexx translation does ad-hoc-conversions using the Rexx datatype to use a Java method in the current class first, one cannot compare (or expect) the same behaviour between the Java compiler and the NetRexx translator!

The NetRexx translator works in Alan's case according to the NetRexx specifications and Alan's surprise (and probably of many savvy Java programmers) has nothing to do with a shortcoming (or error for that matter) of NetRexx. It is just a different behaviour that was intentionally laid out the way it is.

---

So one question would be, whether the new method resolution algorithm should replace the 15 year old method resolution algorithm that has been explicitly documented and applied to numerous NetRexx applications?

One aspect would be whether existing NetRexx programs would break? In the case they need re-translation this seems to be possible!

Another aspect would be whether the new algorithms would always pick the method the programmer expects to be picked? This can probably be doubted. E.g. consider some method xyz(int) in the superclass SC and a method xyz(long) in the current class CC, which extends SC. With the new algorithm, probably the superclass' method SC.xyz(int) would be picked (if the costs of creating an int are cheaper than the costs of creating a long) if the Rexx value was something like "23", whereas the current algorithm would pick CC.xyz(long) in the current class.

Mostlikely NetRexx coders might be surprised by this new behaviour (where not the method in the current class is used/preferred but some method in some superclass instead), if they are not clearly aware of the cost-based algorithm in addition to the method resolution algorithm. (It has to be expected that NetRexx programmers over time forget about the details of the method resolution and cost calculation algorithms. This is probably the reason why Alan and others were surprised about his example as hardly anyone would be always aware of the method resolution algorithms in place.)

---

Having read the current NetRexx specifications about method resolution I would see nothing wrong in NetRexx' behaviour (even though I was surprised myself when I saw Alan's example and scratched my head and had to turn to the NetRexx specifications)!

So another question that arises for me: is this manifested surprise something that is common among the NetRexx programmers or a very rare observation because of runtime errors thrown that do not match the expectations of the coder? If it is a rare phenomenon, then I would rather point to the method resolution algorithm in place and documented in the NetRexx Language book, as a wrong expectation is a probably a user error!

---rony

P.S.: Part of the (expectation) problem may be that knowing there is a cost-based algorithm that helps resolve a method might wrongly lead to the conclusion that NetRexx will always be able to pick the "correct" (i.e. expected) method. That would be an impossible expectation: at translation time the NetRexx translator can only work with assumptions, i.e. trying to guess the best method. If the assumptions do not match the programmer's the "Alan-phenomenon" ;) occurs.

P.P.S.: One could think to solve the problem in a backwardly compatible manner by doing something like BSF4ooRexx: if it is possible that multiple methods could be picked, then let the NetRexx translator do the following:

• use the current method resolution algorithm to pick the method to use, but enclose it into a try block,
• create a catch block in which an algorithm anaylses the Rexx value at hand on the one side and all methods having the same number of argumetns; then try to convert the Rexx value to the argument types until one method is found where no illegal conversion exception is throwin and use that; if no such method can be found even after searching all superclasses then rethrow the original exception.

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

--
Can't tweet, won't tweet!

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Kermit Kiser]

Rony -

When describing this proposed change, I asked for help testing it because of the possibility that existing programs could be affected, something I try hard to avoid where possible.

However you seem to prefer a philosophy discussion over providing actual test cases that show a problem. (Are you a professor of rhetoric?;-) That is OK with me if you don't mind me including some actual test code in the discussion. If nothing else, perhaps we can stimulate additional discussion and testing.

But first I would like to point out some items:

1) Mike has indicated that the method selection/costing algorithms were not carved in stone but were somewhat arbitrary and possibly not the best choice.
2) Alan has encountered an actual problem with running a real practical program due to the current algorithm.
3) I have encountered this problem many times but I will not include that code here because it is a proprietary Android application. To resolve this kind of run time problem I usually have to go back and generate the Java code.
4) I grant that we cannot know for sure if there would be problems reported with the new algorithm until after it is implemented.

Before I address your specific concerns, here is a code sample that I have been using to test this change:

-----------------------------------------------------------------------
s=String 11
i=int 22

test.me(s)
test.me(i)

class base
 method base()
 method me(is=String) static
 say "string="is
 
class test extends base
 method test()
 method me(ii=int) static
 say "int="ii
------------------------------------------------------------------------

Here is the output from compiling with 3.01RC2 and running it:

int=11
int=22

And here is the output from compiling with "after3.01" and running it:

string=11
int=22

As expected the changed algorithm makes both methods visible and the results therefore differ. If this were Java code, anyone looking at it would know that the programmer intended to add a new and separate method to the base class when he extended it because the new "me" method has a different signature. In the current NetRexx however, the new method with different signature does not work alongside of the old one, but overrides it even though the signature is different. Ignore for the moment the fact that anyone extending a class should know what it contains and what they can override. Then either the programmer did not know that he was overriding the original "me" method or in fact he did know it and did it deliberately. If the latter case, he could have accomplished that purpose more obviously by using the same signature for the new method just as a Java programmer would do. If done that way, the new algorithm makes no difference to the results. (Yes, I tested that.)

I submit to you that anyone overriding a method in this way (different signature) did not do so deliberately and did not intend it to happen!

In particular, in the case Alan submitted, the write methods in question were written in Java and so there is no way that the designer intended an override to happen, yet that is exactly what NetRexx has done, creating confusion to all of us! The current NetRexx method selection algorithm does NOT inter-operate compatibly with Java!

As for the what the language manual says, we can and have made changes to it before. Right after this special note in the manual where Mike warns about the search problem:

"Note: When a method is found in a class, superclasses of that class are not searched for methods, even though a lower-cost method may exist in a superclass."

there is a section on Method overriding which describes the normal override procedure which is just like Java uses but there it is termed "exactly overriding", apparently because Mike realized even then that there was a possible problem with "inexact overriding".

You also mentioned the fact that the current algorithm has been in place for "15 years" as a reason that we are stuck with it. But to me that only says that there has been virtually zero maintenance and enhancement to NetRexx since it's creation, probably because IBM to this day does not realize that it is the most valuable thing they ever produced!

Given that we need to clearly tell NetRexx programmers that if they compile old programs with a NetRexx translator "after3.01" the method resolution logic has changed, I contend that recompiling with the new algorithm is more likely to fix problems than to cause them!

-- Kermit


On 10/6/2011 12:05 PM, Rony G. Flatscher wrote:

On 05.10.2011 17:51, Kermit Kiser wrote:

The current method selection logic is a recursive algorithm:

1) If any methods in the current class can be used, select the best fit (least cost) and use it.
2) If any methods in the superclass can be used, select the best fit using this algorithm and use it.
3) If any methods in interfaces can be used, select the best fit and use it.

The proposed logic is:

1) If any methods in the current class can be used, save the best fit and it's cost.
2) If any superclass methods can be used, save the best fit using this algorithm and it's cost.
3) If any interface methods can be used, save the best fit and it's cost.
4) Select the best fit (least cost) of the above three and use it.

Thus the new algorithm examines all possible method calls and then selects the best fit (least cost in conversions) whereas the old algorithm preferred local methods over any others, regardless of fit. Note that local methods are still preferred, but ONLY if they fit as well as inherited methods.

Thank you for this description!

This change would replace the defined method resolution order (in NetRexx 2, p. 54) in the post NetRexx 3.01 implementation.

As the new algorithms change the current method resolution order one question would be, how it will affect existing (old) NetRexx projects that get changed and recompiled with the new algorithms. It is likely, that some of these recompilations might pick a different method than is currently the case, if a method in a superclass is "cheaper" in terms of coercion. The behaviour of such projects may as a result change (maybe slightly, which would be even more "dangerous" as one might overlook this).

The original problem Alan found with the current method resolution should be fixable by informing NetRexx to use a method in the super class of BufferedWriter instead, e.g. like

(Writer filebuff).write(rec)

which will create the following (expected) Java code:

(java.io.Writer)filebuff).write(rec);
  

Having read the current method resolution rules makes it clear that the above is indeed the solution. Alan's surprise about the method resolution has probably to do with him not being acquainted in the method resolution rules at that particular point in time and his observation, that a hand-written Java program would indeed pick the "write(java.lang.String)" method in the java.io.Writer class: this is to be expected, as Java is not behaving the same as NetRexx as it follows different lookup-rules. Or with other words: because the NetRexx translation does ad-hoc-conversions using the Rexx datatype to use a Java method in the current class first, one cannot compare (or expect) the same behaviour between the Java compiler and the NetRexx translator!

The NetRexx translator works in Alan's case according to the NetRexx specifications and Alan's surprise (and probably of many savvy Java programmers) has nothing to do with a shortcoming (or error for that matter) of NetRexx. It is just a different behaviour that was intentionally laid out the way it is.

---

So one question would be, whether the new method resolution algorithm should replace the 15 year old method resolution algorithm that has been explicitly documented and applied to numerous NetRexx applications?

One aspect would be whether existing NetRexx programs would break? In the case they need re-translation this seems to be possible!

Another aspect would be whether the new algorithms would always pick the method the programmer expects to be picked? This can probably be doubted. E.g. consider some method xyz(int) in the superclass SC and a method xyz(long) in the current class CC, which extends SC. With the new algorithm, probably the superclass' method SC.xyz(int) would be picked (if the costs of creating an int are cheaper than the costs of creating a long) if the Rexx value was something like "23", whereas the current algorithm would pick CC.xyz(long) in the current class.

Mostlikely NetRexx coders might be surprised by this new behaviour (where not the method in the current class is used/preferred but some method in some superclass instead), if they are not clearly aware of the cost-based algorithm in addition to the method resolution algorithm. (It has to be expected that NetRexx programmers over time forget about the details of the method resolution and cost calculation algorithms. This is probably the reason why Alan and others were surprised about his example as hardly anyone would be always aware of the method resolution algorithms in place.)

---

Having read the current NetRexx specifications about method resolution I would see nothing wrong in NetRexx' behaviour (even though I was surprised myself when I saw Alan's example and scratched my head and had to turn to the NetRexx specifications)!

So another question that arises for me: is this manifested surprise something that is common among the NetRexx programmers or a very rare observation because of runtime errors thrown that do not match the expectations of the coder? If it is a rare phenomenon, then I would rather point to the method resolution algorithm in place and documented in the NetRexx Language book, as a wrong expectation is a probably a user error!

---rony

P.S.: Part of the (expectation) problem may be that knowing there is a cost-based algorithm that helps resolve a method might wrongly lead to the conclusion that NetRexx will always be able to pick the "correct" (i.e. expected) method. That would be an impossible expectation: at translation time the NetRexx translator can only work with assumptions, i.e. trying to guess the best method. If the assumptions do not match the programmer's the "Alan-phenomenon" ;) occurs.

P.P.S.: One could think to solve the problem in a backwardly compatible manner by doing something like BSF4ooRexx: if it is possible that multiple methods could be picked, then let the NetRexx translator do the following:

• use the current method resolution algorithm to pick the method to use, but enclose it into a try block,
• create a catch block in which an algorithm anaylses the Rexx value at hand on the one side and all methods having the same number of argumetns; then try to convert the Rexx value to the argument types until one method is found where no illegal conversion exception is throwin and use that; if no such method can be found even after searching all superclasses then rethrow the original exception.

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[George Hovey-2]

Kermit,
I agree with your conclusion that the current way introduces too much "surprise" and your solution sounds right to me.  I especially dislike the divergence from Java behavior.

  On the other hand, the possibility of breaking programs must give us pause, especially if it happens silently.  Could the compiler issue a warning when a method selected is other than would have occurred under the current way?  It might also give, on demand, a detailed accounting of its cost deliberations (obviously this should be localized to a specified method that seems to be misbehaving).

If the current way is really unsatisfactory (and I agree it is) we should correct it, take our licks, and move on.
George

On Fri, Oct 7, 2011 at 4:50 PM, Kermit Kiser <[hidden email]> wrote:

Rony -

When describing this proposed change, I asked for help testing it because of the possibility that existing programs could be affected, something I try hard to avoid where possible.

However you seem to prefer a philosophy discussion over providing actual test cases that show a problem. (Are you a professor of rhetoric?;-) That is OK with me if you don't mind me including some actual test code in the discussion. If nothing else, perhaps we can stimulate additional discussion and testing.

But first I would like to point out some items:

1) Mike has indicated that the method selection/costing algorithms were not carved in stone but were somewhat arbitrary and possibly not the best choice.
2) Alan has encountered an actual problem with running a real practical program due to the current algorithm.
3) I have encountered this problem many times but I will not include that code here because it is a proprietary Android application. To resolve this kind of run time problem I usually have to go back and generate the Java code.
4) I grant that we cannot know for sure if there would be problems reported with the new algorithm until after it is implemented.

Before I address your specific concerns, here is a code sample that I have been using to test this change:

-----------------------------------------------------------------------
s=String 11
i=int 22

test.me(s)
test.me(i)

class base
 method base()
 method me(is=String) static
 say "string="is
 
class test extends base
 method test()
 method me(ii=int) static
 say "int="ii
------------------------------------------------------------------------

Here is the output from compiling with 3.01RC2 and running it:

int=11
int=22

And here is the output from compiling with "after3.01" and running it:

string=11
int=22

As expected the changed algorithm makes both methods visible and the results therefore differ. If this were Java code, anyone looking at it would know that the programmer intended to add a new and separate method to the base class when he extended it because the new "me" method has a different signature. In the current NetRexx however, the new method with different signature does not work alongside of the old one, but overrides it even though the signature is different. Ignore for the moment the fact that anyone extending a class should know what it contains and what they can override. Then either the programmer did not know that he was overriding the original "me" method or in fact he did know it and did it deliberately. If the latter case, he could have accomplished that purpose more obviously by using the same signature for the new method just as a Java programmer would do. If done that way, the new algorithm makes no difference to the results. (Yes, I tested that.)

I submit to you that anyone overriding a method in this way (different signature) did not do so deliberately and did not intend it to happen!

In particular, in the case Alan submitted, the write methods in question were written in Java and so there is no way that the designer intended an override to happen, yet that is exactly what NetRexx has done, creating confusion to all of us! The current NetRexx method selection algorithm does NOT inter-operate compatibly with Java!

As for the what the language manual says, we can and have made changes to it before. Right after this special note in the manual where Mike warns about the search problem:

"Note: When a method is found in a class, superclasses of that class are not searched for methods, even though a lower-cost method may exist in a superclass."

there is a section on Method overriding which describes the normal override procedure which is just like Java uses but there it is termed "exactly overriding", apparently because Mike realized even then that there was a possible problem with "inexact overriding".

You also mentioned the fact that the current algorithm has been in place for "15 years" as a reason that we are stuck with it. But to me that only says that there has been virtually zero maintenance and enhancement to NetRexx since it's creation, probably because IBM to this day does not realize that it is the most valuable thing they ever produced!

Given that we need to clearly tell NetRexx programmers that if they compile old programs with a NetRexx translator "after3.01" the method resolution logic has changed, I contend that recompiling with the new algorithm is more likely to fix problems than to cause them!

-- Kermit

On 10/6/2011 12:05 PM, Rony G. Flatscher wrote:

On 05.10.2011 17:51, Kermit Kiser wrote:

The current method selection logic is a recursive algorithm:

1) If any methods in the current class can be used, select the best fit (least cost) and use it.
2) If any methods in the superclass can be used, select the best fit using this algorithm and use it.
3) If any methods in interfaces can be used, select the best fit and use it.

The proposed logic is:

1) If any methods in the current class can be used, save the best fit and it's cost.
2) If any superclass methods can be used, save the best fit using this algorithm and it's cost.
3) If any interface methods can be used, save the best fit and it's cost.
4) Select the best fit (least cost) of the above three and use it.

Thus the new algorithm examines all possible method calls and then selects the best fit (least cost in conversions) whereas the old algorithm preferred local methods over any others, regardless of fit. Note that local methods are still preferred, but ONLY if they fit as well as inherited methods.

Thank you for this description!

This change would replace the defined method resolution order (in NetRexx 2, p. 54) in the post NetRexx 3.01 implementation.

As the new algorithms change the current method resolution order one question would be, how it will affect existing (old) NetRexx projects that get changed and recompiled with the new algorithms. It is likely, that some of these recompilations might pick a different method than is currently the case, if a method in a superclass is "cheaper" in terms of coercion. The behaviour of such projects may as a result change (maybe slightly, which would be even more "dangerous" as one might overlook this).

The original problem Alan found with the current method resolution should be fixable by informing NetRexx to use a method in the super class of BufferedWriter instead, e.g. like

(Writer filebuff).write(rec)

which will create the following (expected) Java code:

(java.io.Writer)filebuff).write(rec);
  

Having read the current method resolution rules makes it clear that the above is indeed the solution. Alan's surprise about the method resolution has probably to do with him not being acquainted in the method resolution rules at that particular point in time and his observation, that a hand-written Java program would indeed pick the "write(java.lang.String)" method in the java.io.Writer class: this is to be expected, as Java is not behaving the same as NetRexx as it follows different lookup-rules. Or with other words: because the NetRexx translation does ad-hoc-conversions using the Rexx datatype to use a Java method in the current class first, one cannot compare (or expect) the same behaviour between the Java compiler and the NetRexx translator!

The NetRexx translator works in Alan's case according to the NetRexx specifications and Alan's surprise (and probably of many savvy Java programmers) has nothing to do with a shortcoming (or error for that matter) of NetRexx. It is just a different behaviour that was intentionally laid out the way it is.

---

So one question would be, whether the new method resolution algorithm should replace the 15 year old method resolution algorithm that has been explicitly documented and applied to numerous NetRexx applications?

One aspect would be whether existing NetRexx programs would break? In the case they need re-translation this seems to be possible!

Another aspect would be whether the new algorithms would always pick the method the programmer expects to be picked? This can probably be doubted. E.g. consider some method xyz(int) in the superclass SC and a method xyz(long) in the current class CC, which extends SC. With the new algorithm, probably the superclass' method SC.xyz(int) would be picked (if the costs of creating an int are cheaper than the costs of creating a long) if the Rexx value was something like "23", whereas the current algorithm would pick CC.xyz(long) in the current class.

Mostlikely NetRexx coders might be surprised by this new behaviour (where not the method in the current class is used/preferred but some method in some superclass instead), if they are not clearly aware of the cost-based algorithm in addition to the method resolution algorithm. (It has to be expected that NetRexx programmers over time forget about the details of the method resolution and cost calculation algorithms. This is probably the reason why Alan and others were surprised about his example as hardly anyone would be always aware of the method resolution algorithms in place.)

---

Having read the current NetRexx specifications about method resolution I would see nothing wrong in NetRexx' behaviour (even though I was surprised myself when I saw Alan's example and scratched my head and had to turn to the NetRexx specifications)!

So another question that arises for me: is this manifested surprise something that is common among the NetRexx programmers or a very rare observation because of runtime errors thrown that do not match the expectations of the coder? If it is a rare phenomenon, then I would rather point to the method resolution algorithm in place and documented in the NetRexx Language book, as a wrong expectation is a probably a user error!

---rony

P.S.: Part of the (expectation) problem may be that knowing there is a cost-based algorithm that helps resolve a method might wrongly lead to the conclusion that NetRexx will always be able to pick the "correct" (i.e. expected) method. That would be an impossible expectation: at translation time the NetRexx translator can only work with assumptions, i.e. trying to guess the best method. If the assumptions do not match the programmer's the "Alan-phenomenon" ;) occurs.

P.P.S.: One could think to solve the problem in a backwardly compatible manner by doing something like BSF4ooRexx: if it is possible that multiple methods could be picked, then let the NetRexx translator do the following:

• use the current method resolution algorithm to pick the method to use, but enclose it into a try block,
• create a catch block in which an algorithm anaylses the Rexx value at hand on the one side and all methods having the same number of argumetns; then try to convert the Rexx value to the argument types until one method is found where no illegal conversion exception is throwin and use that; if no such method can be found even after searching all superclasses then rethrow the original exception.

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Rony G. Flatscher (wu-wien)]

Hi Kermit,

On 07.10.2011 22:50, Kermit Kiser wrote:

When describing this proposed change, I asked for help testing it because of the possibility that existing programs could be affected, something I try hard to avoid where possible.

That is the reason why I try to help.

However you seem to prefer a philosophy discussion over providing actual test cases that show a problem. (Are you a professor of rhetoric?;-)

This seems to be a rhetorical question.
;)

That is OK with me if you don't mind me including some actual test code in the discussion. If nothing else, perhaps we can stimulate additional discussion and testing.

But first I would like to point out some items:

1) Mike has indicated that the method selection/costing algorithms were not carved in stone but were somewhat arbitrary and possibly not the best choice.
2) Alan has encountered an actual problem with running a real practical program due to the current algorithm.
3) I have encountered this problem many times but I will not include that code here because it is a proprietary Android application. To resolve this kind of run time problem I usually have to go back and generate the Java code.
4) I grant that we cannot know for sure if there would be problems reported with the new algorithm until after it is implemented.

Before I address your specific concerns, here is a code sample that I have been using to test this change:

-----------------------------------------------------------------------
s=String 11
i=int 22

test.me(s)
test.me(i)

class base
 method base()
 method me(is=String) static
 say "string="is
 
class test extends base
 method test()
 method me(ii=int) static
 say "int="ii
------------------------------------------------------------------------

Here is the output from compiling with 3.01RC2 and running it:

int=11
int=22

And here is the output from compiling with "after3.01" and running it:

string=11
int=22

As expected the changed algorithm makes both methods visible and the results therefore differ. If this were Java code, anyone looking at it would know that the programmer intended to add a new and separate method to the base class when he extended it because the new "me" method has a different signature. In the current NetRexx however, the new method with different signature does not work alongside of the old one, but overrides it even though the signature is different. Ignore for the moment the fact that anyone extending a class should know what it contains and what they can override. Then either the programmer did not know that he was overriding the original "me" method or in fact he did know it and did it deliberately. If the latter case, he could have accomplished that purpose more obviously by using the same signature for the new method just as a Java programmer would do. If done that way, the new algorithm makes no difference to the results. (Yes, I tested that.)

But what, if the programmer always wants to have test.me(int) picked for "s" and "i" in this NetRexx program?

And if he wanted base.me(String) to be used instead, why didn't he state so explicitly?

I submit to you that anyone overriding a method in this way (different signature) did not do so deliberately and did not intend it to happen!

In particular, in the case Alan submitted, the write methods in question were written in Java and so there is no way that the designer intended an override to happen, yet that is exactly what NetRexx has done, creating confusion to all of us! The current NetRexx method selection algorithm does NOT inter-operate compatibly with Java!

Indeed, the NetRexx language is *not* the Java language ! They have different language specifications. (NetRexx defines different principles compared to Java, trying to make programming easier than possible with the Java language itself, which is probably an important reason to use NetRexx over Java.)

As for the what the language manual says, we can and have made changes to it before. Right after this special note in the manual where Mike warns about the search problem:

"Note: When a method is found in a class, superclasses of that class are not searched for methods, even though a lower-cost method may exist in a superclass."

there is a section on Method overriding which describes the normal override procedure which is just like Java uses but there it is termed "exactly overriding", apparently because Mike realized even then that there was a possible problem with "inexact overriding".

You also mentioned the fact that the current algorithm has been in place for "15 years" as a reason that we are stuck with it. But to me that only says that there has been virtually zero maintenance and enhancement to NetRexx since it's creation, probably because IBM to this day does not realize that it is the most valuable thing they ever produced!

The purpose of this was to hint that it might be possible that quite a few NetRexx programs that have been developed in such a long time, *might* be affected by this change. It is about backward-compatibilty with such systems, that may break, if retranslated with the new behaviour (changing the method resolution mechanism changes the behaviiour).

Given that we need to clearly tell NetRexx programmers that if they compile old programs with a NetRexx translator "after3.01" the method resolution logic has changed, I contend that recompiling with the new algorithm is more likely to fix problems than to cause them!

That would be, short of available hard figures, pure speculation/expectations, but nothing that needs to materialize.
(If the new method resolution algorithms get implemented, then surely warnings would help to alert NetRexx coders that a new translation has changed the method NetRexx will use from then on. )

---

Maybe a few other points for the discussion:

• Studying the NetRexx specification and analyzing the Java code the translator creates and weighing what could be dubbed "Rexx semantics": the Rexx datatype in NetRexx seems to not be strictly typed for the NetRexx programmer: the permitted operations depend on the operand values to be adhering/produceable, which ultimately can only be done at runtime and not at compile time.
  
  
• The NetRexx translator currently decides the method it picks at compile time, and not at runtime!
  
  
• No matter what method resolution algorithms one puts in place, it can always be the case that the picked method cannot be used with the Rexx values at runtime, if their values are not compatible with the method's one. This has been demonstrated by Alan's program. If his program had used different string values it might have worked flawlessly, e.g. changing the String array to:
  
  
  
  players = [String -
      '  0 ' -
    , '1' -
  ]
  
  
• The NetRexx programmer probably would know what values to expect at runtime and hence knows which method should be used. If this method differs from the one that NetRexx picks, a problem at runtime may occur.
  
  
• The current NetRexx translator seems to assume, that the Rexx values it uses can always be used to create any of the types the picked method needs at runtime. Therefore, if there are candidate methods under this assumption, any of these candidate methods could be picked. It seems that using a cost-based algorithm serves two purposes then: one is to make the value coercions "cheap" (faster) and to make method picking reproducable ("stable"). This of course assumes too, that a NetRexx programmer would pick methods adhering to these pragmatics.
  
  
• These assumptions seem to break with Alan's program.
  
• However, if one is aware of how NetRexx currently picks a method (not being Java nor a Java compiler), then it is clear that NetRexx picked the "correct" method. 
• Alan could nevertheless force the NetRexx translator to pick the method, he needs for his particular program, knowing that the type must be a java.lang.String in order to get the method picked out of the superclass, instead of the local class. This is also the resolution of his problem.
  
  
• Following the NetRexx list for many years as a silent reader/lurker, I do not recall that the NetRexx method resolution was indeed surprising or even causing problems to NetRexx programmers. Therefore I would assume, that this observation, although at first sight "surprising", but after re-educating with the NetRexx method picking algorithm a very understandable choice, is not a real problem.
  
  
• What would be the behaviour of a NetRexx programmer who is creating a program and using the Java documentation?
• He would probably look up the definition of the BufferedWriter class, e.g. <http://download.oracle.com/javase/6/docs/api/java/io/BufferedWriter.html> and learn that the write(String) method is not implemented.
  
• Knowing about inheritance he would probably lookup the super class "Writer", e.g. <http://download.oracle.com/javase/6/docs/api/java/io/BufferedWriter.html>, where he finds the desired method.
  
• Knowing that NetRexx will use methods of the current class (which is a BufferWriter object in this case), if available, using coercion with the help of the Rexx methods to the desired target type (if those types are primitive or String), he knows that he has to tell NetRexx to use the method in the superclass instead. So he would simply do that.
  
  
• What would be the behaviour of a programmer who is creating a program and using existing Java programs, if available?
• He would probably transcribe a Java program 1:1 to NetRexx.
• He would be surprised, if the NetRexx version does not run without errors.
• So the question would be: whose fault is this? The programmer's or NetRexx?
  

So these thoughts probably boil down to something like:

• Is the current NetRexx method resolution algorithm difficult to understand and to memorize?
• Is the current NetRexx method resolution algorithm behaving in a surprisable way for NetRexx programmers?
  No, if the NetRexx method resolution is known: no, by no means! It is clear and simple.
• Is the current NetRexx method resolution algorithm behaving differently to Java and therefore surprising at times?
  Yes. But then, Java is a statically typed language where each type is known at compile time. (This is not the case with NetRexx Rexx types; they can be rendered to any primitive type and String type at runtime.)
• Is this a problem?
  Honestly, I do not know, but tend to think that it is a minor problem, if the NetRexx coders are aware of the NetRexx method resolution algorithms. (And they should, if they use NetRexx.)
• Are the new algorithms solving the problem: probably not. They solve the case in hand where a method in a superclass should be picked, if its arguments are strings (or so it seems). But what if indeed the programmer wanted a more costly method in the source class (if Alan had used numbers in his string, then he probably wanted to use the BufferedWriter's write(int) method).
• Are the new algorithms introducing new problems: probably, if a method in a superclass is picked instead, that the NetRexx programmer was not aware of that it existed.
  

HTH,

---rony




_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Kermit Kiser]

Hi George -

The warning message sounds like a good idea but what would it say? Most programmers will not know or care about the difference so it might not make sense to them and certainly I would want an option to NOT issue such distracting warning messages.

As for the "detailed accounting of cost deliberations" idea, I know how to display the final cost (I did that while testing the latest changes) but I would not have any idea how to detail and localize it as it is an iterative process which accumulates costs as each conversion is considered and discards items that it decides are irrelevant. Is it really useful to see that three conversions to int costs less than a conversion to string plus a conversion to char? (That was just hypothetical - I have no idea what they really cost.)

I certainly agree with the need to correct it and move on!

-- Kermit


On 10/7/2011 4:39 PM, George Hovey wrote:

Kermit,
I agree with your conclusion that the current way introduces too much "surprise" and your solution sounds right to me.  I especially dislike the divergence from Java behavior.

  On the other hand, the possibility of breaking programs must give us pause, especially if it happens silently.  Could the compiler issue a warning when a method selected is other than would have occurred under the current way?  It might also give, on demand, a detailed accounting of its cost deliberations (obviously this should be localized to a specified method that seems to be misbehaving).

If the current way is really unsatisfactory (and I agree it is) we should correct it, take our licks, and move on.
George

On Fri, Oct 7, 2011 at 4:50 PM, Kermit Kiser <[hidden email]> wrote:

Rony -

When describing this proposed change, I asked for help testing it because of the possibility that existing programs could be affected, something I try hard to avoid where possible.

However you seem to prefer a philosophy discussion over providing actual test cases that show a problem. (Are you a professor of rhetoric?;-) That is OK with me if you don't mind me including some actual test code in the discussion. If nothing else, perhaps we can stimulate additional discussion and testing.

But first I would like to point out some items:

1) Mike has indicated that the method selection/costing algorithms were not carved in stone but were somewhat arbitrary and possibly not the best choice.
2) Alan has encountered an actual problem with running a real practical program due to the current algorithm.
3) I have encountered this problem many times but I will not include that code here because it is a proprietary Android application. To resolve this kind of run time problem I usually have to go back and generate the Java code.
4) I grant that we cannot know for sure if there would be problems reported with the new algorithm until after it is implemented.

Before I address your specific concerns, here is a code sample that I have been using to test this change:

-----------------------------------------------------------------------
s=String 11
i=int 22

test.me(s)
test.me(i)

class base
 method base()
 method me(is=String) static
 say "string="is
 
class test extends base
 method test()
 method me(ii=int) static
 say "int="ii
------------------------------------------------------------------------

Here is the output from compiling with 3.01RC2 and running it:

int=11
int=22

And here is the output from compiling with "after3.01" and running it:

string=11
int=22

As expected the changed algorithm makes both methods visible and the results therefore differ. If this were Java code, anyone looking at it would know that the programmer intended to add a new and separate method to the base class when he extended it because the new "me" method has a different signature. In the current NetRexx however, the new method with different signature does not work alongside of the old one, but overrides it even though the signature is different. Ignore for the moment the fact that anyone extending a class should know what it contains and what they can override. Then either the programmer did not know that he was overriding the original "me" method or in fact he did know it and did it deliberately. If the latter case, he could have accomplished that purpose more obviously by using the same signature for the new method just as a Java programmer would do. If done that way, the new algorithm makes no difference to the results. (Yes, I tested that.)

I submit to you that anyone overriding a method in this way (different signature) did not do so deliberately and did not intend it to happen!

In particular, in the case Alan submitted, the write methods in question were written in Java and so there is no way that the designer intended an override to happen, yet that is exactly what NetRexx has done, creating confusion to all of us! The current NetRexx method selection algorithm does NOT inter-operate compatibly with Java!

As for the what the language manual says, we can and have made changes to it before. Right after this special note in the manual where Mike warns about the search problem:

"Note: When a method is found in a class, superclasses of that class are not searched for methods, even though a lower-cost method may exist in a superclass."

there is a section on Method overriding which describes the normal override procedure which is just like Java uses but there it is termed "exactly overriding", apparently because Mike realized even then that there was a possible problem with "inexact overriding".

You also mentioned the fact that the current algorithm has been in place for "15 years" as a reason that we are stuck with it. But to me that only says that there has been virtually zero maintenance and enhancement to NetRexx since it's creation, probably because IBM to this day does not realize that it is the most valuable thing they ever produced!

Given that we need to clearly tell NetRexx programmers that if they compile old programs with a NetRexx translator "after3.01" the method resolution logic has changed, I contend that recompiling with the new algorithm is more likely to fix problems than to cause them!

-- Kermit

On 10/6/2011 12:05 PM, Rony G. Flatscher wrote:

On 05.10.2011 17:51, Kermit Kiser wrote:

The current method selection logic is a recursive algorithm:

1) If any methods in the current class can be used, select the best fit (least cost) and use it.
2) If any methods in the superclass can be used, select the best fit using this algorithm and use it.
3) If any methods in interfaces can be used, select the best fit and use it.

The proposed logic is:

1) If any methods in the current class can be used, save the best fit and it's cost.
2) If any superclass methods can be used, save the best fit using this algorithm and it's cost.
3) If any interface methods can be used, save the best fit and it's cost.
4) Select the best fit (least cost) of the above three and use it.

Thus the new algorithm examines all possible method calls and then selects the best fit (least cost in conversions) whereas the old algorithm preferred local methods over any others, regardless of fit. Note that local methods are still preferred, but ONLY if they fit as well as inherited methods.

Thank you for this description!

This change would replace the defined method resolution order (in NetRexx 2, p. 54) in the post NetRexx 3.01 implementation.

As the new algorithms change the current method resolution order one question would be, how it will affect existing (old) NetRexx projects that get changed and recompiled with the new algorithms. It is likely, that some of these recompilations might pick a different method than is currently the case, if a method in a superclass is "cheaper" in terms of coercion. The behaviour of such projects may as a result change (maybe slightly, which would be even more "dangerous" as one might overlook this).

The original problem Alan found with the current method resolution should be fixable by informing NetRexx to use a method in the super class of BufferedWriter instead, e.g. like

(Writer filebuff).write(rec)

which will create the following (expected) Java code:

(java.io.Writer)filebuff).write(rec);
  

Having read the current method resolution rules makes it clear that the above is indeed the solution. Alan's surprise about the method resolution has probably to do with him not being acquainted in the method resolution rules at that particular point in time and his observation, that a hand-written Java program would indeed pick the "write(java.lang.String)" method in the java.io.Writer class: this is to be expected, as Java is not behaving the same as NetRexx as it follows different lookup-rules. Or with other words: because the NetRexx translation does ad-hoc-conversions using the Rexx datatype to use a Java method in the current class first, one cannot compare (or expect) the same behaviour between the Java compiler and the NetRexx translator!

The NetRexx translator works in Alan's case according to the NetRexx specifications and Alan's surprise (and probably of many savvy Java programmers) has nothing to do with a shortcoming (or error for that matter) of NetRexx. It is just a different behaviour that was intentionally laid out the way it is.

---

So one question would be, whether the new method resolution algorithm should replace the 15 year old method resolution algorithm that has been explicitly documented and applied to numerous NetRexx applications?

One aspect would be whether existing NetRexx programs would break? In the case they need re-translation this seems to be possible!

Another aspect would be whether the new algorithms would always pick the method the programmer expects to be picked? This can probably be doubted. E.g. consider some method xyz(int) in the superclass SC and a method xyz(long) in the current class CC, which extends SC. With the new algorithm, probably the superclass' method SC.xyz(int) would be picked (if the costs of creating an int are cheaper than the costs of creating a long) if the Rexx value was something like "23", whereas the current algorithm would pick CC.xyz(long) in the current class.

Mostlikely NetRexx coders might be surprised by this new behaviour (where not the method in the current class is used/preferred but some method in some superclass instead), if they are not clearly aware of the cost-based algorithm in addition to the method resolution algorithm. (It has to be expected that NetRexx programmers over time forget about the details of the method resolution and cost calculation algorithms. This is probably the reason why Alan and others were surprised about his example as hardly anyone would be always aware of the method resolution algorithms in place.)

---

Having read the current NetRexx specifications about method resolution I would see nothing wrong in NetRexx' behaviour (even though I was surprised myself when I saw Alan's example and scratched my head and had to turn to the NetRexx specifications)!

So another question that arises for me: is this manifested surprise something that is common among the NetRexx programmers or a very rare observation because of runtime errors thrown that do not match the expectations of the coder? If it is a rare phenomenon, then I would rather point to the method resolution algorithm in place and documented in the NetRexx Language book, as a wrong expectation is a probably a user error!

---rony

P.S.: Part of the (expectation) problem may be that knowing there is a cost-based algorithm that helps resolve a method might wrongly lead to the conclusion that NetRexx will always be able to pick the "correct" (i.e. expected) method. That would be an impossible expectation: at translation time the NetRexx translator can only work with assumptions, i.e. trying to guess the best method. If the assumptions do not match the programmer's the "Alan-phenomenon" ;) occurs.

P.P.S.: One could think to solve the problem in a backwardly compatible manner by doing something like BSF4ooRexx: if it is possible that multiple methods could be picked, then let the NetRexx translator do the following:

• use the current method resolution algorithm to pick the method to use, but enclose it into a try block,
• create a catch block in which an algorithm anaylses the Rexx value at hand on the one side and all methods having the same number of argumetns; then try to convert the Rexx value to the argument types until one method is found where no illegal conversion exception is throwin and use that; if no such method can be found even after searching all superclasses then rethrow the original exception.

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Kermit Kiser]

Rony -

I see that you have shared an even longer philosophical discussion with no real world examples of programs where the new algorithm would cause a problem. Sadly, I do not have time to respond in kind. However, I promise you that I have read your message twice and considered it deeply although I don't really see much in the way of new concerns there. I will try to share my view briefly.

First, in the example that I offered it seems very clear to me that by extending the base class without deliberately overriding the "me" method that accepts strings, the programmer has indeed stated explicitly that he wants to use that inherited method where possible. That is what inheritance is about.

Second, most of your discussion seems to center on a hypothetical NetRexx programmer who wants to do things the traditional "NetRexx" way even though it is not really compatible with the Java way. I have only one response to that discussion:

NetRexx was designed AND advertised as an easy and efficient way to create Java compatible programs.

I am quite sure that most if not all NetRexx programmers are creating applications that inter-operate with Java systems and they want the maximum compatibility with Java that is possible.

However the possibility of breaking existing programs is always an important concern even when trying to improve NetRexx compatibility with Java. I have compiled over 200 NetRexx programs (all of the Rosettacode examples plus my own test case programs) with the new algorithms without a problem. Of course this change affects runtime results and I was not able to run all of those programs due to time limitations. That is why any example of a real program where the new algorithms affected something would be valuable! In the case of Alan's real world example, the new algorithm fixes a runtime problem. There is no known example of a real program being negatively affected.

Since it is not possible to go back and test every NetRexx program ever written with the new algorithm, the best possible course to improve Java compatibility is to do as much testing as time constraints allow and then to notify programmers who download the new changed version of NetRexx that there is a remote possibility that some old programs might behave differently if compiled with the new version so they should either do a complete retest of any older application they modify and recompile or simply use NetRexx 3.01 or earlier for maintenance of older programs.

Thanks to your inspiration to research this issue more deeply, as well as my confidence that we can improve Java compatibility without hurting existing applications, I have discovered another change (now on the after3.01 branch at rev 156) to the sacred cow method selection/costing algorithms that allows NetRexx to compile the original uncompilable source example (see below) that I submitted on Kenai with issue 11. This change allows programmers to call interface methods that have been implemented by a class, even while using NetRexx automatic type conversions. That was not possible before the change because both the interface method and the implemented method obviously have the same signature and hence the same cost which led to the "ambiguous method" message: "More than one method matches..."

The latest binary version of the NetRexx translator including all method selection changes can be found on the NetRexx Plus project at this URL:

http://kenai.com/projects/netrexx-plus/downloads

-- Kermit

PS: Here is the test code that will not compile with 3.01 or earlier because NetRexx could not handle the CharSequence interfaces Java 1.5+ uses -

say "test charseq conversion issues"

stringdata=String "xyz"
csq=CharSequence stringdata
xxx(csq)
rexxdata=Rexx "xyz"
yyy(rexxdata)
yyy(stringdata)
is=int 0
rexxis=Rexx 0
ie=int 1
rexxie=Rexx 1

mystream=PrintStream("\\Documents and Settings\\dad\\My Documents\\somename")

ps=PrintStream
ps=mystream.append(stringdata,is,ie)
mystream.append(stringdata,is,ie)
mystream.append(CharSequence stringdata,is,ie)
mystream.append(CharSequence stringdata,rexxis,rexxie)
mystream.append(CharSequence stringdata,int rexxis,int rexxie)
mystream.append(rexxdata,is,ie)
mystream.append(CharSequence rexxdata,is,ie)
mystream.append(CharSequence rexxdata,rexxis,rexxie)
mystream.append(CharSequence rexxdata,int rexxis,int rexxie)
mystream.append(rexxdata,rexxis,rexxie)

mystream.print(ie)
mystream.print(rexxie)
   
    method xxx(lv=Rexx) static
        say lv
       
   
    method yyy(lv=CharSequence) static
        say lv
               
       
class newstream extends PrintStream
    method newstream()
        super("something")
   
    method xxx
       


On 10/8/2011 7:59 AM, Rony G. Flatscher wrote:

Hi Kermit,

On 07.10.2011 22:50, Kermit Kiser wrote:

When describing this proposed change, I asked for help testing it because of the possibility that existing programs could be affected, something I try hard to avoid where possible.

That is the reason why I try to help.

However you seem to prefer a philosophy discussion over providing actual test cases that show a problem. (Are you a professor of rhetoric?;-)

This seems to be a rhetorical question.
;)

That is OK with me if you don't mind me including some actual test code in the discussion. If nothing else, perhaps we can stimulate additional discussion and testing.

But first I would like to point out some items:

1) Mike has indicated that the method selection/costing algorithms were not carved in stone but were somewhat arbitrary and possibly not the best choice.
2) Alan has encountered an actual problem with running a real practical program due to the current algorithm.
3) I have encountered this problem many times but I will not include that code here because it is a proprietary Android application. To resolve this kind of run time problem I usually have to go back and generate the Java code.
4) I grant that we cannot know for sure if there would be problems reported with the new algorithm until after it is implemented.

Before I address your specific concerns, here is a code sample that I have been using to test this change:

-----------------------------------------------------------------------
s=String 11
i=int 22

test.me(s)
test.me(i)

class base
 method base()
 method me(is=String) static
 say "string="is
 
class test extends base
 method test()
 method me(ii=int) static
 say "int="ii
------------------------------------------------------------------------

Here is the output from compiling with 3.01RC2 and running it:

int=11
int=22

And here is the output from compiling with "after3.01" and running it:

string=11
int=22

As expected the changed algorithm makes both methods visible and the results therefore differ. If this were Java code, anyone looking at it would know that the programmer intended to add a new and separate method to the base class when he extended it because the new "me" method has a different signature. In the current NetRexx however, the new method with different signature does not work alongside of the old one, but overrides it even though the signature is different. Ignore for the moment the fact that anyone extending a class should know what it contains and what they can override. Then either the programmer did not know that he was overriding the original "me" method or in fact he did know it and did it deliberately. If the latter case, he could have accomplished that purpose more obviously by using the same signature for the new method just as a Java programmer would do. If done that way, the new algorithm makes no difference to the results. (Yes, I tested that.)

But what, if the programmer always wants to have test.me(int) picked for "s" and "i" in this NetRexx program?

And if he wanted base.me(String) to be used instead, why didn't he state so explicitly?

I submit to you that anyone overriding a method in this way (different signature) did not do so deliberately and did not intend it to happen!

In particular, in the case Alan submitted, the write methods in question were written in Java and so there is no way that the designer intended an override to happen, yet that is exactly what NetRexx has done, creating confusion to all of us! The current NetRexx method selection algorithm does NOT inter-operate compatibly with Java!

Indeed, the NetRexx language is *not* the Java language ! They have different language specifications. (NetRexx defines different principles compared to Java, trying to make programming easier than possible with the Java language itself, which is probably an important reason to use NetRexx over Java.)

As for the what the language manual says, we can and have made changes to it before. Right after this special note in the manual where Mike warns about the search problem:

"Note: When a method is found in a class, superclasses of that class are not searched for methods, even though a lower-cost method may exist in a superclass."

there is a section on Method overriding which describes the normal override procedure which is just like Java uses but there it is termed "exactly overriding", apparently because Mike realized even then that there was a possible problem with "inexact overriding".

You also mentioned the fact that the current algorithm has been in place for "15 years" as a reason that we are stuck with it. But to me that only says that there has been virtually zero maintenance and enhancement to NetRexx since it's creation, probably because IBM to this day does not realize that it is the most valuable thing they ever produced!

The purpose of this was to hint that it might be possible that quite a few NetRexx programs that have been developed in such a long time, *might* be affected by this change. It is about backward-compatibilty with such systems, that may break, if retranslated with the new behaviour (changing the method resolution mechanism changes the behaviiour).

Given that we need to clearly tell NetRexx programmers that if they compile old programs with a NetRexx translator "after3.01" the method resolution logic has changed, I contend that recompiling with the new algorithm is more likely to fix problems than to cause them!

That would be, short of available hard figures, pure speculation/expectations, but nothing that needs to materialize.
(If the new method resolution algorithms get implemented, then surely warnings would help to alert NetRexx coders that a new translation has changed the method NetRexx will use from then on. )

---

Maybe a few other points for the discussion:

• Studying the NetRexx specification and analyzing the Java code the translator creates and weighing what could be dubbed "Rexx semantics": the Rexx datatype in NetRexx seems to not be strictly typed for the NetRexx programmer: the permitted operations depend on the operand values to be adhering/produceable, which ultimately can only be done at runtime and not at compile time.
  
  
• The NetRexx translator currently decides the method it picks at compile time, and not at runtime!
  
  
• No matter what method resolution algorithms one puts in place, it can always be the case that the picked method cannot be used with the Rexx values at runtime, if their values are not compatible with the method's one. This has been demonstrated by Alan's program. If his program had used different string values it might have worked flawlessly, e.g. changing the String array to:
  
  
  
  players = [String -
      '  0 ' -
    , '1' -
  ]
  
  
• The NetRexx programmer probably would know what values to expect at runtime and hence knows which method should be used. If this method differs from the one that NetRexx picks, a problem at runtime may occur.
  
  
• The current NetRexx translator seems to assume, that the Rexx values it uses can always be used to create any of the types the picked method needs at runtime. Therefore, if there are candidate methods under this assumption, any of these candidate methods could be picked. It seems that using a cost-based algorithm serves two purposes then: one is to make the value coercions "cheap" (faster) and to make method picking reproducable ("stable"). This of course assumes too, that a NetRexx programmer would pick methods adhering to these pragmatics.
  
  
• These assumptions seem to break with Alan's program.
  
• However, if one is aware of how NetRexx currently picks a method (not being Java nor a Java compiler), then it is clear that NetRexx picked the "correct" method. 
• Alan could nevertheless force the NetRexx translator to pick the method, he needs for his particular program, knowing that the type must be a java.lang.String in order to get the method picked out of the superclass, instead of the local class. This is also the resolution of his problem.
  
  
• Following the NetRexx list for many years as a silent reader/lurker, I do not recall that the NetRexx method resolution was indeed surprising or even causing problems to NetRexx programmers. Therefore I would assume, that this observation, although at first sight "surprising", but after re-educating with the NetRexx method picking algorithm a very understandable choice, is not a real problem.
  
  
• What would be the behaviour of a NetRexx programmer who is creating a program and using the Java documentation?
• He would probably look up the definition of the BufferedWriter class, e.g. <http://download.oracle.com/javase/6/docs/api/java/io/BufferedWriter.html> and learn that the write(String) method is not implemented.
  
• Knowing about inheritance he would probably lookup the super class "Writer", e.g. <http://download.oracle.com/javase/6/docs/api/java/io/BufferedWriter.html>, where he finds the desired method.
  
• Knowing that NetRexx will use methods of the current class (which is a BufferWriter object in this case), if available, using coercion with the help of the Rexx methods to the desired target type (if those types are primitive or String), he knows that he has to tell NetRexx to use the method in the superclass instead. So he would simply do that.
  
  
• What would be the behaviour of a programmer who is creating a program and using existing Java programs, if available?
• He would probably transcribe a Java program 1:1 to NetRexx.
• He would be surprised, if the NetRexx version does not run without errors.
• So the question would be: whose fault is this? The programmer's or NetRexx?
  

So these thoughts probably boil down to something like:

• Is the current NetRexx method resolution algorithm difficult to understand and to memorize?
• Is the current NetRexx method resolution algorithm behaving in a surprisable way for NetRexx programmers?
  No, if the NetRexx method resolution is known: no, by no means! It is clear and simple.
• Is the current NetRexx method resolution algorithm behaving differently to Java and therefore surprising at times?
  Yes. But then, Java is a statically typed language where each type is known at compile time. (This is not the case with NetRexx Rexx types; they can be rendered to any primitive type and String type at runtime.)
• Is this a problem?
  Honestly, I do not know, but tend to think that it is a minor problem, if the NetRexx coders are aware of the NetRexx method resolution algorithms. (And they should, if they use NetRexx.)
• Are the new algorithms solving the problem: probably not. They solve the case in hand where a method in a superclass should be picked, if its arguments are strings (or so it seems). But what if indeed the programmer wanted a more costly method in the source class (if Alan had used numbers in his string, then he probably wanted to use the BufferedWriter's write(int) method).
• Are the new algorithms introducing new problems: probably, if a method in a superclass is picked instead, that the NetRexx programmer was not aware of that it existed.
  

HTH,

---rony

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Aviatrexx]

On 10/9/11 22:22 Kermit Kiser said:
> I see that you have shared an even longer philosophical discussion with
> no real world examples of programs where the new algorithm would cause a
> problem.

I don't think that anyone on this list doubts for a moment that Rony
could flood Kermit with the "real world examples" necessary to make
his points explicitly.  Nor do I think that Kermit believes that the
absence of such examples proves the superiority of his solution.

Kermit addressed a specific problem that Alan's example exhibited.
His suggested modification to the searching/costing algorithm solves
it in an elegant manner, which answers the specific question "Can we
fix this specific unexpected behavior?"

Rony raised "philosophical" questions about the new algorithm, among
which are:

   "What might be the unintentional consequences of the new algorithm?"

   "Does the raised problem need to be fixed this way?"

   "How much should NetRexx behavior mimic the Java language?"

and ultimately,

   "SHOULD the problem be fixed?"

His questions raise the meta-questions:

   "For whom is NetRexx designed?"

   "Which of the many competing NetRexx design characteristics should
be favored over others?

and fundamentally,

   "What is the PHILOSOPHY behind the design of NetRexx?"

Every non-trivial change to NetRexx should be subjected to the
scrutiny of such philosophical questions.

The solution that Kermit has proposed is not a trivial bugfix.

The problem it addresses could be solved by changing an internal
NetRexx algorithm, or by updating the documentation, or by some yet to
be proposed method.

The only way to determine the best solution for the NetRexx language
is to subject them all to a thoughtful evaluation on their
philosophical merits.

-Chip-

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[George Hovey-2]

I agree.  I'd like a quick solution, but as Rony and Chip point out there are a lot of ramifications.  Rony's points need to be taken seriously.
George

On Mon, Oct 10, 2011 at 12:36 PM, Chip Davis <[hidden email]> wrote:

On 10/9/11 22:22 Kermit Kiser said:

I see that you have shared an even longer philosophical discussion with no real world examples of programs where the new algorithm would cause a problem.

I don't think that anyone on this list doubts for a moment that Rony could flood Kermit with the "real world examples" necessary to make his points explicitly.  Nor do I think that Kermit believes that the absence of such examples proves the superiority of his solution.

Kermit addressed a specific problem that Alan's example exhibited. His suggested modification to the searching/costing algorithm solves it in an elegant manner, which answers the specific question "Can we fix this specific unexpected behavior?"

Rony raised "philosophical" questions about the new algorithm, among which are:

 "What might be the unintentional consequences of the new algorithm?"

 "Does the raised problem need to be fixed this way?"

 "How much should NetRexx behavior mimic the Java language?"

and ultimately,

 "SHOULD the problem be fixed?"

His questions raise the meta-questions:

 "For whom is NetRexx designed?"

 "Which of the many competing NetRexx design characteristics should be favored over others?

and fundamentally,

 "What is the PHILOSOPHY behind the design of NetRexx?"

Every non-trivial change to NetRexx should be subjected to the scrutiny of such philosophical questions.

The solution that Kermit has proposed is not a trivial bugfix.

The problem it addresses could be solved by changing an internal NetRexx algorithm, or by updating the documentation, or by some yet to be proposed method.

The only way to determine the best solution for the NetRexx language is to subject them all to a thoughtful evaluation on their philosophical merits.

-Chip-

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[KP Kirchdörfer]

Am Montag, 10. Oktober 2011, 18:36:28 schrieben Sie:

> On 10/9/11 22:22 Kermit Kiser said:

> > I see that you have shared an even longer philosophical discussion

> > with no real world examples of programs where the new algorithm would

> > cause a problem.

>

> I don't think that anyone on this list doubts for a moment that Rony

> could flood Kermit with the "real world examples" necessary to make

> his points explicitly. Nor do I think that Kermit believes that the

> absence of such examples proves the superiority of his solution.

>

> Kermit addressed a specific problem that Alan's example exhibited.

> His suggested modification to the searching/costing algorithm solves

> it in an elegant manner, which answers the specific question "Can we

> fix this specific unexpected behavior?"

>

> Rony raised "philosophical" questions about the new algorithm, among

> which are:

>

> "What might be the unintentional consequences of the new algorithm?"

>

> "Does the raised problem need to be fixed this way?"

>

> "How much should NetRexx behavior mimic the Java language?"

>

> and ultimately,

>

> "SHOULD the problem be fixed?"

>

> His questions raise the meta-questions:

>

> "For whom is NetRexx designed?"

>

> "Which of the many competing NetRexx design characteristics should

> be favored over others?

>

> and fundamentally,

>

> "What is the PHILOSOPHY behind the design of NetRexx?"

>

> Every non-trivial change to NetRexx should be subjected to the

> scrutiny of such philosophical questions.

>

> The solution that Kermit has proposed is not a trivial bugfix.

>

> The problem it addresses could be solved by changing an internal

> NetRexx algorithm, or by updating the documentation, or by some yet to

> be proposed method.

>

> The only way to determine the best solution for the NetRexx language

> is to subject them all to a thoughtful evaluation on their

> philosophical merits.

 

I'm a mere (Net)Rexx user and I ran into exactly the same pb Alan explained starting this discussion a few weeks ago.

I figured out, how to work around/fix while reading the javadoc for the BufferedReader class - but I was astonished, that it could be sometimes that easy to program in NetRexx (:= get rid of the java syntax/type conversion) and sometimes that hard, that reading javadoc is necessary.

 

I've tried to follow the discussion and I understand, that Kermits changes may break programs. But I'll give the changes a test ASAP, to verify it works at least with my small programs.

 

It seems to me, that Kermit tries to resolve a pb that I ran into from to time, the "least astonishment factor" - why does it work sometimes to just use type REXX (even it's an INT) and sometime it fails?

 

The more it can be resolved by the NetRexx compiler, the better IMHO.

If that's not possible, additional classes (like Max Marsglietties RXFile, though they probably too much tried to mimic original REXX behaviour) may help.

 

just my 2c

kp

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Mike Cowlishaw]

 

 

I agree.  I'd like a quick solution, but as Rony and Chip point out there are a lot of ramifications.  Rony's points need to be taken seriously.
 

As usual, I'll be boring and suggest that before anyone spends time publishing code they first publish the suggested changes to the documentation.  Given that, then others can then read those changes and explain any problems.

 

When everyone is happy with the documentation ... then implementation is easy, and one can concentrate on the implementation issues (performance, etc.). 

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Kermit Kiser]

My apologies!

I should have made it clear that I was teasing Rony - that is why I gave him the tongue-in-cheek credit for inspiring the algorithm changes. But I did very seriously consider everything he said several times.

Now that I have been through the RxClasser module three times, refining and optimizing the changes to the method selection logic, I am extremely confident that these changes improve NetRexx with virtually zero chance of having a negative impact on older programs. Although it would be hard to see this without careful study of the algorithms, the logic changes that I made do not affect most method selections done by NetRexx, but only the corner cases where NetRexx is trying to "coerce" a method call by finding a way to automatically convert the method arguments to match a signature somehow. The normal "zero-cost" method selections where the arguments correctly match a method signature just as they do in Java are not touched. Even the extra scans of superclasses and interfaces don't happen if a zero-cost match is found - the search just stops. In the coerced case, the changes expand the search to compare costs of local methods to superclasses and interfaces using Mike's original costing system which is very good at picking the best match I think. But even superclass or interface methods cannot get preference over a local method unless they have a LOWER cost and are thus more likely to be the correct choice.

As Rony pointed out, it is theoretically possible that the changes could alter the behavior in some older programs if they are recompiled with the updated compiler without being retested. In order for that to happen however, the original programmer would not only have to have used a method call for which there is no matching signature but he would have had to accept a method resolution different than a Java programmer would have expected. As we saw from Alan's case (and I found in my own encounters with that situation), the problems that causes would likely be detected during program testing and the call modified to accurately match a method signature. Therefore, I think it is unlikely and extremely rare if such cases exist at all. (BTW - In my previous email I was not alluding to any deficiency in Rony but to the rarity of real world programs that might have a problem.)

Although my code changes are trivial, the underlying logic is not trivial. On the other hand, the changes in behavior are trivial in the sense that they only affect a tiny percentage of NetRexx method resolutions. If there were a more significant number of method calls in this category, I think we would have long since seen a flood of complaints like Alan's report since the current algorithm causes definite problems in those cases with it's "un-Java" behavior. I suspect that most programmers who encounter those problems work around them by downcasting or something like that to achieve an exact method match just as Alan and I did. In that case, the algorithm change will not affect those programs.

I am confident enough with the modifications that I will soon begin testing them in my real world commercial applications with thousands of users. We have plenty of time before the next release to spot any problems if we are willing to test the proposed changes. Here is the location for the improved NetRexx translator binary:

http://kenai.com/projects/netrexx-plus/downloads/download/Experimental%20NetRexx%20Build/NetRexxC.jar

-- Kermit



On 10/10/2011 10:38 AM, George Hovey wrote:

I agree.  I'd like a quick solution, but as Rony and Chip point out there are a lot of ramifications.  Rony's points need to be taken seriously.
George

On Mon, Oct 10, 2011 at 12:36 PM, Chip Davis <[hidden email]> wrote:

On 10/9/11 22:22 Kermit Kiser said:

I see that you have shared an even longer philosophical discussion with no real world examples of programs where the new algorithm would cause a problem.

I don't think that anyone on this list doubts for a moment that Rony could flood Kermit with the "real world examples" necessary to make his points explicitly.  Nor do I think that Kermit believes that the absence of such examples proves the superiority of his solution.

Kermit addressed a specific problem that Alan's example exhibited. His suggested modification to the searching/costing algorithm solves it in an elegant manner, which answers the specific question "Can we fix this specific unexpected behavior?"

Rony raised "philosophical" questions about the new algorithm, among which are:

 "What might be the unintentional consequences of the new algorithm?"

 "Does the raised problem need to be fixed this way?"

 "How much should NetRexx behavior mimic the Java language?"

and ultimately,

 "SHOULD the problem be fixed?"

His questions raise the meta-questions:

 "For whom is NetRexx designed?"

 "Which of the many competing NetRexx design characteristics should be favored over others?

and fundamentally,

 "What is the PHILOSOPHY behind the design of NetRexx?"

Every non-trivial change to NetRexx should be subjected to the scrutiny of such philosophical questions.

The solution that Kermit has proposed is not a trivial bugfix.

The problem it addresses could be solved by changing an internal NetRexx algorithm, or by updating the documentation, or by some yet to be proposed method.

The only way to determine the best solution for the NetRexx language is to subject them all to a thoughtful evaluation on their philosophical merits.

-Chip-

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Aviatrexx]

On 10/10/11 19:32 Mike Cowlishaw said:
>     As usual, I'll be boring and suggest that before anyone spends time
>     publishing code they first publish the suggested changes to
>     the documentation.

Gee, where have I heard that before... ?

Must have been some long-ago Preacher, exhorting his flock to give up
their sinful ways and follow the path of enlightenment to truth and
redemption.  Poor fellow, nobody much listened to him either...  :-))

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Aviatrexx]

You are making a good case, Kermit.

Indeed, it was not clear to me that your response to Rony was in jest.
  Lacking any obvious smileys, there seemed to be a defensive edge to
it.  I will admit, however, to be somewhat overly sensitive to the
"That's  not a real-world case, it's just a {philosophical,
theoretical, conceptual, esoteric, etc., etc.} argument."

I would suggest that we try to identify the point in the Astonishment
Factor curve where one does the user a disservice by trying too hard
to "fix" a mistake.  Sometimes, the best solution is to simply present
a sensible error message, and let him find the error of his ways.

-Chip-

On 10/10/11 20:07 Kermit Kiser said:
>   My apologies!
>
> I should have made it clear that I was teasing Rony - that is why I gave
> him the tongue-in-cheek credit for inspiring the algorithm changes. But
> I did very seriously consider everything he said several times.

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Kermit Kiser]

Mike -

I see your logic but then I don't see your logic. By which I mean that it takes me back almost 40 years to my COBOL/FORTRAN internship when I was struggling to grasp the order to "complete the structured design and review before starting coding". But I lack the theoretical brilliance of folks like you, so now, as then, I often have to ponder some code and make some changes and tests to see what is really happening and what can actually be done. And to be honest, I often find that documentation is a weak and fuzzy approximation to real code behavior, especially in open source projects. NetRexx is quite good in that respect, compared to Android where the documentation isn't just fuzzy - it's often downright wrong!

Based on your comments here and in the code, I have the impression that 15 years ago compiler performance was an important issue that motivated some of your decisions such as the logic to skip checking superclasses and interfaces if a local method could be made to work. With modern computers being faster and more powerful by orders of magnitude would you make different choices now than back then?

-- Kermit

PS: Is it true that Europeans have to pay a fee for each touch-and-go?


On 10/10/2011 12:32 PM, Mike Cowlishaw wrote:

 

 

I agree.  I'd like a quick solution, but as Rony and Chip point out there are a lot of ramifications.  Rony's points need to be taken seriously.
 

As usual, I'll be boring and suggest that before anyone spends time publishing code they first publish the suggested changes to the documentation.  Given that, then others can then read those changes and explain any problems.

 

When everyone is happy with the documentation ... then implementation is easy, and one can concentrate on the implementation issues (performance, etc.). 

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Mike Cowlishaw]

 

I see your logic but then I don't see your logic. By which I mean that it takes me back almost 40 years to my COBOL/FORTRAN internship when I was struggling to grasp the order to "complete the structured design and review before starting coding". But I lack the theoretical brilliance of folks like you, so now, as then, I often have to ponder some code and make some changes and tests to see what is really happening and what can actually be done. And to be honest, I often find that documentation is a weak and fuzzy approximation to real code behavior, especially in open source projects.  

 

Experimentation to research a new design is fine, up to a point.  As for why, do you have a copy of the Rexx book?  If so, have a read through the section "Design principles" (pp 13&14 in the 2nd Ed.).  If not, let me know and I'll send you those 2 pages.

 

 NetRexx is quite good in that respect, compared to Android where the documentation isn't just fuzzy - it's often downright wrong!  

 

I believe only one or two errors have been found, to date. 

Based on your comments here and in the code, I have the impression that 15 years ago compiler performance was an important issue that motivated some of your decisions such as the logic to skip checking superclasses and interfaces if a local method could be made to work. With modern computers being faster and more powerful by orders of magnitude would you make different choices now than back then? 

 

Well, one never has enough performance, so it's always a concern (in my view, today's computers are less responsive and waste more of my time than old ones used to -- mostly due to software bloat and people being careless about performance).  Perhaps I wouldn't weight performance as heavily now as back then.

 

The risk factor in the proposed changes might be where one is starting with a number as type Rexx which you might expect to map to the foo(int) of the current class but instead goes to the foo(String) of a superclass (assuming that's cheaper -- I forget).

PS: Is it true that Europeans have to pay a fee for each touch-and-go? 

 

Depends on the airport/airfield.  My home airfield has no landing fees of any kind.  Coventry (a commercial airport) charges £15/landing and £8 for a touch-and-go.  In France, almost all small airfields are free.  In the UK, almost any 'manned' airfield usually charges a small fee or asks for a donation to a charity, etc.
 

Mike 

On 10/10/2011 12:32 PM, Mike Cowlishaw wrote:

 

 

I agree.  I'd like a quick solution, but as Rony and Chip point out there are a lot of ramifications.  Rony's points need to be taken seriously.
 

As usual, I'll be boring and suggest that before anyone spends time publishing code they first publish the suggested changes to the documentation.  Given that, then others can then read those changes and explain any problems.

 

When everyone is happy with the documentation ... then implementation is easy, and one can concentrate on the implementation issues (performance, etc.). 

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[rvjansen]

About the problem at hand: I am running with your fix since it is there
and I encounter no problems, but I yet have to dig up a large body of
code that I remember to suffer from method resolution problems here and
there.

I always check in every compiler level for a project into version
management. If more people would do that, we could just handle tool
changes over time in a more relaxed way. There seems to be something
with the time dimension that people do not want to accept.

Of course I subscribe to Mike's point of view, but in this case I
wonder if the method lookup was documented in a form that would make a
documentation change, and circulation of it in advance, feasible at all.
I think, from a project perspective, this is a question for the Project
Editor.

best regards,

René Jansen.



On Mon, 10 Oct 2011 16:08:11 -0700, Kermit Kiser wrote:

> Mike -
>
>  I see your logic but then I don't see your logic. By which I mean
> that it takes me back almost 40 years to my COBOL/FORTRAN internship
> when I was struggling to grasp the order to "complete the structured
> design and review before starting coding". But I lack the theoretical
> brilliance of folks like you, so now, as then, I often have to ponder
> some code and make some changes and tests to see what is really
> happening and what can actually be done. And to be honest, I often
> find that documentation is a weak and fuzzy approximation to real
> code
> behavior, especially in open source projects. NetRexx is quite good
> in
> that respect, compared to Android where the documentation isn't just
> fuzzy - it's often downright wrong!
>
>  Based on your comments here and in the code, I have the impression
> that 15 years ago compiler performance was an important issue that
> motivated some of your decisions such as the logic to skip checking
> superclasses and interfaces if a local method could be made to work.
> With modern computers being faster and more powerful by orders of
> magnitude would you make different choices now than back then?
>
>  -- Kermit
>
>  PS: Is it true that Europeans have to pay a fee for each
> touch-and-go?
>
>  On 10/10/2011 12:32 PM, Mike Cowlishaw wrote:
>
>>> I agree. I'd like a quick solution, but as Rony and Chip point
>> out
>>> there are a lot of ramifications. Rony's points need to be taken
>>> seriously.
>>>
>>> As usual, I'll be boring and suggest that before anyone spends
>>> time publishing code they first publish the suggested changes to
>>> the documentation. Given that, then others can then read those
>>> changes and explain any problems.
>>>
>>> When everyone is happy with the documentation ... then
>>> implementation is easy, and one can concentrate on the
>>> implementation issues (performance, etc.).
>>
>> _______________________________________________
>> Ibm-netrexx mailing list
>> [hidden email] [1]
>> Online Archive : http://ibm-netrexx.215625.n3.nabble.com/ [2]
>
>
> Links:
> ------
> [1] mailto:[hidden email]
> [2] http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Robert L Hamilton]

My Boss said write everything in Machine Code; That eliminates the documentation arguments.

Bob Hamilton

On Mon, Oct 10, 2011 at 4:52 PM, Chip Davis <[hidden email]> wrote:

On 10/10/11 19:32 Mike Cowlishaw said:

   As usual, I'll be boring and suggest that before anyone spends time
   publishing code they first publish the suggested changes to
   the documentation.

Gee, where have I heard that before... ?

Must have been some long-ago Preacher, exhorting his flock to give up their sinful ways and follow the path of enlightenment to truth and redemption.  Poor fellow, nobody much listened to him either...  :-))

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Rony G. Flatscher (wu-wien)]

On 11.10.2011 14:23, Robert Hamilton wrote:
> My Boss said write everything in Machine Code; That eliminates the
> documentation arguments.
Your boss is probably dead by now and the machine code programs got
replaced by Visual Basic ones, once the Y2K problem made your bosses
boss aware about the problems of missing documentation and trying to
solve it once and forever listening to what the media suggested all
along, hence demanding Windows and Visual Basic to the rescue ...
;)

---rony

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[ThSITC]

Hello Kermit, Rony,

1.) I must admit I'm using very simply NetRexx (actually nealy always NetRexx1.0 compliant)
2.) I would like to report, that I did download the latest BUILD from Kermit's
NetRexx-PLUS project and did compile all of my soft without any problems.

Great work, Kermit! Thanks a lot :-)

Thomas.
=======================================================
Am 10.10.2011 00:22, schrieb Kermit Kiser:

Rony -

I see that you have shared an even longer philosophical discussion with no real world examples of programs where the new algorithm would cause a problem. Sadly, I do not have time to respond in kind. However, I promise you that I have read your message twice and considered it deeply although I don't really see much in the way of new concerns there. I will try to share my view briefly.

First, in the example that I offered it seems very clear to me that by extending the base class without deliberately overriding the "me" method that accepts strings, the programmer has indeed stated explicitly that he wants to use that inherited method where possible. That is what inheritance is about.

Second, most of your discussion seems to center on a hypothetical NetRexx programmer who wants to do things the traditional "NetRexx" way even though it is not really compatible with the Java way. I have only one response to that discussion:

NetRexx was designed AND advertised as an easy and efficient way to create Java compatible programs.

I am quite sure that most if not all NetRexx programmers are creating applications that inter-operate with Java systems and they want the maximum compatibility with Java that is possible.

However the possibility of breaking existing programs is always an important concern even when trying to improve NetRexx compatibility with Java. I have compiled over 200 NetRexx programs (all of the Rosettacode examples plus my own test case programs) with the new algorithms without a problem. Of course this change affects runtime results and I was not able to run all of those programs due to time limitations. That is why any example of a real program where the new algorithms affected something would be valuable! In the case of Alan's real world example, the new algorithm fixes a runtime problem. There is no known example of a real program being negatively affected.

Since it is not possible to go back and test every NetRexx program ever written with the new algorithm, the best possible course to improve Java compatibility is to do as much testing as time constraints allow and then to notify programmers who download the new changed version of NetRexx that there is a remote possibility that some old programs might behave differently if compiled with the new version so they should either do a complete retest of any older application they modify and recompile or simply use NetRexx 3.01 or earlier for maintenance of older programs.

Thanks to your inspiration to research this issue more deeply, as well as my confidence that we can improve Java compatibility without hurting existing applications, I have discovered another change (now on the after3.01 branch at rev 156) to the sacred cow method selection/costing algorithms that allows NetRexx to compile the original uncompilable source example (see below) that I submitted on Kenai with issue 11. This change allows programmers to call interface methods that have been implemented by a class, even while using NetRexx automatic type conversions. That was not possible before the change because both the interface method and the implemented method obviously have the same signature and hence the same cost which led to the "ambiguous method" message: "More than one method matches..."

The latest binary version of the NetRexx translator including all method selection changes can be found on the NetRexx Plus project at this URL:

http://kenai.com/projects/netrexx-plus/downloads

-- Kermit

PS: Here is the test code that will not compile with 3.01 or earlier because NetRexx could not handle the CharSequence interfaces Java 1.5+ uses -

say "test charseq conversion issues"

stringdata=String "xyz"
csq=CharSequence stringdata
xxx(csq)
rexxdata=Rexx "xyz"
yyy(rexxdata)
yyy(stringdata)
is=int 0
rexxis=Rexx 0
ie=int 1
rexxie=Rexx 1

mystream=PrintStream("\\Documents and Settings\\dad\\My Documents\\somename")

ps=PrintStream
ps=mystream.append(stringdata,is,ie)
mystream.append(stringdata,is,ie)
mystream.append(CharSequence stringdata,is,ie)
mystream.append(CharSequence stringdata,rexxis,rexxie)
mystream.append(CharSequence stringdata,int rexxis,int rexxie)
mystream.append(rexxdata,is,ie)
mystream.append(CharSequence rexxdata,is,ie)
mystream.append(CharSequence rexxdata,rexxis,rexxie)
mystream.append(CharSequence rexxdata,int rexxis,int rexxie)
mystream.append(rexxdata,rexxis,rexxie)

mystream.print(ie)
mystream.print(rexxie)
   
    method xxx(lv=Rexx) static
        say lv
       
   
    method yyy(lv=CharSequence) static
        say lv
               
       
class newstream extends PrintStream
    method newstream()
        super("something")
   
    method xxx
       


On 10/8/2011 7:59 AM, Rony G. Flatscher wrote:

Hi Kermit,

On 07.10.2011 22:50, Kermit Kiser wrote:

When describing this proposed change, I asked for help testing it because of the possibility that existing programs could be affected, something I try hard to avoid where possible.

That is the reason why I try to help.

However you seem to prefer a philosophy discussion over providing actual test cases that show a problem. (Are you a professor of rhetoric?;-)

This seems to be a rhetorical question.
;)

That is OK with me if you don't mind me including some actual test code in the discussion. If nothing else, perhaps we can stimulate additional discussion and testing.

But first I would like to point out some items:

1) Mike has indicated that the method selection/costing algorithms were not carved in stone but were somewhat arbitrary and possibly not the best choice.
2) Alan has encountered an actual problem with running a real practical program due to the current algorithm.
3) I have encountered this problem many times but I will not include that code here because it is a proprietary Android application. To resolve this kind of run time problem I usually have to go back and generate the Java code.
4) I grant that we cannot know for sure if there would be problems reported with the new algorithm until after it is implemented.

Before I address your specific concerns, here is a code sample that I have been using to test this change:

-----------------------------------------------------------------------
s=String 11
i=int 22

test.me(s)
test.me(i)

class base
 method base()
 method me(is=String) static
 say "string="is
 
class test extends base
 method test()
 method me(ii=int) static
 say "int="ii
------------------------------------------------------------------------

Here is the output from compiling with 3.01RC2 and running it:

int=11
int=22

And here is the output from compiling with "after3.01" and running it:

string=11
int=22

As expected the changed algorithm makes both methods visible and the results therefore differ. If this were Java code, anyone looking at it would know that the programmer intended to add a new and separate method to the base class when he extended it because the new "me" method has a different signature. In the current NetRexx however, the new method with different signature does not work alongside of the old one, but overrides it even though the signature is different. Ignore for the moment the fact that anyone extending a class should know what it contains and what they can override. Then either the programmer did not know that he was overriding the original "me" method or in fact he did know it and did it deliberately. If the latter case, he could have accomplished that purpose more obviously by using the same signature for the new method just as a Java programmer would do. If done that way, the new algorithm makes no difference to the results. (Yes, I tested that.)

But what, if the programmer always wants to have test.me(int) picked for "s" and "i" in this NetRexx program?

And if he wanted base.me(String) to be used instead, why didn't he state so explicitly?

I submit to you that anyone overriding a method in this way (different signature) did not do so deliberately and did not intend it to happen!

In particular, in the case Alan submitted, the write methods in question were written in Java and so there is no way that the designer intended an override to happen, yet that is exactly what NetRexx has done, creating confusion to all of us! The current NetRexx method selection algorithm does NOT inter-operate compatibly with Java!

Indeed, the NetRexx language is *not* the Java language ! They have different language specifications. (NetRexx defines different principles compared to Java, trying to make programming easier than possible with the Java language itself, which is probably an important reason to use NetRexx over Java.)

As for the what the language manual says, we can and have made changes to it before. Right after this special note in the manual where Mike warns about the search problem:

"Note: When a method is found in a class, superclasses of that class are not searched for methods, even though a lower-cost method may exist in a superclass."

there is a section on Method overriding which describes the normal override procedure which is just like Java uses but there it is termed "exactly overriding", apparently because Mike realized even then that there was a possible problem with "inexact overriding".

You also mentioned the fact that the current algorithm has been in place for "15 years" as a reason that we are stuck with it. But to me that only says that there has been virtually zero maintenance and enhancement to NetRexx since it's creation, probably because IBM to this day does not realize that it is the most valuable thing they ever produced!

The purpose of this was to hint that it might be possible that quite a few NetRexx programs that have been developed in such a long time, *might* be affected by this change. It is about backward-compatibilty with such systems, that may break, if retranslated with the new behaviour (changing the method resolution mechanism changes the behaviiour).

Given that we need to clearly tell NetRexx programmers that if they compile old programs with a NetRexx translator "after3.01" the method resolution logic has changed, I contend that recompiling with the new algorithm is more likely to fix problems than to cause them!

That would be, short of available hard figures, pure speculation/expectations, but nothing that needs to materialize.
(If the new method resolution algorithms get implemented, then surely warnings would help to alert NetRexx coders that a new translation has changed the method NetRexx will use from then on. )

---

Maybe a few other points for the discussion:

• Studying the NetRexx specification and analyzing the Java code the translator creates and weighing what could be dubbed "Rexx semantics": the Rexx datatype in NetRexx seems to not be strictly typed for the NetRexx programmer: the permitted operations depend on the operand values to be adhering/produceable, which ultimately can only be done at runtime and not at compile time.
  
  
• The NetRexx translator currently decides the method it picks at compile time, and not at runtime!
  
  
• No matter what method resolution algorithms one puts in place, it can always be the case that the picked method cannot be used with the Rexx values at runtime, if their values are not compatible with the method's one. This has been demonstrated by Alan's program. If his program had used different string values it might have worked flawlessly, e.g. changing the String array to:
  
  
  
  players = [String -
      '  0 ' -
    , '1' -
  ]
  
  
• The NetRexx programmer probably would know what values to expect at runtime and hence knows which method should be used. If this method differs from the one that NetRexx picks, a problem at runtime may occur.
  
  
• The current NetRexx translator seems to assume, that the Rexx values it uses can always be used to create any of the types the picked method needs at runtime. Therefore, if there are candidate methods under this assumption, any of these candidate methods could be picked. It seems that using a cost-based algorithm serves two purposes then: one is to make the value coercions "cheap" (faster) and to make method picking reproducable ("stable"). This of course assumes too, that a NetRexx programmer would pick methods adhering to these pragmatics.
  
  
• These assumptions seem to break with Alan's program.
  
• However, if one is aware of how NetRexx currently picks a method (not being Java nor a Java compiler), then it is clear that NetRexx picked the "correct" method. 
• Alan could nevertheless force the NetRexx translator to pick the method, he needs for his particular program, knowing that the type must be a java.lang.String in order to get the method picked out of the superclass, instead of the local class. This is also the resolution of his problem.
  
  
• Following the NetRexx list for many years as a silent reader/lurker, I do not recall that the NetRexx method resolution was indeed surprising or even causing problems to NetRexx programmers. Therefore I would assume, that this observation, although at first sight "surprising", but after re-educating with the NetRexx method picking algorithm a very understandable choice, is not a real problem.
  
  
• What would be the behaviour of a NetRexx programmer who is creating a program and using the Java documentation?
• He would probably look up the definition of the BufferedWriter class, e.g. <http://download.oracle.com/javase/6/docs/api/java/io/BufferedWriter.html> and learn that the write(String) method is not implemented.
  
• Knowing about inheritance he would probably lookup the super class "Writer", e.g. <http://download.oracle.com/javase/6/docs/api/java/io/BufferedWriter.html>, where he finds the desired method.
  
• Knowing that NetRexx will use methods of the current class (which is a BufferWriter object in this case), if available, using coercion with the help of the Rexx methods to the desired target type (if those types are primitive or String), he knows that he has to tell NetRexx to use the method in the superclass instead. So he would simply do that.
  
  
• What would be the behaviour of a programmer who is creating a program and using existing Java programs, if available?
• He would probably transcribe a Java program 1:1 to NetRexx.
• He would be surprised, if the NetRexx version does not run without errors.
• So the question would be: whose fault is this? The programmer's or NetRexx?
  

So these thoughts probably boil down to something like:

• Is the current NetRexx method resolution algorithm difficult to understand and to memorize?
• Is the current NetRexx method resolution algorithm behaving in a surprisable way for NetRexx programmers?
  No, if the NetRexx method resolution is known: no, by no means! It is clear and simple.
• Is the current NetRexx method resolution algorithm behaving differently to Java and therefore surprising at times?
  Yes. But then, Java is a statically typed language where each type is known at compile time. (This is not the case with NetRexx Rexx types; they can be rendered to any primitive type and String type at runtime.)
• Is this a problem?
  Honestly, I do not know, but tend to think that it is a minor problem, if the NetRexx coders are aware of the NetRexx method resolution algorithms. (And they should, if they use NetRexx.)
• Are the new algorithms solving the problem: probably not. They solve the case in hand where a method in a superclass should be picked, if its arguments are strings (or so it seems). But what if indeed the programmer wanted a more costly method in the source class (if Alan had used numbers in his string, then he probably wanted to use the BufferedWriter's write(int) method).
• Are the new algorithms introducing new problems: probably, if a method in a superclass is picked instead, that the NetRexx programmer was not aware of that it existed.
  

HTH,

---rony

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Kermit Kiser]

Mike - thanks for suggesting a very informative test case which may help people see what is happening!

Although it seems a little unnatural to create a NetRexx method that accepts a Java type object (for local access - I do it a lot for external callbacks from Java code), one could hypothesize a case where the method is called both from Java code and NetRexx code. So here is my hypothetical code with a String method in a superclass that matches an "int" method in a subclass along with the results it achieved:

----------------------- code: -------------------------------------
Subguy.main()

Class Superguy
    method Superguy()
   
    method mymethod(s=String) static
        say "processing String="s
       
Class Subguy extends Superguy
    method Subguy()
   
    method mymethod(i=int) static
        say "processing int="i
       
    method main static
        s=String "77"
        i=int 88
        r=Rexx "99"
        mymethod(s)
        mymethod(i)
        mymethod(r)
        Superguy.mymethod(s)
        Superguy.mymethod(i)
        Superguy.mymethod(r)

----- run time output using 3.01 compiler: --------------
   
processing int=77
processing int=88
processing int=99
processing String=77
processing String=88
processing String=99

----------------------------------------------------------------

You can see the very "un Java-like" behavior above where the superclass method is not extended by the subclass method but completely replaced so that the only way to reach it is to restrict the method search to the superclass by downcasting or specific qualification, etc.

----- run time output using after3.01 compiler: --------------  (I removed the uninteresting qualified calls.)

processing String=77
processing int=88
processing String=99

----------------------------------------------------------------------

From the above output after compiling with the "after3.01" compiler, the change in behavior is clear: An int goes to the int method while strings go to the String method. It is also clear that Mike correctly guessed that a Rexx to String conversion costs less than a Rexx to int conversion.

Now things get more interesting! Add the following code to the end of the program:

        sd=String "string data"
        mymethod(sd)

        rd=Rexx "rexx data"
        mymethod(rd)

----------------- This is from the 3.01 compiler output: ----------------

[C:\NetRexx\testing\methtest1.nrx 28 3 8] Error: Cannot convert constant value 'string data' to type 'int'
[C:\NetRexx\testing\methtest1.nrx 30 3 8] Error: Cannot convert constant value 'rexx data' to type 'int'
Compilation of 'methtest1.nrx' failed [3 classes, 2 errors]

Even though the program has a String method available to process these strings, NetRexx 3.01 cannot see it! I was surprised!

On the other hand, NetRexx "after3.01" compiles and runs that program correctly with no problem:

----- run time output using after3.01 compiler: --------------

processing String=77
processing int=88
processing String=99
processing String=string data
processing String=rexx data
------------------------------------------------------------------------------

Now I changed that last added bit of code to bypass the compile time error:

        sd=String "string data"
        snan=sd
        mymethod(snan)

        rd=Rexx "rexx data"
        rnan=rd
        mymethod(rnan)

NetRexx 3.01 now compiles the program but gives the following runtime output:

---- run time output after 3.01 compile:  ----------------

processing int=77
processing int=88
processing int=99
Exception in thread "main" java.lang.NumberFormatException: string data
    at netrexx.lang.Rexx.toint(Rexx.nrx:627)
    at Subguy.main(methtest1.nrx:32)
    at methtest1.main(methtest1.nrx:1)
----------------------------------------------------------------------

Once again, NetRexx "after3.01" compiles and runs the program without error:

----- run time output using after3.01 compiler: --------------

processing String=77
processing int=88
processing String=99
processing String=string data
processing String=rexx data
-----------------------------------------------------------------------------

This test sequence does not provide much new information but it may clarify the effect of the changes for some people. I think it also shows that the changed "after3.01" compiler is much more likely to compile and run programs as most programmers would expect.

BTW, I have two copies of the book "The NetRexx Language" next to my computer and probably have a copy of the Rexx book buried somewhere nearby also.

Also I have not touched any documentation because I think René was correct to require an SVN lock on the manuals to reduce conflicts in updates since we cannot easily identify changes at this time. So I have been waiting to update the documents until 3.01 is final release to prevent difficult doc merges from alternate branches.

-- Kermit


On 10/11/2011 12:31 AM, Mike Cowlishaw wrote:

 

I see your logic but then I don't see your logic. By which I mean that it takes me back almost 40 years to my COBOL/FORTRAN internship when I was struggling to grasp the order to "complete the structured design and review before starting coding". But I lack the theoretical brilliance of folks like you, so now, as then, I often have to ponder some code and make some changes and tests to see what is really happening and what can actually be done. And to be honest, I often find that documentation is a weak and fuzzy approximation to real code behavior, especially in open source projects.  

 

Experimentation to research a new design is fine, up to a point.  As for why, do you have a copy of the Rexx book?  If so, have a read through the section "Design principles" (pp 13&14 in the 2nd Ed.).  If not, let me know and I'll send you those 2 pages.

 

 NetRexx is quite good in that respect, compared to Android where the documentation isn't just fuzzy - it's often downright wrong!  

 

I believe only one or two errors have been found, to date. 

Based on your comments here and in the code, I have the impression that 15 years ago compiler performance was an important issue that motivated some of your decisions such as the logic to skip checking superclasses and interfaces if a local method could be made to work. With modern computers being faster and more powerful by orders of magnitude would you make different choices now than back then? 

 

Well, one never has enough performance, so it's always a concern (in my view, today's computers are less responsive and waste more of my time than old ones used to -- mostly due to software bloat and people being careless about performance).  Perhaps I wouldn't weight performance as heavily now as back then.

 

The risk factor in the proposed changes might be where one is starting with a number as type Rexx which you might expect to map to the foo(int) of the current class but instead goes to the foo(String) of a superclass (assuming that's cheaper -- I forget).

PS: Is it true that Europeans have to pay a fee for each touch-and-go? 

 

Depends on the airport/airfield.  My home airfield has no landing fees of any kind.  Coventry (a commercial airport) charges £15/landing and £8 for a touch-and-go.  In France, almost all small airfields are free.  In the UK, almost any 'manned' airfield usually charges a small fee or asks for a donation to a charity, etc.
 

Mike 

On 10/10/2011 12:32 PM, Mike Cowlishaw wrote:

 

 

I agree.  I'd like a quick solution, but as Rony and Chip point out there are a lot of ramifications.  Rony's points need to be taken seriously.
 

As usual, I'll be boring and suggest that before anyone spends time publishing code they first publish the suggested changes to the documentation.  Given that, then others can then read those changes and explain any problems.

 

When everyone is happy with the documentation ... then implementation is easy, and one can concentrate on the implementation issues (performance, etc.). 

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/