NetRexx Timing(?) Problems

[Jeff Hennick-3]

In testing the new sql stage for NetRexx Pipelines, I am again getting random errors from NetRexx.

Here are two sequential runs of the same test pipelines.  I did nothing between them, just pressed the up arrow (to get the previous command) and enter. In the first run, all 5 tests pass: my code and the SQLite code is good.  In the second run, errors occur in tests 1, 2, & 4.  There are error messages from the stage code for 2 of them, and error messages (multiple lines beginning with "#") from my tests for the output from the sql stage. In other runs, these 3 tests will be OK and others will fail.  Totally random from what I can make out.

PS C:\Users\Jeff\documents\pipe tests> pipc sql PS C:\Users\Jeff\documents\pipe tests> java sql_tests01 #1 sql from command line options *OK* #2 sql from secondary *OK* #3 sql from default file options *OK* #4 sql from named file options *OK* #5 sql headers *OK* PS C:\Users\Jeff\documents\pipe tests> java sql_tests01 eSQLException(s) caught ! org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (table person already exists) Message: [SQLITE_ERROR] SQL error or missing database (table person already exists) Cause: null Vendor error code: 1 eSQLException(s) caught ! org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (table person already exists) Message: [SQLITE_ERROR] SQL error or missing database (table person already exists) Cause: null Vendor error code: 1 #1 sql from command line options **FAIL** at rec 5 col 1. More: Actual: 3 Expected: 1 #2 sql from secondary **FAIL** at rec 5 col 1. More: Actual: 2 Expected: 1 #3 sql from default file options *OK* #4 sql from named file options **FAIL** at rec 5 col 1. More: Actual: 2 Expected: 1 #5 sql headers *OK* PS C:\Users\Jeff\documents\pipe tests>

My temporary work around will be to not put the five tests in a single pipe file, but to run them in independent test files.  I have never had the problem when doing that [I have had my own bugs!].

These tests are in this form:

 pipe (ct)
 o: faninany |
    sort singleok |
    specs 2-* 1 |
    cons ?


-- from command line options ?
   literal ,
     drop table if exists person; ,
     create table person (id integer, name string); ,
     insert into person values(1, 'leo-rexx'); ,
     insert into person values(2, 'yui-rexx'); ,
     update person set id = 3 where name = 'yui-rexx'; ,
     select * from person |
   split ; |
   sql ( dbms sqlite db_name flightroute-iata.sqb ) |
   c1: compare any ~2#1 sql from command line options ~ equal ~*OK*~ notequal ~**FAIL**  at rec \\c col \\b.~ less /Less:\nActual:\n\\p\nExpected:\n\\s\n/ more /More:\nActual:\n\\p\nExpected:\n\\s\n/ |
   o: ?
   literal 0;0;1;1;1;1   leo-rexx;3   yui-rexx | -- expected output from sql |
   split ; |
   c1: ?

Any help, or pointers to what I can do to alleviate the problem would be appreciated.

Jeff

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[rvjansen]

Jeff,

with the words ‘totally random’ my first guess is that this too must be a threading issue. Do you have the methods and variables properly synchronized (‘protect’)?

On 15 Aug 2020, at 16:57, Jeff Hennick <[hidden email]> wrote:

In testing the new sql stage for NetRexx Pipelines, I am again getting random errors from NetRexx.

Here are two sequential runs of the same test pipelines.  I did nothing between them, just pressed the up arrow (to get the previous command) and enter. In the first run, all 5 tests pass: my code and the SQLite code is good.  In the second run, errors occur in tests 1, 2, & 4.  There are error messages from the stage code for 2 of them, and error messages (multiple lines beginning with "#") from my tests for the output from the sql stage. In other runs, these 3 tests will be OK and others will fail.  Totally random from what I can make out.

PS C:\Users\Jeff\documents\pipe tests> pipc sql PS C:\Users\Jeff\documents\pipe tests> java sql_tests01 #1 sql from command line options *OK* #2 sql from secondary *OK* #3 sql from default file options *OK* #4 sql from named file options *OK* #5 sql headers *OK* PS C:\Users\Jeff\documents\pipe tests> java sql_tests01 eSQLException(s) caught ! org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (table person already exists) Message: [SQLITE_ERROR] SQL error or missing database (table person already exists) Cause: null Vendor error code: 1 eSQLException(s) caught ! org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (table person already exists) Message: [SQLITE_ERROR] SQL error or missing database (table person already exists) Cause: null Vendor error code: 1 #1 sql from command line options **FAIL** at rec 5 col 1. More: Actual: 3 Expected: 1 #2 sql from secondary **FAIL** at rec 5 col 1. More: Actual: 2 Expected: 1 #3 sql from default file options *OK* #4 sql from named file options **FAIL** at rec 5 col 1. More: Actual: 2 Expected: 1 #5 sql headers *OK* PS C:\Users\Jeff\documents\pipe tests>

My temporary work around will be to not put the five tests in a single pipe file, but to run them in independent test files.  I have never had the problem when doing that [I have had my own bugs!].

These tests are in this form:

 pipe (ct)
 o: faninany |
    sort singleok |
    specs 2-* 1 |
    cons ?


-- from command line options ?
   literal ,
     drop table if exists person; ,
     create table person (id integer, name string); ,
     insert into person values(1, 'leo-rexx'); ,
     insert into person values(2, 'yui-rexx'); ,
     update person set id = 3 where name = 'yui-rexx'; ,
     select * from person |
   split ; |
   sql ( dbms sqlite db_name flightroute-iata.sqb ) |
   c1: compare any ~2#1 sql from command line options ~ equal ~*OK*~ notequal ~**FAIL**  at rec \\c col \\b.~ less /Less:\nActual:\n\\p\nExpected:\n\\s\n/ more /More:\nActual:\n\\p\nExpected:\n\\s\n/ |
   o: ?
   literal 0;0;1;1;1;1   leo-rexx;3   yui-rexx | -- expected output from sql |
   split ; |
   c1: ?

Any help, or pointers to what I can do to alleviate the problem would be appreciated.

Jeff

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Jeff Hennick-3]

Thank you.  No, I have never used protect in NetRexx.  And after reading the Language Reference, I'm not sure what it does, but I'll try putting onto the run method as a magic word, and see what happens.  This is all a learning experience for me, the Language Reference definitely is not a tutorial, and I can't find one of the proper depth to get to this option.  Do you know of one I have missed? 

On 8/15/2020 11:50 AM, René Jansen wrote:

Jeff,

with the words ‘totally random’ my first guess is that this too must be a threading issue. Do you have the methods and variables properly synchronized (‘protect’)?

On 15 Aug 2020, at 16:57, Jeff Hennick <[hidden email]> wrote:

In testing the new sql stage for NetRexx Pipelines, I am again getting random errors from NetRexx.

Here are two sequential runs of the same test pipelines.  I did nothing between them, just pressed the up arrow (to get the previous command) and enter. In the first run, all 5 tests pass: my code and the SQLite code is good.  In the second run, errors occur in tests 1, 2, & 4.  There are error messages from the stage code for 2 of them, and error messages (multiple lines beginning with "#") from my tests for the output from the sql stage. In other runs, these 3 tests will be OK and others will fail.  Totally random from what I can make out.

PS C:\Users\Jeff\documents\pipe tests> pipc sql PS C:\Users\Jeff\documents\pipe tests> java sql_tests01 #1 sql from command line options *OK* #2 sql from secondary *OK* #3 sql from default file options *OK* #4 sql from named file options *OK* #5 sql headers *OK* PS C:\Users\Jeff\documents\pipe tests> java sql_tests01 eSQLException(s) caught ! org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (table person already exists) Message: [SQLITE_ERROR] SQL error or missing database (table person already exists) Cause: null Vendor error code: 1 eSQLException(s) caught ! org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (table person already exists) Message: [SQLITE_ERROR] SQL error or missing database (table person already exists) Cause: null Vendor error code: 1 #1 sql from command line options **FAIL** at rec 5 col 1. More: Actual: 3 Expected: 1 #2 sql from secondary **FAIL** at rec 5 col 1. More: Actual: 2 Expected: 1 #3 sql from default file options *OK* #4 sql from named file options **FAIL** at rec 5 col 1. More: Actual: 2 Expected: 1 #5 sql headers *OK* PS C:\Users\Jeff\documents\pipe tests>

My temporary work around will be to not put the five tests in a single pipe file, but to run them in independent test files.  I have never had the problem when doing that [I have had my own bugs!].

These tests are in this form:

 pipe (ct)
 o: faninany |
    sort singleok |
    specs 2-* 1 |
    cons ?


-- from command line options ?
   literal ,
     drop table if exists person; ,
     create table person (id integer, name string); ,
     insert into person values(1, 'leo-rexx'); ,
     insert into person values(2, 'yui-rexx'); ,
     update person set id = 3 where name = 'yui-rexx'; ,
     select * from person |
   split ; |
   sql ( dbms sqlite db_name flightroute-iata.sqb ) |
   c1: compare any ~2#1 sql from command line options ~ equal ~*OK*~ notequal ~**FAIL**  at rec \\c col \\b.~ less /Less:\nActual:\n\\p\nExpected:\n\\s\n/ more /More:\nActual:\n\\p\nExpected:\n\\s\n/ |
   o: ?
   literal 0;0;1;1;1;1   leo-rexx;3   yui-rexx | -- expected output from sql |
   split ; |
   c1: ?

Any help, or pointers to what I can do to alleviate the problem would be appreciated.

Jeff

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Jeff Hennick-3]

With method run() protect it ran great, once.  Then it gave 6 (yes 6 out of 5 tests) SQLExceptions and need a ctrl-C to stop.  Not so great.

On 8/15/2020 11:50 AM, René Jansen wrote:

Jeff,

with the words ‘totally random’ my first guess is that this too must be a threading issue. Do you have the methods and variables properly synchronized (‘protect’)?

On 15 Aug 2020, at 16:57, Jeff Hennick <[hidden email]> wrote:

In testing the new sql stage for NetRexx Pipelines, I am again getting random errors from NetRexx.

Here are two sequential runs of the same test pipelines.  I did nothing between them, just pressed the up arrow (to get the previous command) and enter. In the first run, all 5 tests pass: my code and the SQLite code is good.  In the second run, errors occur in tests 1, 2, & 4.  There are error messages from the stage code for 2 of them, and error messages (multiple lines beginning with "#") from my tests for the output from the sql stage. In other runs, these 3 tests will be OK and others will fail.  Totally random from what I can make out.

PS C:\Users\Jeff\documents\pipe tests> pipc sql PS C:\Users\Jeff\documents\pipe tests> java sql_tests01 #1 sql from command line options *OK* #2 sql from secondary *OK* #3 sql from default file options *OK* #4 sql from named file options *OK* #5 sql headers *OK* PS C:\Users\Jeff\documents\pipe tests> java sql_tests01 eSQLException(s) caught ! org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (table person already exists) Message: [SQLITE_ERROR] SQL error or missing database (table person already exists) Cause: null Vendor error code: 1 eSQLException(s) caught ! org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (table person already exists) Message: [SQLITE_ERROR] SQL error or missing database (table person already exists) Cause: null Vendor error code: 1 #1 sql from command line options **FAIL** at rec 5 col 1. More: Actual: 3 Expected: 1 #2 sql from secondary **FAIL** at rec 5 col 1. More: Actual: 2 Expected: 1 #3 sql from default file options *OK* #4 sql from named file options **FAIL** at rec 5 col 1. More: Actual: 2 Expected: 1 #5 sql headers *OK* PS C:\Users\Jeff\documents\pipe tests>

My temporary work around will be to not put the five tests in a single pipe file, but to run them in independent test files.  I have never had the problem when doing that [I have had my own bugs!].

These tests are in this form:

 pipe (ct)
 o: faninany |
    sort singleok |
    specs 2-* 1 |
    cons ?


-- from command line options ?
   literal ,
     drop table if exists person; ,
     create table person (id integer, name string); ,
     insert into person values(1, 'leo-rexx'); ,
     insert into person values(2, 'yui-rexx'); ,
     update person set id = 3 where name = 'yui-rexx'; ,
     select * from person |
   split ; |
   sql ( dbms sqlite db_name flightroute-iata.sqb ) |
   c1: compare any ~2#1 sql from command line options ~ equal ~*OK*~ notequal ~**FAIL**  at rec \\c col \\b.~ less /Less:\nActual:\n\\p\nExpected:\n\\s\n/ more /More:\nActual:\n\\p\nExpected:\n\\s\n/ |
   o: ?
   literal 0;0;1;1;1;1   leo-rexx;3   yui-rexx | -- expected output from sql |
   split ; |
   c1: ?

Any help, or pointers to what I can do to alleviate the problem would be appreciated.

Jeff

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[rvjansen]

That is good news - the first part. The second part must be solved.

Do you have the exceptions from the jdbc driver?

René.

On 15 Aug 2020, at 18:33, Jeff Hennick <[hidden email]> wrote:

With method run() protect it ran great, once.  Then it gave 6 (yes 6 out of 5 tests) SQLExceptions and need a ctrl-C to stop.  Not so great.

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Jeff Hennick-3]

I think this is from, or through, the driver:

eSQLException(s) caught ! org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (table person already exists) Message: [SQLITE_ERROR] SQL error or missing database (table person already exists) Statement: create table person (id integer, name string) Vendor error code: 1

That comes from this block of code:

     key = query.word(1).lower
    select case key
      when 'create' then res = Rexx stmt.execute(query)
      when 'insert', 'update', 'delete' then res = Rexx stmt.executeUpdate(query)
      when 'select' then rs = stmt.executeQuery(query)
      otherwise
       res = Rexx stmt.executeUpdate(query)

   catch e3 = SQLException
     say 'eSQLException(s) caught !' e3
     loop while (e3 \= null)
       if e3.getSQLState() \= null then say 'SQLState:' e3.getSQLState()
       if e3.getMessage() \= null then say 'Message: ' e3.getMessage()
       say 'Statement: ' query
       say 'Vendor error code:  ' e3.getErrorCode()
       say
       e3 = e3.getNextException()
     end

   end

On 8/15/2020 2:33 PM, René Jansen wrote:

That is good news - the first part. The second part must be solved.

Do you have the exceptions from the jdbc driver?

René.

On 15 Aug 2020, at 18:33, Jeff Hennick <[hidden email]> wrote:

With method run() protect it ran great, once.  Then it gave 6 (yes 6 out of 5 tests) SQLExceptions and need a ctrl-C to stop.  Not so great.

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Jeff Hennick-3]

After some out of list communication with ColinK, (thank you) it occurs to me that the problem is likely a result of NetRexx Pipelines multi-threading nature and the structure of my tests.

Each test starts with these SQL statements:

     drop table if exists person; ,
     create table person (id integer, name string); ,

One test running by itself is no problem.  But when I have a half dozen or more tests running in parallel hitting the same database,  no wonder that at least one of them finds that after it has dropped the table, the next time it looks, there is the table!  I'm driving the system mad!

Note: SQLite is designed to be hit multiply nearly in sync with normal inserts, updates, and selects.  It is not designed for some guy multiply dropping and creating the same table that others are using.

My bad.

I apologize for wasting everybody's time on this.

Jeff

On 8/15/2020 2:33 PM, René Jansen wrote:

That is good news - the first part. The second part must be solved.

Do you have the exceptions from the jdbc driver?

René.

On 15 Aug 2020, at 18:33, Jeff Hennick <[hidden email]> wrote:

With method run() protect it ran great, once.  Then it gave 6 (yes 6 out of 5 tests) SQLExceptions and need a ctrl-C to stop.  Not so great.

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

[Jeff Hennick-3]

I changed my tests such that each one of the five in the suite is now dropping and creating is own named table, and dropping it again at the end of the test.

This has almost solved it.  Running the test suite 16 times in succession, only the first and the last reported any problem, and that was on a single test each time, rather than the multiples earlier.

I guess the moral is "Don't abuse your database."

Again, thanks to all.

Jeff

On 8/15/2020 4:56 PM, Jeff Hennick wrote:

After some out of list communication with ColinK, (thank you) it occurs to me that the problem is likely a result of NetRexx Pipelines multi-threading nature and the structure of my tests.

Each test starts with these SQL statements:

     drop table if exists person; ,
     create table person (id integer, name string); ,

One test running by itself is no problem.  But when I have a half dozen or more tests running in parallel hitting the same database,  no wonder that at least one of them finds that after it has dropped the table, the next time it looks, there is the table!  I'm driving the system mad!

Note: SQLite is designed to be hit multiply nearly in sync with normal inserts, updates, and selects.  It is not designed for some guy multiply dropping and creating the same table that others are using.

My bad.

I apologize for wasting everybody's time on this.

Jeff

On 8/15/2020 2:33 PM, René Jansen wrote:

That is good news - the first part. The second part must be solved.

Do you have the exceptions from the jdbc driver?

René.

On 15 Aug 2020, at 18:33, Jeff Hennick <[hidden email]> wrote:

With method run() protect it ran great, once.  Then it gave 6 (yes 6 out of 5 tests) SQLExceptions and need a ctrl-C to stop.  Not so great.

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/

_______________________________________________
Ibm-netrexx mailing list
[hidden email]
Online Archive : http://ibm-netrexx.215625.n3.nabble.com/