Stall Array
Tymeac Asynchronous Request processing is a background function. There is no timing for the Tymeac Function. Therefore, when something goes wrong, Tymeac uses an alternate method to recover. This section discusses that alternative.
The Stall Array is both an in-doubt as well as a hard stall mechanism for Asynchronous Requests.
The first in-doubt situation arises during initial scheduling.
The Tymeac Server adds an entry to this array during back-out processing, when in-doubt.
Back-out occurs when a Queue, within a Function, fails scheduling. The Tymeac Server informs all previously scheduled Queue's to ignore processing. If the request is in a Wait List, then Tymeac removes it. In the small window between the time the application class finishes processing but before the Queue Thread can update the status, the Tymeac Server is in doubt whether it can safely release resources. The Tymeac Server places an entry in the Stall Array.
The Monitor Thread cleans-up such an entry when the Queue Thread updates the status.
An example is helpful.
Tymeac Function, "Function_6", in the Demonstration System uses two Queues, 'AAAA' and 'BBBB'.
When there is a request for "Function_6":
The user-written Tymeac Client application forms the parameters needed by the Tymeac Server and does an asyncRequest. When all Tymeac Queues successfully schedule, the Tymeac Server returns to the user-written client application. The application may inform the requester that the request scheduled and free the connection for other work.
However, the following may occur:
The Tymeac Server determines that the Function consists of two Tymeac Queues. The Tymeac Server schedules Queue 'AAAA'; scheduling complete. The Tymeac Server schedules Queue 'BBBB'; scheduling fails (see Scheduler Return Codes).
This Function cannot complete successfully. A back-out of all previously scheduled Queues is desirable either to avoid unnecessary processing when the request is in a Queue's Wait List pending execution, or, to limit the amount of post-application processing done by the Queue Thread.
In this example, Queue 'AAAA' is the only previously scheduled Queue in need of back-out processing. The Tymeac Server attempts to back-out Queue 'AAAA' by removing the Wait List entry.
When it finds the entry and removes it, the back-out is successful. When a successful back-out occurs, the Tymeac Server frees all associated resources.
When an unsuccessful back-out occurs, the Tymeac Server cannot free the request's resources without the possibility of jeopardizing system integrity. That is, a Queue Thread may be reading the input Object or updating internal tables. The stage of processing for the request is unknown.
This may simply be a case where one Thread is locking a resource and another Thread is waiting on that lock before it updates the resource. This is only a problem in that the Tymeac Server cannot free valuable system resources at this time. Therefore, the Tymeac Server adds an entry to the Stall Array.
When the Queue Thread for Queue 'AAAA' finishes processing the request, it determines that no further Queues remain for the request, and the request is 'in error'. Therefore, the Queue Thread frees the request's resources.
The entry in the Stall Array remains. However, no resources for the request remain. When the Monitor Thread runs, it deletes the Stall Array Entry.
The Second in-doubt situation arises during the Monitor Thread scan of Tymeac resources.
The Monitor Thread places an entry in the Stall Array when in-doubt.
Tymeac is a process now system. The Monitor Thread examines the Tymeac System for entries remaining for a lengthy period and adds an entry to the Stall Array when "this has been here much too long". An example of a request remaining for a lengthy period is a situation caused by an loop in an Processing Application Class or an operating system purge outside of this image.
When Tymeac flags the Asynchronous Request due to lengthy processing time in applications (that is, the Request is not hanging, it is just taking a long time) then this Stall Array Entry should be ignored. The Monitor Thread eventually cleans-up this entry.
The Third situation arises during the scheduling of the Output Agent Queue. This is not an in-doubt situation but a hard stall.
When Tymeac flags an Asynchronous Request due to a failure to schedule the Output Agent Queue, then recovery and re-scheduling are possible.
The Output Agent Queue is the Tymeac Queue that processes the combined output of all the previously executed asynchronous component processes of the Function.
The last Queue Thread to finish processing schedules the Output Agent Queue. However, when scheduling fails, the Asynchronous Request stalls. The Queue Thread puts the stalled request into the Stall Array and writes messages to the Log and Notification facility.
After the system administrator resolves the problem with the Output Agent Queue, the Stalled Request Display (TyStalled) 'ReSchd OA' button, attempts to re-schedule the Output Agent Queue. This may finish the request.
The Forth situation arises when a Processing Application Class abnormally terminates. This is a hard stall.
Through decades of experience we know there is no such thing as a secure computer system. Therefore, when an application abnormally terminates (that is: it does not catch an exception or re-throws an exception) the potential for system damage is great.
Tymeac's response to an exception in a Processing Application Class is to flag the Queue Thread (in which it executes) as "disabled", terminate the [physical] Queue Thread, Log and print the error, and, add an entry to the Stall Array.
It is the designers responsibility to handle all exceptions in Processing Application Classes. In a multi-threading application it is often difficult to pin point the cause of the crash without a record. This is like an airplane's "black box" after a crash. If there is no Tymeac Log in use and no Tymeac error printing (console window) then user logging is vital. Tymeac cannot diagnose application errors.
After the system administrator resolves the problem with the Processing Application Class, the Queue Thread Class (TyQueThd) 'enable all' button instantiates and, when there are pending requests in the Wait Lists, activates a new Queue Thread for the Queue.
The Fifth situation arises when an AsyncRequest is cancelled. This is an in-doubt situation.
Prior to release 5.0 we used a course grained locking mechanism for controlling access to Tymeac resources. Basically, we "locked" the environment when we added a new request to the system as well as when we cancelled a request. This way we were sure of what parts of the request were never run, running and finished. Therefore, we could be sure when we cancelled a request that no Queue Threads needed resources and those resources could be freed.
As of release 5.0 the environment is no longer "locked" during cancel processing. When Tymeac can remove every part of the request before any Queue Threads begin processing, then Tymeac may safely free the resources. Otherwise, Tymeac is once again "in doubt" whether it can free the resources so the AsyncRequest resources are left alone. The Monitor may remove these resources after the last Queue Thread finishes but it may also add a stall array entry if any doubt remains.