We are executing CICS TS 4.2 with transaction rate of 4 to 5 million a day.
We usually get sub-second response time.
There are about 4,000 MQ transactions each day in the mix with sub-second response time
We are about to implement a project with many more MQ transactions.
We currently execute about 700 of one of the new transactions in production each day QIN2.
While we execute up to 150 transactions per second, up to 16 of the QIN2 are completed in the 150.
In production we are using 1 input queue which initiates 1 MQ task MQIP, to retrieve the data from the queue followed by the application QIN2
We are testing an environment where we are using 4 queues which will initiate the MQIP task for each queue followed by the QIN2.
The 4 queues were instituted to balance the the number of calls to the system.
We stress test the calls in the test system with all 4 queues and got sub-second response time.
We stress test the calls in the production environment using the 4 queues while executing 150 normal transactions per second.
we entered 850 MQ calls to the system at the rate of 4 calls per second.
About 10% of the QIN2 transactions experience excessive response times,from 1 second to 10 seconds.
After the test was completed we began experiencing slow response times for our normal transactions up to 30 minutes after the test.
We have 2 problems.
What is causing the slow response times for 10% of the QIN2 transaction
What is the cause of the residual effect on the system where normal transactions are executing at 1 through 10 second response times.
Did anyone experience such anomaly before or have any ideas what we can try?
This QIN2 transaction ran while we were were processing 140 trans per sec using the 1 queue
which is run every day with mix of normal transactions.
The response time is .0386 sec and similar for the 700 QIN2s executed at that time
This is a TMON detail stat of a QIN2 at that time.
ID CPU TCB SWCT TCB SWTM WRD CNT WRD TIME TOT CNT TOT TIME
* * * * * * * *
L8 0.0019 3 0.0003 0 0.0000 3 0.0039
QR 0.0110 3 0.0040 0 0.0000 10 0.0123
This is one of the QIN2 that was part of the 10% with response time over 1.0 seconds.
Response time was 10.7393 seconds. There were 850 QIN2 transactions entered at 2 per sec
and the response time began increasing over 1.0 second after approximately 500 transactions.
The system was executing over 110 transactions per sec at that time.
The wait times were attributed to File I/O but there is no reason for it.
The other questions raised.
Is there actual VSAM I-O being performed (mostly PUTS)? How about DB2 and/or IMS PUT's?
There is no DB2 and no IMS
There is 1 VSAM put on an ESDS file for logging
In Production, are your transid's competing with higher priority transid's?
You are correct about the priority. It was left unnoticed.
QIN2 is running at 100 whereas all other transactions are at 228. We will change it to 228
The MQIP which reads the queue is set at 250.
The priority does not affect the QIN2s that run every day in production.
Ask your CICS Sysprog if your transid's are defined to a specific TRANCLASS, reducing the number of active transid's which can execute concurrently.
All the Transactions run in the same class. there is no limit.
You can get some idea of the bottleneck with a TRACE and review the API's and times.
We will be running some traces the next few days when there is a window to do the stress test again.