Oracle
Application Testing Suite (OATS): Few Tips & Tricks
OATS is a suite of applications that can be used for performance and scalability testing, functional and regression testing. It is a thin client application that runs within a web browser - so, it is easy to use the tool from anywhere as long as the web server running on the host node is accessible. Hopefully the following tips and tricks will benefit some of the users of the Oracle Application Testing Suite.
OATS is a suite of applications that can be used for performance and scalability testing, functional and regression testing. It is a thin client application that runs within a web browser - so, it is easy to use the tool from anywhere as long as the web server running on the host node is accessible. Hopefully the following tips and tricks will benefit some of the users of the Oracle Application Testing Suite.
Few technical details first - OATS
is a 32-bit Java application that runs in a WebLogic container (WLS) with
Oracle XE database being the backend store for test session data.
[Trick] Issue : OATS software fails to install on 64-bit Windows
systems
Resolution:
Download and install 64-bit .NET framework manually before installing the OATS software. Look for .NET framework on Microsoft's downloads website.
Download and install 64-bit .NET framework manually before installing the OATS software. Look for .NET framework on Microsoft's downloads website.
[Trick] Issue : OATS software fails to install on systems with
large number of [virtual] CPUs
Resolution:
On systems with many cores/vCPUs, Oracle database in general requires large amounts of memory to be configured for SGA - so, one solution would be to allocate as much memory as required. However Oracle XE limits the memory utilization within the database to 1 GB. Besides, XE uses only one CPU even if there are multiple CPUs available on a system. Hence one workaround is to limit the number of vCPUs that the system exposes during the installation of OATS software. The steps are shown below.
On systems with many cores/vCPUs, Oracle database in general requires large amounts of memory to be configured for SGA - so, one solution would be to allocate as much memory as required. However Oracle XE limits the memory utilization within the database to 1 GB. Besides, XE uses only one CPU even if there are multiple CPUs available on a system. Hence one workaround is to limit the number of vCPUs that the system exposes during the installation of OATS software. The steps are shown below.
- Start button -> Run -> type "msconfig"
- Click on Boot tab -> Advanced Options
- Check "Number of processors" and set appropriate value (I believe we can go up to 16)
- Reboot Windows
- Uninstall failed OATS installation and try installing again
- Undo the above made changes after the successful installation of OATS
- Reboot Windows one final time
Thanks to my colleague Bao Doan for
providing this workaround.
[Trick] Issue : During runtime, OATS drive the load and executes
the test as expected but fails to collect runtime statistics
Resolution:
This is another limitation of Oracle XE database. Until 10g, XE limits the maximum amount of user data in the database to 4 GB. This limit was raised to 11 GB in release Oracle 11g XE. OATS 9.x releases bundle Oracle 10g XE. To take advantage of the larger limit for data, install Oracle 11g XE manually before installing OATS software. OATS installer gives the option to use an existing installation of Oracle XE. Besides, it is not possible to have multiple Oracle XE installations on a single box anyway (that's another XE limitation).
This is another limitation of Oracle XE database. Until 10g, XE limits the maximum amount of user data in the database to 4 GB. This limit was raised to 11 GB in release Oracle 11g XE. OATS 9.x releases bundle Oracle 10g XE. To take advantage of the larger limit for data, install Oracle 11g XE manually before installing OATS software. OATS installer gives the option to use an existing installation of Oracle XE. Besides, it is not possible to have multiple Oracle XE installations on a single box anyway (that's another XE limitation).
For existing installations, one
workaround is to remove old and unwanted sessions to make room for new sessions
in the database. Listed below are the steps.
- Connect to the Oracle Load Testing (OLT) tool
- Click on "Manage" top-level menu (upper right corner) -> Sessions
- Click on any unwanted session and press "Delete" button (I recommend deleting one session at a time)
[Trick] Issue : Under load, there are many network timeouts with
ton of sockets in TIME_WAIT state on OATS agent systems including the OATS
Controller node
Resolution:
Tune TCP/IP parameters on Windows as shown below.
Tune TCP/IP parameters on Windows as shown below.
- Launch Windows registry
- Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TcpIP\parameters
- Configure the following two parameters. If not found, create those parameters by selecting Edit -> New -> DWORD Value from the menu bar. Select "Decimal" under Base.
TcpTimedWaitDelay
: 30 [seconds]
MaxUserPort : 65534
MaxUserPort : 65534
- Reboot Windows
Thanks to my colleagues Dino and
Vishnu for sharing this workaround.
[Trick] Issue : OATS Controller does not show any graphs or
analysis reports
Resolution:
Install Adobe Flash Plugin and try again.
Install Adobe Flash Plugin and try again.
[Trick] Issue : Under load, OATS Controller stops collecting
runtime statistics at some random point
Resolution:
Check Oracle database alert log for some clue(s). If there is an error message such as "ORA-12516: TNS:listener could not find available handler with matching protocol stack", connect to the database, query v$resource_limit view and compare the values reported under CURRENT_UTILIZATION and MAX_UTILIZATION for the resource "processes". If the current utilization is pretty close to the configured maximum value, raise the value for processes parameter in [S]PFILE.
Check Oracle database alert log for some clue(s). If there is an error message such as "ORA-12516: TNS:listener could not find available handler with matching protocol stack", connect to the database, query v$resource_limit view and compare the values reported under CURRENT_UTILIZATION and MAX_UTILIZATION for the resource "processes". If the current utilization is pretty close to the configured maximum value, raise the value for processes parameter in [S]PFILE.
[Tip] Balancing the load among multiple OATS agent systems
One simple way is to create a VU
Agent System Group based on the available agent systems. Steps listed below.
- Connect to the Oracle Load Testing (OLT) tool
- Click on "Manage" top-level menu (upper right corner) -> Systems
- Click on "VU Agent System Group" in the left hand side
- On the right hand side, click on "New" option
- Select all the agent systems that you want to be part of the "VU Agent System Group"
- Finally name the newly created system group and save
Note that it is not possible to
attach weights to the agent systems - so, it is suggested to have agent systems
with similar hardware configurations in the VU Agent System Group.
[Tip] Balancing the load among multiple web servers using OATS
Controller
If there are multiple web server
instances running in a enterprise application deployment; and OATS software is
being used to test the performance and scalability of the application,
parameterizing the web server hostname and port number in OATS test script will
take care of the web server load balancing problem. Of course there are many
alternatives to this approach such as using a hardware load balancer, using web
server Reverse Proxy etc.,
[Added on 01/19/2012]
[Tip] How-To check the available space in USERS tablespace?
Run the following on OATS Controller
node:
Start -> All Programs ->
Oracle Database XX Express Edition -> Run SQL Command Line
SQL>
connect / as sysdba
SQL>
SELECT /* + RULE */ df.tablespace_name
"Tablespace",
df.bytes / (1024 * 1024) "Size
(MB)",
SUM(fs.bytes) / (1024 * 1024) "Free
(MB)",
Nvl(Round(SUM(fs.bytes) * 100 /
df.bytes),1) "% Free",
Round((df.bytes - SUM(fs.bytes)) * 100 /
df.bytes) "% Used"
FROM dba_free_space fs,
(SELECT tablespace_name,SUM(bytes) bytes
FROM dba_data_files
GROUP BY tablespace_name) df
WHERE fs.tablespace_name (+) = df.tablespace_name
GROUP BY df.tablespace_name,df.bytes
UNION
ALL
SELECT
/* + RULE */ df.tablespace_name tspace,
fs.bytes / (1024 * 1024),
SUM(df.bytes_free) / (1024 * 1024),
Nvl(Round((SUM(fs.bytes) -
df.bytes_used) * 100 / fs.bytes), 1),
Round((SUM(fs.bytes) - df.bytes_free) *
100 / fs.bytes)
FROM dba_temp_files fs,
(SELECT
tablespace_name,bytes_free,bytes_used
FROM v$temp_space_header
GROUP BY
tablespace_name,bytes_free,bytes_used) df
WHERE fs.tablespace_name (+) = df.tablespace_name
GROUP BY
df.tablespace_name,fs.bytes,df.bytes_free,df.bytes_used
ORDER BY 4 DESC;
Copy/paste the above SQL code in a
text file with sql extension and execute that SQL statement by calling the SQL
script from SQL> command prompt. eg., assuming the above code was saved in a
plain text file called chktblspcusg.sql under C:\ drive, execute the SQL script
as shown below:
SQL> @C:\chktblspcusg.sql
[Added on 06/27/2012]
[Trick] Issue : An attempt to open a test script in OpenScript
fails with error
'Failed
to open script' has encountered a problem.
Failed
to open <script_name>. See error log for details.
Clicking on "Details"
button provides the following clue.
The project description file (.project) for
'<script_name>' is missing"
In addition the title bar shows
"Relocating Eclipse Projects: The
project description file (.project) for XXX is missing".
Resolution:
Navigate to C:\Documents and Settings\Administrator\osworkspace\.metadata\.plugins\org.eclipse.core.resources\.projects\
Look for the directory by name "<failing_script_name>" and remove it
Navigate to C:\Documents and Settings\Administrator\osworkspace\.metadata\.plugins\org.eclipse.core.resources\.projects\
Look for the directory by name "<failing_script_name>" and remove it
[Added on 08/03/2012]
[Trick] Issue: Unexpected Agent exit.
Code = 51 in the middle of an OLT load test
When running a load scenario in
Oracle Load Testing (OLT) that uses a databank, the scenario runs fine for some
time and then all of a sudden fails with the following error: Unexpected
Agent exit. Code = 51.
Workaround:
The following settings may alleviate
the issue.
- - toggle/experiment with the settings for "Clear cache between iterations" and "Clear cache before playing back"
- those settings can be found under the test script preferences -> Playback -> Web Functional -> Miscellaneous
- - experiment with different values for "Maximum users per process" setting
- this setting is under OLT -> Configure all parameters -> Advanced
- - increase the Java heap size (both min & max) in file <OATS_HOME>\agentmanager\bin\AgentManagerService.conf
- default values: min heap size: 16 MB; max heap size: 64 MB
Contributors: John Snyder, Richard
Barry
[Added 02/25/13]
Another colleague Dave Suri has an
alternate tip to resolve the Agent 51 issue.
Edit
<OATS_HOME>\agentmanager\processDescriptors\JavaAgent.properties
Change the following lines:
#process.debug=y
#process.debug.suspend=y
#process.debug.port=8123
#process.debug.custom=
To:
process.debug=y
#process.debug.suspend=y
#process.debug.port=8123
process.debug.custom=-verbose:gc
-XX:+HeapDumpOnOutOfMemoryError -Xms512M
-Xmx1536M
-jrockit -Xrs -XgcPrio:deterministic -XpauseTarget=50ms
-XX:+UseCallProfiling
-XX:+UseAdaptiveFatSpin -XX:+ExitOnOutOfMemoryError
-XXnoSystemGC
-XX:+UseFastTime
PS:
I'm no OATS expert - merely documented what we found along with solutions and workarounds. Feel free to post your question(s) in the comments section. Hopefully someone will respond.
No comments:
Post a Comment