Back | Next | Contents Cams Administrator's Guide

Performance Tuning

Because Cams is an integration product, server hardware, network design, network load and configuration settings can all effect performance. This document discusses some ways that you can maximize Cams policy server performance.

Java Virtual Machine Switches

Java Virtual Machines (JVMs) offer a variety of switches that tune memory allocation and garbage collection behavior. Because the Cams policy server runs within a Java JVM, this section discusses settings that can effect Cams policy server performance.

JVM garbage collection

Garbage collection refers to how the Java Virtual Machine (JVM) removes unused program objects to free memory. JVM memory is managed in separate areas:

  1. New (young) object space: used for newly created objects
  2. Old (tenured) object space: used for objects that have been around a while
  3. Permanent space: used for permanent JVM class and method objects

Object-oriented programs such as the Cams policy server generally create many new objects with very short lifetimes, so most objects are created in the new object heap (also called the young generation heap). Objects that are longer-lived are known as survivors and eventually copied from the new object head to the old object heap (also called the tenured generation heap). Permanent objects are created in a special portion of the tenured generation heap.

Different algorithms can be chosen for garbage collection, which can be specific to a heap. When the JVM does garbage collection on young objects, this is called a minor garbage collection. Garbage collection occurs in each generation when the generation fills up. Some surviving objects are moved to a tenured generation. When the tenured generation needs to be garbage collected there is a major collection that is often much slower because it involves all live objects.

On systems that have multiple CPUs, garbage collection can be parallelized to eliminate stop-the-world pause times (where Cams policy server effectively stops until garbage collection completes). With parallelization, the Cams policy server can continue to run on CPUs that are not garbage collecting. This can even benefit single processor systems, more so with the advent of CPU features such as hyperthreading and dual core processors.

On systems with one CPU, stop-the-world pause times can be minimized by setting incremental garbage collection. Long pause times are reduced in exchange for somewhat longer minor garbage collection times.

How to specify JVM switches

Java switches are passed on the command line that starts the JVM. Since the Cams policy server is started differently on Windows than it is on Linux/UNIX, please note the following.

Windows

Name/value pairs found in the CAMS_HOME/conf/runcams.conf file pass settings to the JVM. Most values are passed as wrapper.additional.n values, where n represents the nth consecutive value.

wrapper.java.additional.7=-Xverify:none

Some name/value pair options are already explicitly defined, such as the initial and maximum heap memory settings:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=64

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=128

Linux/UNIX

The CAMS_HOME/bin/runcams.sh script sets all switches and invokes the JVM and the Cams policy server. You can edit this file to change existing values or supply new name/value pairs. For example, you'll see that the setting for initial and maximum memory are already defined using the default names of -Xms and -Xmx. To set the -Xverify:none switch:

${JAVA_HOME}/bin/java -server -Xms64m -Xmx128m -Xverify:none -classpath ${CP} ${P} com.cafesoft.security.provider.CamsPolicyServer

NOTE: The switches in this document are available on Java SE 1.4, 1.5 and 1.6 JVMs. The -X and especially -XX JVM switches are often JVM-vendor specific. The Sun JVM does not start when passed switches it doesn't understand. Instead, it will return a message pointing out the problem. For example, if you use the switch -foo:

Unrecognized VM option '-foo'
Could not create the Java virtual machine.

Default Cams policy server switches

The following default switches are set for the Cams policy server and should produce satisfactory performance on most systems:

  • -server - Optimizes the JVM for server-based applications like the Cams policy server.
  • -Xms64m - Instructs the JVM to set its initial heap size to 64 megabytes. By telling the JVM how much memory it should initially allocate for the heap, you save growing the heap as the Cams policy server consumes more memory. This switch also improves startup time.
  • -Xmx128m - Instructs the JVM of the maximum amount of memory it should use for the heap. Placing a hard upper limit means that the Cams policy server cannot consume more memory than the available physical memory. This limit can be raised on systems with more memory - the 128 megabyte setting shown here helps to ensure that the Cams policy server will perform well on 256MB systems.
    NOTE: Do not set this value near or greater than the amount of physical memory in your system or it will cause severe swapping during major garbage collections.

Other switches

These switches are not set by default but may provide performance benefits without any harmful side-effects.

  • -Xverify:none - Turns off Java bytecode verification, making class loading faster and eliminating the need for classes to be loaded during startup solely for the purposes of verification. This switch improves startup time.
  • -XX:PermSize=32m - Sizes the JVM permanent area, where permanent JVM class and method objects are kept. Since Cams policy server classes will take up a specific amount of memory, use this value to hint to the JVM how much memory it will need. This setting eliminates major garbage collection events during startup and improves startup time.
  • -XX:CompileThreshold=100 - Forces the JVM HotSpot just-in-time compiler to compile more methods to native code sooner than it otherwise would. The result is snappier performance once the Cams policy server is running, since more of the code will be compiled rather than interpreted. This value represents the number of times a method must be called before it will be compiled. Setting this value will slow Cams policy server startup time.

Garbage collectors

For most Cams policy server installations, the default JVM switch settings work fine. If performance is reasonable, you don't need to try any of the switches in this section. As the saying goes, "If it ain't broke, don't fix it."

WARNING: If you decide to try some of the following switches, you should ALWAYS benchmark performance before and after making any changes so you can accurately measure your results.

If you have a multiprocessor system and are experiencing pause times in the Cams policy server, try the throughput parallel garbage collector:

  • -XX:+UseParallelGC - Uses multiple threads to execute a minor young generation garbage collection. On systems fairly well equipped with memory the duration of minor garbage collections is halved when using this collection algorithm, even on uniprocessor systems. This is somewhat paradoxical as this collector is designed to work best on multiprocessor systems.
    NOTE: Do not use this garbage collector with -XX:+UseConcMarkSweepGC switch, they are mutually exclusive.

If you have a single-processor system and are experiencing pause times in the Cams policy server, try the incremental garbage collector:

  • -Xincgc - Used to trade longer and more frequent young generation minor garbage collection pauses for shorter tenured generation major garbage collection pauses. This switch is most useful on single processor systems that are memory constrained. This might help with perceived performance, even though it may also decrease overall Cams policy server performance.

If the throughput or incremental garbage collectors don't help (or don't help enough) and you have lots of memory (1GB or more), then try the concurrent garbage collector:

  • -XX:+UseConcMarkSweepGC and -XX:+UseParNewGC - Uses parallel garbage collection for both young and tenured objects resulting in shorter garbage collector pauses, especially on multiprocessor systems.
    NOTE: If you are using the PermSize switch, you should also use -XX:+CMSClassUnloadingEnabled and -XX:+CMSPermGenSweepingEnabled so that class unloading is enabled (it isn't by default when using this collector).
  • -XX:+CMSParallelRemarkEnabled - reduces remark pauses

For example, on multiple CPU systems try:

Windows

wrapper.java.additional.7=-XX:+UseParNewGC
wrapper.java.additional.8=-XX:+UseConcMarkSweepGC
wrapper.java.additional.9=-XX:+CMSParallelRemarkEnabled

Linux/UNIX

${JAVA_HOME}/bin/java -server -Xms64m -Xmx128m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -classpath ${CP} ${P} com.cafesoft.security.provider.CamsPolicyServer

Additional Resources

There are many other switches you can use to optimize a JVM. The following resources are helpful to further your understanding of JVM performance tuning:

Enterprise Server Hardware

The Cams policy server runs on virtually any system with an operating system that supports a compliant 1.4.x JVM or greater. Because it is lightweight, the Cams policy server has a minimal footprint and requirements. However, Cams policy server performance will increase when installed on more powerful systems, which is important for enterprise deployments. This section provides performance considerations when selecting servers that will be used with the Cams policy server in the enterprise.

NOTE: The Cams policy server runs without any special configuration requirements one virtual machines such as VMWare, Virtual PC and VirtualBox.

Server processor(s)

You should consider using two or more processors or cores so that JVM concurrent garbage collection can be enabled. You must ensure that the operating system and JVM to be used support multiprocessors effectively. For example, the Sun JVM supports Solaris, Linux and Windows Server deployed on many but not all multi-processor systems. Make sure you check with the vendor of the server to ensure that the JVM you will use provides multiprocessor support for your system.

NOTE: Intel P4 processors with Hyperthreading support provide a significant performance boost when running a multithreaded Java server such as the Cams policy server. Additionally, the multi-core Intel and AMD chips effectively offer the same garbage collection efficiencies within a single CPU.

Memory

1GB memory is recommended (though 512MB is generally sufficient). Because memory is relatively inexpensive you should avoid any chance of swapping to disk as this will dramatically reduce performance! You'll need to change the default Cams JVM startup settings to take advantage of the additional memory. For example, for 1GB:

-Xms256m
-Xmx512m

NOTE: It is safe practice to set the -Xmx maximum heap JVM switch to no more than 50 percent of the physical memory on the system. That leaves plenty of memory for system level resources. You should NEVER set it to the same size as the physical memory on the system.

Disks

The Cams policy server is loaded into and runs memory resident. However, the hard disk is used for writing Cams policy server logs. Because the Cams policy server logging service is threaded, hard disk I/O performance will not impact the performance of Cams web agent requests to the Cams policy server.

You should consider how you'll manage the Cams policy server logs. The access control log grows the fastest. By default, it is rotated every 10MB in size (you can configure this in the security domains security-domain.xml file). 10MB represents approximately 85,000 requests. The same 10MB file compressed is about 700KB in size. You should calculate your log growth based on your request expectations (also consider that most access checks for unconditionally granted resources will be cached by the Cams web agent and not logged by the Cams policy for each HTTP request). So, the relevant questions you should address are:

  1. How fast will the logs grows (based on access check requests to the Cams policy server)?
  2. When (or will) you compress old logs?
  3. When (or will) you archive old logs to another storage device?

The Cams policy server installation only requires about 5mb of disk space. You'll also need enough disk space to install the Java virtual machine too.

RAID and hot swappable disks are generally optional. If a Cams server policy server running on a single disk system fails because the disks fails, the second Cams policy server in your cluster is still available. Obviously preventing such a failure and enabling easier system recovery would be facilitated with RAID and hot swappable disks. When using just one Cams policy server, consider using RAID and hot swappable disks.

Network

100MB Ethernet is sufficient between the Cams web agents and policy server. Obviously the faster the throughput between the Cams web agents and Cams policy server, the faster the response time. It is more important is to consider how much non-Cams network traffic might effect the Cams traffic. If the network is overloaded, then the environment will potentially reduce Cams policy server response times. In this case, you can either increase the available bandwidth or isolate traffic between the Cams web agents and Cams policy server by routing them over a dedicated network (or VLAN).

NOTE: Network performance is also an important issue to consider with Cams authentication. For example, if Cams is configured to use a centralized LDAP server that is saturated, Cams authentication responses will be slow or may even timeout. Though the tendency would be to perceive this as a Cams performance issue, it is not.

Back | Next | Contents