# Configuring the Cluster/Load Balancer with GlassFish V2

### Contents

- Installing and Configuring GlassFish and the Web Server
- Setting up a One-machine Cluster using GlassFish V2
- Configuring the Load Balancer Plugin
- Deploying and Running the `clusterjsp` Sample Application
- High Availability/Failover using In-memory Replication feature
- Stop and Cleanup Cluster Settings
- Create Cluster using Glassfish Admin CLI commands
- Trouble Shooting

### Installing and Configuring GlassFish and the Web Server

Install the GlassFish server by downloading it from GlassFish Download Page at: https://glassfish.dev.java.net/public/downloadsindex.html

GlasssFish installation requires JDK 5 (1.5.0_10) and ant 1.6.5 (included).

1. Download one of the bundles to disk
2. Run:

       java -Xmx256m -jar filename.jar

3. cd glassfish
4. ant -f setup-cluster.xml

To download and install Load Balancer plugin to work with GlassFish, follow the steps below.

1. Download aslb (appserver load balancer plugin) from the link: SunOS_X86 , SunOS , Linux , Windows http://download.java.net/javaee5/external/<OS>/aslb/jars/aslb-9.1-MS4-b1.jar
2. This document explains the steps for Sun Webserver 61 plugin on Solaris, the bundle contains Apache Webserver plugin also. Please note that GlassFish V2 officially tested and supported only with Sun Webserver 6.1 and not supported for other webserver for the load balancer plugin.
3. Unjar to install and configure in GlassFish installation (this is ${glassfish.home}same as $S1AS_HOME) directory

- Unjar aslb-9.1-MS4-b1.jar file in "${glassfish.home}/lib/lbplugin"
- There are 2 zip files bundled inside the aslb jar: SUNWaslb.zip, SUNWaspx.zip, unzip these 2 zip files in the same directory and delete the zip files.
- Change permissions on all shared libraries of lbplugin as below
- chmod -R 755 ${glassfish.home}/lib/lbplugin/lib

Before you install the load balancer plugin, you must install Sun Java System Webserver 6.1 (SJSWS) first. SJSWS 6.1 is the supported
webserver for GlassFish V2 and these steps being specific to SJSWS 6.1. You can download the Web Server from:
http://www.sun.com/download/products.xml?id=434aec1d
To install and manually configure the SJSWS 6.1 for GlassFish HTPP load balancing, follow these steps.
Legends: `
websrvr_install_dir - Sun Java System Webserver install Directory
websrvr_instance_dir - Sun Java System Webserver Instance Directory
` `as_install_dir - GlassFish installation Directory.`

1. Create the following directories:
   - `<websrvr_install_dir>/plugins/lbplugin/bin`
   - `<websrvr_install_dir>/plugins/lbplugin/resource`
   - ``<websrvr_install_dir>/plugins/lbplugin/errorpages``
2. Copy `<as_install_dir/lib/lbplugin/lib/webserver-plugin/<OS>/iws61/libpassthrough.so`

   as `<websrvr_install_dir>/plugins/lbplugin/bin/libpassthrough.so`

   Where <OS> refers to 'linux' for the linux platform, and 'solaris' for the solaris platform.

3. Add execute permission to `<websrvr_install_dir>/plugins/lbplugin/bin/libpassthrough.so`.
4. Copy `<as_install_dir>/lib/lbplugin/lib/webserver-plugin/<OS>/iws61/errorpages/default-error.html`

   as `<websrvr_install_dir>/plugins/lbplugin/errorpages/default-error.html`

5. Copy `<as_install_dir>/lib/lbplugin/lib/webserver-plugin/<OS>/iws61/errorpages/sun-http-lberror.html`

   as `<websrvr_install_dir>/plugins/lbplugin/errorpages/sun-http-lberror.html`

6. Copy `<as_install_dir>/lib/lbplugin/lib/webserver-plugin/<OS>/iws61/*.res`

   To `<websrvr_install_dir>/plugins/lbplugin/resource`

7. Copy `<as_install_dir>/lib/lbplugin/lib/install/templates/loadbalancer.xml.example`

   as `<websrvr_instance_dir>/config/loadbalancer.xml.example`

- This is just an example for notational purpose, the user should manually edit (if choosen to do this way) this file prior to using this as loadbalancer.xml, to reflect the correct cluster configuration. Refer to “Configuring the Load Balancer Plugin” for this. Note manual editing is not the endorsed way to configure the load balancer. GlassFish Admin CLI or GUI are the two supported approaches for generating this file, while providing for error free load balancer configuration generation.

8. Copy `<as_install_dir>/lib/lbplugin/lib/dtds/sun-loadbalancer_1_2.dtd`

   as `<websrvr_instance_dir>/config/sun-loadbalancer_1_2.dtd`

9. Append the following lines to `<websrvr_instance_dir>/config/magnus.conf`.

          ##BEGIN EE LB Plugin Parameters
        Init fn="load-modules" shlib="<websrvr_install_dir>/plugins/lbplugin/bin/libpassthrough.so"
    funcs="init-passthrough,service-passthrough,name-trans-passthrough" Thread="no"
        Init fn="init-passthrough"
          ##END EE LB Plugin Parameters

The above lines need to be before following line.

    Init fn="load-modules" shlib=".../libj2eeplugin.so" shlib_flags="(global|now)"

Replace <websrvr_install_dir> with your Web Server installation directory.

10. Insert this line before the line of the first occurence of the string "nametrans" in `<websrvr_instance_dir>/config/obj.conf`

        NameTrans fn="name-trans-passthrough" name="lbplugin" config-file="<websrvr_instance_dir>/config/loadbalancer.xml"

    Replace `<websrvr_instance_dir>` with your Web Server Instance Directory.

11. Append the following lines to `<websrvr_instance_dir>/config/obj.conf`

             <Object name="lbplugin">
               ObjectType fn="force-type" type="magnus-internal/lbplugin"
                PathCheck fn="deny-existence" path="*/WEB-INF/*"
                Service type="magnus-internal/lbplugin" fn="service-passthrough"
                Error reason="Bad Gateway" fn="send-error" uri="$docroot/badgateway.html"
            </Object>

12. (Optional) For DAS based Load Balancer admininstration (the new feature), configure the WebServer for SSL. Refer to inputs at http://blogs.sun.com/sv96363/entry/ssl_setup_of_http_load
13. Start the Web Server if not running.

### Setting Up a One Machine Cluster

If you haven't already done so, download and install GlassFish latest promoted build.

Now to create the cluster, we need to first configure and start GlassFish V2 domain (which is in cluster mode), run following commands:

    % ant -f setup-cluster.xml
    % cd ${glassfish.home}/bin
    % asadmin start-domain --user ${adminuser} --passwordfile ${Admin-PasswordFile} domain-name

To create a cluster on the machine running the above Domain Administartion Server (DAS), follow these steps:

1. In a shell window, change your working directory to `${glassfish.home}/samples/quickstart/clusterjsp`
2. Run the command: `${glassfish.home}/bin/asant setup-one-machine-cluster` which automates following steps.
   - The values of `cluster.name` and `nodeagent.name` are specified here, change it appropriately. `${glassfish.home}/samples/quickstart/clusterjsp/cluster.properties`.
   - Creates a cluster with name `cluster.name (cluster1)`
   - Creates a node-agent with name `nodeagent.name (cluster1-nodeagent)
     `
   - Starts the node-agent
   - Creates two instances under the cluster that will use the node agent just created
3. Starts the cluster `${glassfish.home}/bin/asant start-cluster
   `
4. Troble-shooting:

- if you see the errors while starting the cluster, you can ignore some harmless errors. For more details, refer to troubleshooting section.
- On some Windows XP PCs, if this command hangs at "start-node-agent", do CNTR-C and retry again. Second time this goes through, starts node-agent, cluster successfully.
- If asant setup-one-machine-cluster is not starting the cluster, in that case asant start_cluster must be explicitly invoked.

5. You can do the above steps using Adminstration GUI by accessing https://<glassfish-host-name>:<admin-port>
6. GlassFish Admin CLI commands to create cluster can be found HERE.

### Configuring the Load Balancer Plugin

After installing the Load Balancer Plugin, configure the `<websrvr_instance_dir>/config/loadbalancer.xml``` file as follows:

    <!DOCTYPE loadbalancer PUBLIC "-//Sun Microsystems Inc.//DTD Sun One Application Server 8.1//EN"
    "sun-loadbalancer_1_2.dtd">
    <loadbalancer>
         <cluster name="cluster1"  policy="round-robin">
            <instance  name="instance1" enabled="true" disable-timeout-in-minutes="60"
    			listeners="http://instance-one-host:instance-one-port"  weight="100"/>
            <instance  name="instance2" enabled="true" disable-timeout-in-minutes="60"
    			listeners="http://instance-two-host:instance-two-port weight="100"/>
            <web-module context-root="clusterjsp" enabled="true" disable-timeout-in-minutes="60"
    			error-url="sun-http-lberror.html" />
            <health-checker url="/" interval-in-seconds="10" timeout-in-seconds="30" />
        </cluster>

        <property name="reload-poll-interval-in-seconds" value="60"/>
        <property name="response-timeout-in-seconds" value="30"/>
        <property name="https-routing" value="true"/>
        <property name="require-monitor-data" value="false"/>
        <property name="active-healthcheck-enabled" value="false"/>
        <property name="number-healthcheck-retries" value="3"/>
        <property name="rewrite-location" value="true"/>
    </loadbalancer>

To turn on load balancer plug-in monitor messages, set the `require-monitor-data` property value to true. The load balancer plug-in logs record the following information:

- Request start/stop information for every request.
- Failed-over request information when a request fails over from an unhealthy instance to a healthy instance.
- List of unhealthy instances at the end of every health-check cycle.

Restart the Web Server after configuring the LB plugin from `loadbalancer.xml file.`

### Deploying and Running the `clusterjsp` Sample Application

To test `cluster1` of two GlassFish V2 Server instances, use the "clusterjsp" application located in `${glassfish.home}/samples/quickstart/clusterjsp directory as clusterjsp.ear`

1. To deploy the `clusterjsp` application on "cluster1" of two instances, run following command

-     cd ${glassfish.home}/samples/quickstart/clusterjsp

-     % ${glassfish.home}/bin/asant deploy

- Now all the clustering, load balancing configurations and application deployment is done.

- You can access the `clusterjsp` application using `http://web-server-host:web-server-port/clusterjsp` (webserver port) and add sessions to the application.

2. You can see webserver logs from ${webserver instance directory}/logs/errors to see which instance is serving the http request, and also from the `server.log` in `${glassfish.home}/nodeagents/cluster1-nodeagent/instance-ONE (or instance-TWO)/logs`
3. This demostrates the cluster of two instances instance-ONE, and instance-TWO which are managed by the Node-agents and Domain Admininstration Server of GlassFish V2.

To simulate application server clustering along with load-balancing, to access a deployed clusterjsp sample application, follow the steps below.

1. Shut down the application server instance that is servicing a request (from the above setup)
2. Use this command to stop the instance:

       asadmin stop-instance --user adminuser --password adminpassword instance-name

Now you access the clusterjsp application again through webserver `http://web-server-host:web-server-port/clusterjsp`
and see the request is served by another instance that is running, Healthy and the Unhealthy instance is been detected by webserver load-balancer plugin. This ensures that requests are served by instances in the order set in the `loadbalancer.xml` file in Round-Robin fashion.

You can also find out which instance will serve the next request by checking access logs of both server instances. Since there are only two instances, the next request will be served by the other instance. The access log is `${glassfish.home}/nodeagents/node-agent-name/instance-name/logs/access/server_access_log
"10.5.79.63" "NULL-AUTH-USER" "17/Aug/2006:15:26:36 -0800" "POST /clusterjsp/HaJsp.jsp HTTP/1.1" 200 1572
`

This verifies the load balancing of GlassFish V2 with the Sun Java System Load Balancer Plugin.

High Availability/Failover using In-memory Replication feature

To verify GlassFish server High-Availability or failover while accessing a deployed clusterjsp sample application, follow the steps below:

1. You can make a particular instance serve a request by restarting the web server that has the load balancer plugin installed before deploying an application. This ensures that requests are served by instances in the order set in the `loadbalancer.xml` file. From the above `loadbalancer.xml` instance-ONE will serve the first request.
2. We have already deployed a sample application that stores session data, e.g. clusterjsp web application as above. With this we should be able to see that successive requests are served by the same instance that served the first request and the session data is maintained across the requests.
3. Send few requests and note down the instance that served those requests and shutdown that particular instance. Use this command to stop the instance:

- asadmin stop-instance --user adminuser --password adminpassword instance-name

4. Now send in the next request and verify that the new data is stored properly and that the previously added data is still there in the session.

This demonstrate the same shopping-cart scenario to avoid single-point of failure. If one of the server serving requests and adding items in shopping cart initially crashed, or not available anymore, then another server in the same cluster will take-over the request with all earlier session data (added items of shopping cart) and complete the request. Here the session data is stored in Memory, so this demonstrates the in-memory replication feature to get GlassFish server Highly Available.

### Stop and Cleanup Cluster Settings

1. To stop the `cluster1` Cluster of two instances and `cluster1-nodeagent` nodeagent, run following command

-     cd ${glassfish.home}/samples/quickstart/clusterjsp

-     % ${glassfish.home}/bin/asant stop_nodeagent

2. To remove cluster1 of two instances and node-agent, run following command

-     cd ${glassfish.home}/samples/quickstart/clusterjsp

-     % ${glassfish.home}/bin/asant delete_nodeagent

### Create Cluster using GlassFish Admin CLI commands

- The values of `cluster.name` and `nodeagent.name` are specified here, change it appropriately.`${glassfish.home}/samples/quickstart/clusterjsp/cluster.properties`.
- Creates a cluster with name `cluster.name (cluster1)`

-     asadmin create-cluster --user admin --passwordfile adminpassword.txt --host localhost -port 4848 cluster1

- Creates a node-agent with name `nodeagent.name (cluster1-nodeagent)`

-     asadmin create-node-agent --user admin --passwordfile adminpassword.txt --host localhost -port 4848 cluster1-nodeagent

- Starts the node-agent

-     asadmin start-node-agent --user admin --passwordfile adminpassword.txt --host localhost -port 4848 cluster1-nodeagent

- Creates two instances under the cluster that will use the node agent just created

    asadmin create-instance --user admin --passwordfile
    adminpassword.txt --host localhost -port 4848 --cluster cluster1
    --nodeagent cluster1-nodeagent --systemproperties
    "JMX_SYSTEM_CONNECTOR_PORT=8687:IIOP_LISTENER_PORT=3330:IIOP_SSL_LISTENER_PORT=4440:IIOP_SSL_

    MUTUALAUTH_PORT=5550:HTTP_LISTENER_PORT=1110:HTTP_SSL_LISTENER_PORT=2220"
    instance-ONE    asadmin create-instance --user admin --passwordfile
    adminpassword.txt
    --host localhost -port 4848 --cluster cluster1 --nodeagent
    cluster1-nodeagent --systemproperties
    "JMX_SYSTEM_CONNECTOR_PORT=8688:IIOP_LISTENER_PORT=3331:IIOP_SSL_LISTENER_PORT=4441:IIOP_SSL_

    MUTUALAUTH_PORT=5551:HTTP_LISTENER_PORT=1111:HTTP_SSL_LISTENER_PORT=2221"
    instance-TWO

- Starts the cluster

-     asadmin start-cluster --user admin --passwordfile adminpassword.txt
      --host localhost -port 4848 cluster1

### Trouble Shooting

1. If you see the following error message during start-cluster, you can safely ignore it, we are working on resolving this.
start_cluster:
[echo] Starting cluster cluster1
[exec] instance-ONE is running, does not require restart
[exec] error 0 [#|2006-08-18T11:22:58.078-0700|WARNING|sun-appserver-ee9.1|javax.jms|_ThreadID=10;_ThreadName=main;_RequestID=94652747-8288-451a-83d0-12fb326ce7d7;|[I500]: Caught JVM Exception: java.net.ConnectException: Connection refused|#]
2. If you see webserver startup failures for loading the libraries, you may need to do the following:
Edit the <websrv_instance_dir>/start script to update the LD_LIBRARY_PATH value to include <as_install_dir>/lib/lbplugin/lib. This contains the various binary dependencies for the loadbalancer plugin.