Java Remote Method Invocation: 10 - RMI Wire Protocol

• Overview
• RMI Transport Protocol
• RMI's Use of Object Serialization
• RMI's Use of HTTP POST Protocol
• Application-Specific Values for RMI
• RMI's Multiplexing Protocol

10.1 Overview

The RMI protocol makes use of two other protocols for its on-the-wire format: Java Object Serialization and HTTP. The Object Serialization protocol is used to marshal call and return data. The HTTP protocol is used to "POST" a remote method invocation and obtain return data when circumstances warrant. Each protocol is documented as a separate grammar. Nonterminal symbols in production rules may refer to rules governed by another protocol (either Object Serialization or HTTP). When a protocol boundary is crossed, subsequent productions use that embedded protocol.

Notes about Grammar Notation

• We use a similar notation to that used in The Java Language Specification.
• Control codes in the stream are represented by literal values expressed in hexadecimal.
• Some nonterminal symbols in the grammar represent application specific values supplied in a method invocation. The definition of such a nonterminal consists of its Java programming language type. A table mapping each of these nonterminals to its respective type follows the grammar.

10.2 RMI Transport Protocol

The wire format for RMI is represented by a Stream. The terminology adopted here reflects a client perspective. Out refers to output messages and In refers to input messages. The contents of the transport header are not formatted using object serialization.

Stream:
    Out
    In

The input and output streams used by RMI are paired. Each Out stream has a corresponding In stream. An Out stream in the grammar maps to the output stream of a socket (from the client's perspective). An In stream (in the grammar) is paired with the corresponding socket's input stream. Since output and input streams are paired, the only header information needed on an input stream is an acknowledgment as to whether the protocol is understood; other header information (such as the magic number and version number) can be implied by the context of stream pairing.

10.2.1 Format of an Output Stream

An output stream in RMI consists of transport Header information followed by a sequence of Messages. Alternatively, an output stream can contain an invocation embedded in the HTTP protocol.

Out:
    Header Messages
    HttpMessage

Header:
    0x4a 0x52 0x4d 0x49 Version Protocol

Version:
    0x00 0x01

Protocol:
    StreamProtocol
    SingleOpProtocol
    MultiplexProtocol

StreamProtocol:
    0x4b

SingleOpProtocol:
    0x4c

MultiplexProtocol:
    0x4d

Messages:
    Message
    Messages Message

The Messages are wrapped within a particular protocol as specified by Protocol. For the SingleOpProtocol, there may only be one Message after the Header, and there is no additional data that the Message is wrapped in. The SingleOpProtocol is used for invocation embedded in HTTP requests, where interaction beyond a single request and response is not possible.

For the StreamProtocol and the MultiplexProtocol, the server must respond with a byte 0x4e acknowledging support for the protocol, and an EndpointIdentifier that contains the host name and port number that the server can see is being used by the client. The client can use this information to determine its host name if it is otherwise unable to do that for security reasons. The client must then respond with another EndpointIdentifier that contains the client's default endpoint for accepting connections. This can be used by a server in the MultiplexProtocol case to identify the client.

For the StreamProtocol, after this endpoint negotiation, the Messages are sent over the output stream without any additional wrapping of the data. For the MultiplexProtocol, the socket connection is used as the concrete connection for a multiplexed connection, as described in Section 10.6, "RMI's Multiplexing Protocol". Virtual connections initiated over this multiplexed connection consist of a series of Messages as described below.

There are three types of output messages: Call, Ping and DgcAck. A Call encodes a method invocation. A Ping is a transport-level message for testing liveness of a remote virtual machine. A DgcAck is an acknowledgment directed to a server's distributed garbage collector that indicates that remote objects in a return value from a server have been received by the client.

Message:
    Call
    Ping
    DgcAck

Call:
    0x50 CallData

Ping:
    0x52

DgcAck:
    0x54 UniqueIdentifier

10.2.2 Format of an Input Stream

There are currently three types of input messages: ReturnData, HttpReturn and PingAck. ReturnData is the result of a "normal" RMI call. An HttpReturn is a return result from an invocation embedded in the HTTP protocol. A PingAck is the acknowledgment for a Ping message.

In:
    ProtocolAck Returns
    ProtocolNotSupported
    HttpReturn

ProtocolAck:
    0x4e

ProtocolNotSupported:
    0x4f

Returns:
    Return
    Returns Return

Return:
    ReturnData
    PingAck

ReturnData:
    0x51 ReturnValue

PingAck:
    0x53

10.3 RMI's Use of Object Serialization

Call and return data in RMI calls are formatted using the Java Object Serialization protocol. Each method invocation's CallData is written to a Java object output stream that contains the ObjectIdentifier (the target of the call), an Operation (a number representing the method to be invoked), a Hash (a number that verifies that client stub and remote object skeleton use the same stub protocol), followed by a list of zero or more Arguments for the call.

In the JDK1.1 stub protocol the Operation represents the method number as assigned by rmic, and the Hash was the stub/skeleton hash which is the stub's interface hash. As of the Java 2 stub protocol (Java 2 stubs are generated using the -v1.2 option with rmic), Operation has the value -1 and the Hash is a hash representing the method to call. The hash is described in the section "The RemoteRef Interface".

CallData:
    ObjectIdentifier Operation Hash Arguments[opt]
    ObjectIdentifier:
    ObjectNumber UniqueIdentifier

UniqueIdentifier:
    Number Time Count

Arguments:
    Value
    Arguments Value

Value:
    Object
    Primitive

A ReturnValue of an RMI call consists of a return code to indicate either a normal or exceptional return, a UniqueIdentifier to tag the return value (used to send a DGCAck if necessary) followed by the return result: either the Value returned or the Exception thrown.

ReturnValue:
    0x01 UniqueIdentifier Value[opt]
    0x02 UniqueIdentifier Exception

Note: ObjectIdentifier, UniqueIdentifier, and EndpointIdentifier are not written out using default serialization, but each uses its own special write method (this is not the writeObject method used by object serialization); the write method for each type of identifier adds its component data consecutively to the output stream.

10.3.1 Class Annotation and Class Loading

RMI overrides the annotateClass and resolveClass methods of ObjectOutputStream and ObjectInputStream respectively. Each class is annotated with the codebase URL (the location from which the class can be loaded). In the annotateClass method, the classloader that loaded the class is queried for its codebase URL. If the classloader is non-null and the classloader has a non-null codebase, then the codebase is written to the stream using the ObjectOutputStream.writeObject method; otherwise a null is written to the stream using the writeObject method. Note: as an optimization, classes in the "java" package are not annotated, since they are always available to the receiver.

The class annotation is resolved during deserialization using the ObjectInputStream.resolveClass method. The resolveClass method first reads the annotation via the ObjectInputStream.readObject method. If the annotation, a codebase URL, is non-null, then it obtains the classloader for that URL and attempts to load the class. The class is loaded by using a java.net.URLConnection to fetch the class bytes (the same mechanism used by a web browser's applet classloader).

10.4 RMI's Use of HTTP POST Protocol

The implementation of RMI calls through firewalls via proxies has been removed as of JDK 9.

10.5 Application-Specific Values for RMI

This table lists the nonterminal symbols that represent application-specific values used by RMI. The table maps each symbol to its respective type. Each is formatted using the protocol in which it is embedded.

10.6 RMI's Multiplexing Protocol

The purpose of multiplexing is to provide a model where two endpoints can each open multiple full duplex connections to the other endpoint in an environment where only one of the endpoints is able to open such a bidirectional connection using some other facility (e.g., a TCP connection). RMI use this simple multiplexing protocol to allow a client to connect to an RMI server object in some situations where that is otherwise not possible. For example, some security managers for applet environments disallow the creation of server sockets to listen for incoming connections, thereby preventing such applets to export RMI objects and service remote calls from direct socket connections. If the applet can open a normal socket connection to its codebase host, however, then it can use the multiplexing protocol over that connection to allow the codebase host to invoke methods on RMI objects exported by the applet. This section describes how the format and rules of the multiplexing protocol.

10.6.1 Definitions

This sections defines some terms as they are used in the rest of the description of the protocol.

An endpoint is one of the two users of a connection using the multiplexing protocol.

The multiplexing protocol must layer on top of one existing bidirectional, reliable byte stream, presumably initiated by one of the endpoints to the other. In current RMI usage, this is always a TCP connection, made with a java.net.Socket object. This connection will be referred to as the concrete connection.

The multiplexing protocol facilitates the use of virtual connections, which are themselves bidirectional, reliable byte streams, representing a particular session between two endpoints. The set of virtual connections between two endpoints over a single concrete connection comprises a multiplexed connection. Using the multiplexing protocol, virtual connections can be opened and closed by either endpoint. The state of an virtual connection with respect to a given endpoint is defined by the elements of the multiplexing protocol that are sent and received over the concrete connection. Such state involves if the connection is open or closed, the actual data that has been transmitted across, and the related flow control mechanisms. If not otherwise qualified, the term connection used in the remainder of this section means virtual connection.

A virtual connections within a given multiplexed connection is identified by a 16 bit integer, known as the connection identifier. Thus, there exist 65,536 possible virtual connections in one multiplexed connection. The implementation may limit the number of these virtual connections that may be used simultaneously.

10.6.2 Connection State and Flow Control

Connections are manipulated using the various operations defined by the multiplexing protocol. The following are the names of the operations defined by the protocol: OPEN, CLOSE, CLOSEACK, REQUEST, and TRANSMIT. The exact format and rules for all the operations are detailed in Section 10.6.3, "Protocol Format".

The OPEN, CLOSE, and CLOSEACK operations control connections becoming opened and closed, while the REQUEST and TRANSMIT operations are used to transmit data across an open connection within the constraints of the flow control mechanism.

Connection States

A virtual connection is open with respect to a particular endpoint if the endpoint has sent an OPEN operation for that connection, or it has received an OPEN operation for that connection (and it had not been subsequently closed). The various protocol operations are described below.

A virtual connection is pending close with respect to a particular endpoint if the endpoint has sent a CLOSE operation for that connection, but it has not yet received a subsequent CLOSE or CLOSEACK operation for that connection.

A virtual connection is closed with respect to a particular endpoint if it has never been opened, or if it has received a CLOSE or a CLOSEACK operation for that connection (and it has not been subsequently opened).

Flow Control

The multiplexing protocol uses a simple packeting flow control mechanism to allow multiple virtual connections to exist in parallel over the same concrete connection. The high level requirement of the flow control mechanism is that the state of all virtual connections is independent; the state of one connection may not affect the behavior of others. For example, if the data buffers handling data coming in from one connection become full, this cannot prevent the transmission and processing of data for any other connection. This is necessary if the continuation of one connection is dependent on the completion of the use of another connection, such as would happen with recursive RMI calls. Therefore, the practical implication is that the implementation must always be able to consume and process all of the multiplexing protocol data ready for input on the concrete connection (assuming that it conforms to this specification).

Each endpoint has two state values associated with each connection: how many bytes of data the endpoint has requested but not received (input request count) and how many bytes the other endpoint has requested but have not been supplied by this endpoint (output request count).

An endpoint's output request count is increased when it receives a REQUEST operation from the other endpoint, and it is decreased when it sends a TRANSMIT operation. An endpoint's input request count is increased when it sends a REQUEST operation, and it is decreased when it receives a TRANSMIT operation. It is a protocol violation if either of these values becomes negative.

It is a protocol violation for an endpoint to send a REQUEST operation that would increase its input request count to more bytes that it can currently handle without blocking. It should, however, make sure that its input request count is greater than zero if the user of the connection is waiting to read data.

It is a protocol violation for an endpoint to send a TRANSMIT operation containing more bytes that its output request count. It may buffer outgoing data until the user of the connection requests that data written to the connection be explicitly flushed. If data must be sent over the connection, however, by either an explicit flush or because the implementation's output buffers are full, then the user of the connection may be blocked until sufficient TRANSMIT operations can proceed.

Beyond the rules outlined above, implementations are free to send REQUEST and TRANSMIT operations as deemed appropriate. For example, an endpoint may request more data for a connection even if its input buffer is not empty.

10.6.3 Protocol Format

The byte stream format of the multiplexing protocol consists of a contiguous series of variable length records. The first byte of the record is an operation code that identifies the operation of the record and determines the format of the rest of its content. The following legal operation codes are defined:

It is a protocol violation if the first byte of a record is not one of the defined operation codes. The following sections describe the format of the records for each operation code.

OPEN operation

This is the format for records of the OPEN operation:

An endpoint sends an OPEN operation to open the indicated connection. It is a protocol violation if ID refers to a connection that is currently open or pending close with respect to the sending endpoint. After the connection is opened, both input and request count states for the connection are zero for both endpoints.

Receipt of an OPEN operation indicates that the other endpoint is opening the indicated connection. After the connection is opened, both input and output request count states for the connection are zero for both endpoints.

To prevent identifier collisions between the two endpoints, the space of valid connection identifiers is divided in half, depending on the value of the most significant bit. Each endpoint is only allowed to open connections with a particular value for the high bit. The endpoint that initiated the concrete connection must only open connections with the high bit set in the identifier and the other endpoint must only open connections with a zero in the high bit. For example, if an RMI applet that cannot create a server socket initiates a multiplexed connection to its codebase host, the applet may open virtual connections in the identifier range 0x8000-7FFF, and the server may open virtual connection in the identifier range 0-0x7FFF.

CLOSE operation

This is the format for records of the CLOSE operation:

An endpoint sends a CLOSE operation to close the indicated connection. It is a protocol violation if ID refers to a connection that is currently closed or pending close with respect to the sending endpoint (it may be pending close with respect to the receiving endpoint if it has also sent a CLOSE operation for this connection). After sending the CLOSE, the connection becomes pending close for the sending endpoint. Thus, it may not reopen the connection until it has received a CLOSE or a CLOSEACK for it from the other endpoint.

Receipt of a CLOSE operation indicates that the other endpoint has closed the indicated connection, and it thus becomes closed on the receiving endpoint. Although the receiving endpoint may not send any more operations for this connection (until it is opened again), it still should provide data in the implementation's input buffers to readers of the connection. If the connection had previously been open instead of pending close, the receiving endpoint must respond with a CLOSEACK operation for the connection.

CLOSEACK operation

The following is the format for records with the CLOSEACK operation:

An endpoint sends a CLOSEACK operation to acknowledge a CLOSE operation from the receiving endpoint. It is a protocol violation if ID refers to a connection that is not pending close for the receiving endpoint when the operation is received.

Receipt of a CLOSEACK operation changes the state of the indicated connection from pending close to closed, and thus the connection may be reopened in the future.

REQUEST operation

This is the format for records of the REQUEST operation:

An endpoint sends a REQUEST operation to increase its input request count for the indicated connection. It is a protocol violation if ID does not refer to a connection that is open with respect to the sending endpoint. The endpoint's input request count is incremented by the value count. The value of count is a signed 32 bit integer, and it is a protocol violation if it is negative or zero.

Receipt of a REQUEST operation causes the output request count for the indicated connection to increase by count. If the connection is pending close by the receiving endpoint, then any REQUEST operations may be ignored.

TRANSMIT operation

This is the format for records of the TRANSMIT operation.

An endpoint sends a TRANSMIT operation to actually transmit data over the indicated connection. It is a protocol violation if ID does not refer to a connection that is open with respect to the sending endpoint. The endpoint's output request count is decremented by the value count. The value of count is a signed 32 bit integer, and it is a protocol violation if it is negative or zero. It is also a protocol violation if the TRANSMIT operation would cause the sending endpoint's output request count to become negative.

Receipt of a TRANSMIT operation causes the count bytes of data to be added to the queue of bytes available for reading from the connection. The receiving endpoint's input request count is decremented by count. If this causes the input request count to become zero and the user of the connection is trying to read more data, the endpoint should respond with another REQUEST operation. If the connection is pending close by the receiving endpoint, then any TRANSMIT operations may be ignored.

Protocol Violations

If a protocol violation occurs, as defined above or if a communication error is detected in the concrete connection, then the multiplexed connection is shut down. The real connection is terminated, and all virtual connections become closed immediately. Data already available for reading from virtual connections may be read by the users of the connections.