com.sun.labs.minion.indexer.partition
Class AsyncDumper

java.lang.Object
  com.sun.labs.minion.indexer.partition.AsyncDumper

All Implemented Interfaces:

Dumper, com.sun.labs.util.props.Component, com.sun.labs.util.props.Configurable, java.lang.Runnable

public class AsyncDumper
extends java.lang.Object
implements java.lang.Runnable, Dumper

A class that will be used to dump partitions in an orderly fashion. This class dumps partitions asynchronously from the threads that did the indexing, so a thread that dumps a partition may continue indexing new data before the old data has been dumped to disk.

We use a bounded queue for dump requests in order to throttle the indexers when they attempt to dump if the dumper cannot keep up. Note that the longer the queue the more memory will be required to store the indexed but undumped data.

Field Summary

protected boolean	done Whether we are done.
protected static java.lang.String	logTag
protected java.lang.String	name Our configuration name.
protected int	pollInterval The interval for polling the partition queue.
static java.lang.String	PROP_DO_GC
static java.lang.String	PROP_POLL_INTERVAL The poll interval for the queue, in seconds.
static java.lang.String	PROP_QUEUE_LENGTH The queue length property for configuration.
protected java.lang.Thread	t The thread that is running our dumping.
protected java.util.concurrent.BlockingQueue<com.sun.labs.minion.indexer.partition.AsyncDumper.StageHolder>	toDump A queue onto which partitions will be placed for dumping.

Constructor Summary

AsyncDumper()
Default constructor used for configuration.

Method Summary

void	dump(Stage s) Dumps the pipeline stage, which is assumed to be something that contains indexed data.
void	finish() Tells the thread we're finished dumping and then waits for it to catch up.
java.lang.String	getName()
int	getQueueLength() Gets the length of the queue that this dumper is using to store partitions before dumping them.
void	newProperties(com.sun.labs.util.props.PropertySheet ps)
void	run()
void	setSearchEngine(SearchEngine e) Sets the search engine that this dumper will be dumping partitions for.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

name

protected java.lang.String name

Our configuration name.

toDump

protected java.util.concurrent.BlockingQueue<com.sun.labs.minion.indexer.partition.AsyncDumper.StageHolder> toDump

A queue onto which partitions will be placed for dumping.

pollInterval

protected int pollInterval

The interval for polling the partition queue.

done

protected boolean done

Whether we are done.

t

protected java.lang.Thread t

The thread that is running our dumping.

PROP_QUEUE_LENGTH

@ConfigInteger(defaultValue=1)
public static java.lang.String PROP_QUEUE_LENGTH

The queue length property for configuration.

PROP_POLL_INTERVAL

@ConfigInteger(defaultValue=3)
public static java.lang.String PROP_POLL_INTERVAL

The poll interval for the queue, in seconds.

PROP_DO_GC

@ConfigBoolean(defaultValue=true)
public static final java.lang.String PROP_DO_GC

See Also:

Constant Field Values

logTag

protected static java.lang.String logTag

Constructor Detail

AsyncDumper

public AsyncDumper()

Default constructor used for configuration.

Method Detail

setSearchEngine

public void setSearchEngine(SearchEngine e)

Description copied from interface: Dumper

Sets the search engine that this dumper will be dumping partitions for.

Specified by:

setSearchEngine in interface Dumper

Parameters:

e - the engine that this dumper will be used by

getQueueLength

public int getQueueLength()

Description copied from interface: Dumper

Gets the length of the queue that this dumper is using to store partitions before dumping them.

Specified by:

getQueueLength in interface Dumper

Returns:

the queue length.

dump

public void dump(Stage s)

Description copied from interface: Dumper

Dumps the pipeline stage, which is assumed to be something that contains indexed data.

Specified by:

dump in interface Dumper

Parameters:

s - a pipeline stage containing data to be dumped.

run

public void run()

Specified by:

run in interface java.lang.Runnable

finish

public void finish()

Tells the thread we're finished dumping and then waits for it to catch up.

Specified by:

finish in interface Dumper

newProperties

public void newProperties(com.sun.labs.util.props.PropertySheet ps)
                   throws com.sun.labs.util.props.PropertyException

Specified by:

newProperties in interface com.sun.labs.util.props.Configurable

Throws:

com.sun.labs.util.props.PropertyException

getName

public java.lang.String getName()