com.sun.labs.minion.pipeline
Class AsyncPipelineImpl

java.lang.Object
  com.sun.labs.minion.pipeline.AbstractPipelineImpl
      com.sun.labs.minion.pipeline.AsyncPipelineImpl

All Implemented Interfaces:

Pipeline, java.lang.Runnable

public class AsyncPipelineImpl
extends AbstractPipelineImpl
implements java.lang.Runnable

A class that encapsulates the machinery of a single indexing pipeline. A pipeline can be used for indexing data or for performing highlighting operations.

Field Summary

protected boolean	doDump Whether we need to dump our partition.
protected boolean	doFlush Whether we should flush all of the data currently in the pipeline.
protected boolean	doPurge Whether we need to purge the data in this partition/pipeline.
protected boolean	finished Whether we've finished and must quit.
protected java.util.concurrent.BlockingQueue<Indexable>	indexingQueue The queue from which we will take documents to index.
protected boolean	inDoc Whether we're currently in a document.
protected boolean	simpleIndexingFinished Whether we're finished as a SimpleIndexer.

Fields inherited from class com.sun.labs.minion.pipeline.AbstractPipelineImpl

currKey, d64, docSize, dumper, engine, head, logTag, pipeline, text

Constructor Summary

AsyncPipelineImpl(PipelineFactory factory, SearchEngine engine, java.util.List<Stage> pipeline, Dumper dumper, java.util.concurrent.BlockingQueue<Indexable> indexingQueue)
Instantiates a pipeline.

Method Summary

protected void	drain() Drains the indexing queue, indexing each of the elements.
void	dump() Dumps the current indexing data to disk.
void	flush() Flushes all the data currently held in the queue.
void	index(Indexable doc) Adds an indexable object to the pipeline for indexing.
void	purge() Purge the data currently in the pipeline.
void	run() Removes documents from the queue and indexes them.
void	shutdown() Shuts down this pipeline, making sure that any documents in the queue have been finished.

Methods inherited from class com.sun.labs.minion.pipeline.AbstractPipelineImpl

addImpliedField, endDocument, getEngine, getHead, getIndexer, handleField, handleField, indexDoc, isIndexed, realDump, realPurge, setIndexer, startDocument

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

indexingQueue

protected java.util.concurrent.BlockingQueue<Indexable> indexingQueue

The queue from which we will take documents to index.

inDoc

protected boolean inDoc

Whether we're currently in a document.

simpleIndexingFinished

protected boolean simpleIndexingFinished

Whether we're finished as a SimpleIndexer.

finished

protected boolean finished

Whether we've finished and must quit.

doFlush

protected boolean doFlush

Whether we should flush all of the data currently in the pipeline.

doDump

protected boolean doDump

Whether we need to dump our partition.

doPurge

protected boolean doPurge

Whether we need to purge the data in this partition/pipeline.

Constructor Detail

AsyncPipelineImpl

public AsyncPipelineImpl(PipelineFactory factory,
                         SearchEngine engine,
                         java.util.List<Stage> pipeline,
                         Dumper dumper,
                         java.util.concurrent.BlockingQueue<Indexable> indexingQueue)

Instantiates a pipeline.

Parameters:

factory - the factory from which to make this pipeline

engine - The search engine for which this pipeline will be processing documents.

pipeline - the stages in the pipeline

dumper - the dumper that will dump partitions after indexing

indexingQueue - the queue of items to index

Method Detail

run

public void run()

Removes documents from the queue and indexes them.

Specified by:

run in interface java.lang.Runnable

drain

protected void drain()

Drains the indexing queue, indexing each of the elements.

flush

public void flush()

Flushes all the data currently held in the queue. If the pipeline is asynchronous, then the flush to disk will occur asynchronously when all of the data in the pipline has been indexed.

Specified by:

flush in interface Pipeline

dump

public void dump()

Dumps the current indexing data to disk. This will not empty out the indexing pipeline before the data are written.

Specified by:

dump in interface Pipeline

purge

public void purge()

Purge the data currently in the pipeline. The data is thrown out, not getting written out to disk. It is best not to be indexing while a purge is in progress.

Specified by:

purge in interface Pipeline

Throws:

SearchEngineException

shutdown

public void shutdown()

Shuts down this pipeline, making sure that any documents in the queue have been finished.

Specified by:

shutdown in interface Pipeline

index

public void index(Indexable doc)

Description copied from interface: Pipeline

Adds an indexable object to the pipeline for indexing.

Specified by:

index in interface Pipeline

Parameters:

doc - a document to index.