de.uni_mannheim.informatik.dws.winter.processing.parallel

Class ParallelProcessableCollection<RecordType>

java.lang.Object

de.uni_mannheim.informatik.dws.winter.processing.ProcessableCollection<RecordType>

de.uni_mannheim.informatik.dws.winter.processing.parallel.ParallelProcessableCollection<RecordType>

All Implemented Interfaces:

Processable<RecordType>, Serializable

Direct Known Subclasses:

ParallelHashedDataSet

public class ParallelProcessableCollection<RecordType>
extends ProcessableCollection<RecordType>

Multi-threaded implementation of Processable

Author:

Oliver Lehmberg (oli@dwslab.de)

See Also:

Serialized Form

Field Summary

Fields inherited from class de.uni_mannheim.informatik.dws.winter.processing.ProcessableCollection

elements

Constructor Summary

Constructors

Constructor and Description
ParallelProcessableCollection()
ParallelProcessableCollection(Collection<RecordType> elements)
ParallelProcessableCollection(Processable<RecordType> elements)

Method Summary

Modifier and Type	Method and Description
<KeyType,OutputRecordType,ResultType> Processable<Pair<KeyType,ResultType>>	aggregate(RecordKeyValueMapper<KeyType,RecordType,OutputRecordType> groupBy, DataAggregator<KeyType,OutputRecordType,ResultType> aggregator) Groups records based on the given groupBy mapper and then aggregates the groups using the aggregator.
<OutputRecordType> Processable<OutputRecordType>	createProcessable(OutputRecordType dummyForTypeInference) Creates a new Processable for the type given as parameter.
<OutputRecordType> Processable<OutputRecordType>	createProcessableFromCollection(Collection<OutputRecordType> data) Creates a new Processable with the provided elements as content
void	foreach(Action<RecordType> action) Iterates over the processable and executes the provided action for each element.
void	foreach(DataIterator<RecordType> iterator) Iterates over the given dataset without producing a result
<KeyType,OutputRecordType> Processable<Group<KeyType,OutputRecordType>>	group(RecordKeyValueMapper<KeyType,RecordType,OutputRecordType> groupBy) Groups records based on the given groupBy mapper.
protected <KeyType,ElementType> Map<KeyType,List<ElementType>>	hashRecords(Processable<ElementType> dataset, Function<KeyType,ElementType> hash) Applies the hash function to all records
<KeyType,RecordType2> Processable<Pair<RecordType,RecordType2>>	join(Processable<RecordType2> dataset2, Function<KeyType,RecordType> joinKeyGenerator1, Function<KeyType,RecordType2> joinKeyGenerator2) Joins two processables using the provided joinKeyGenerators (inner join).
<OutputRecordType> Processable<OutputRecordType>	map(RecordMapper<RecordType,OutputRecordType> transformation) Iterates over all elements and produces a result
Collection<Collection<RecordType>>	partitionRecords()
<KeyType> Processable<Pair<RecordType,RecordType>>	symmetricJoin(Function<KeyType,RecordType> joinKeyGenerator, ProcessableCollector<Pair<RecordType,RecordType>> collector) Joins the data to itself via the provided joinKeyGenerator (inner join).

Methods inherited from class de.uni_mannheim.informatik.dws.winter.processing.ProcessableCollection

add, addAll, append, assignUniqueRecordIds, coGroup, copy, distinct, firstOrNull, get, join, leftJoin, leftJoin, map, merge, remove, remove, size, sort, sort, symmetricJoin, take, where

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParallelProcessableCollection

public ParallelProcessableCollection()

ParallelProcessableCollection

public ParallelProcessableCollection(Collection<RecordType> elements)

ParallelProcessableCollection

public ParallelProcessableCollection(Processable<RecordType> elements)

Method Detail

createProcessable

public <OutputRecordType> Processable<OutputRecordType> createProcessable(OutputRecordType dummyForTypeInference)

Description copied from interface: Processable

Creates a new Processable for the type given as parameter.

Specified by:

createProcessable in interface Processable<RecordType>

Overrides:

createProcessable in class ProcessableCollection<RecordType>

Parameters:

dummyForTypeInference - a dummy variable that is used to infer the type parameter

Returns:

a new Processable

createProcessableFromCollection

public <OutputRecordType> Processable<OutputRecordType> createProcessableFromCollection(Collection<OutputRecordType> data)

Description copied from interface: Processable

Creates a new Processable with the provided elements as content

Specified by:

createProcessableFromCollection in interface Processable<RecordType>

Overrides:

createProcessableFromCollection in class ProcessableCollection<RecordType>

Parameters:

data - the data from which the new processable should be created

Returns:

a new Processable

foreach

public void foreach(DataIterator<RecordType> iterator)

Description copied from interface: Processable

Iterates over the given dataset without producing a result

Specified by:

foreach in interface Processable<RecordType>

Overrides:

foreach in class ProcessableCollection<RecordType>

Parameters:

iterator - the iterator that should be executed

foreach

public void foreach(Action<RecordType> action)

Description copied from interface: Processable

Iterates over the processable and executes the provided action for each element.

Specified by:

foreach in interface Processable<RecordType>

Overrides:

foreach in class ProcessableCollection<RecordType>

Parameters:

action - the action that should be executed

partitionRecords

public Collection<Collection<RecordType>> partitionRecords()

map

public <OutputRecordType> Processable<OutputRecordType> map(RecordMapper<RecordType,OutputRecordType> transformation)

Description copied from interface: Processable

Iterates over all elements and produces a result

Specified by:

map in interface Processable<RecordType>

Overrides:

map in class ProcessableCollection<RecordType>

Parameters:

transformation - the transformation that should be applied

Returns:

A Processable with the result of the operation

group

public <KeyType,OutputRecordType> Processable<Group<KeyType,OutputRecordType>> group(RecordKeyValueMapper<KeyType,RecordType,OutputRecordType> groupBy)

Description copied from interface: Processable

Groups records based on the given groupBy mapper. KeyType = Type of the grouping key RecordType = Type of the input records OutputType = Type of the records in the resulting groups (can be the same as RecordType)

Specified by:

group in interface Processable<RecordType>

Overrides:

group in class ProcessableCollection<RecordType>

Returns:

A Processable with the result of the operation

aggregate

public <KeyType,OutputRecordType,ResultType> Processable<Pair<KeyType,ResultType>> aggregate(RecordKeyValueMapper<KeyType,RecordType,OutputRecordType> groupBy,
                                                                                             DataAggregator<KeyType,OutputRecordType,ResultType> aggregator)

Description copied from interface: Processable

Groups records based on the given groupBy mapper and then aggregates the groups using the aggregator.

Specified by:

aggregate in interface Processable<RecordType>

Overrides:

aggregate in class ProcessableCollection<RecordType>

Returns:

A Processable with the result of the operation

hashRecords

protected <KeyType,ElementType> Map<KeyType,List<ElementType>> hashRecords(Processable<ElementType> dataset,
                                                                           Function<KeyType,ElementType> hash)

Description copied from class: ProcessableCollection

Applies the hash function to all records

Overrides:

hashRecords in class ProcessableCollection<RecordType>

Returns:

A map from the hash value to the respective records

join

public <KeyType,RecordType2> Processable<Pair<RecordType,RecordType2>> join(Processable<RecordType2> dataset2,
                                                                            Function<KeyType,RecordType> joinKeyGenerator1,
                                                                            Function<KeyType,RecordType2> joinKeyGenerator2)

Description copied from interface: Processable

Joins two processables using the provided joinKeyGenerators (inner join).

Specified by:

join in interface Processable<RecordType>

Overrides:

join in class ProcessableCollection<RecordType>

Returns:

A Processable with the result of the operation

symmetricJoin

public <KeyType> Processable<Pair<RecordType,RecordType>> symmetricJoin(Function<KeyType,RecordType> joinKeyGenerator,
                                                                        ProcessableCollector<Pair<RecordType,RecordType>> collector)

Description copied from interface: Processable

Joins the data to itself via the provided joinKeyGenerator (inner join). Assumes that the join is symmetric, i.e., a result a/b is equal to b/a and hence only a/b is created.

Specified by:

symmetricJoin in interface Processable<RecordType>

Overrides:

symmetricJoin in class ProcessableCollection<RecordType>

Parameters:

joinKeyGenerator - a function that returns the join key for each element

Returns:

A Processable with the result of the operation