Java Code Examples for com.hazelcast.jet.pipeline.SourceBuilder#SourceBuffer

The following examples show how to use com.hazelcast.jet.pipeline.SourceBuilder#SourceBuffer . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: RedisSources.java    From hazelcast-jet-contrib with Apache License 2.0 6 votes vote down vote up 
void fillBuffer(SourceBuilder.SourceBuffer<T> buffer) throws InterruptedException {
    if (exception != null) { //something went wrong on the Redis client thread
        throw exception;
    }

    int itemsFetched = queue.drainTo(batchHolder, NO_OF_ITEMS_TO_FETCH_AT_ONCE);
    if (itemsFetched <= 0) {
        if (commandFuture.isDone()) {
            buffer.close();
        }
    } else {
        batchHolder.stream()
                   .map(mapFn)
                   .forEach(buffer::add);
        batchHolder.clear();
    }
}
Example 2
Source File: RedisSources.java    From hazelcast-jet-contrib with Apache License 2.0 6 votes vote down vote up 
void fillBuffer(SourceBuilder.SourceBuffer<ScoredValue<V>> buffer) throws InterruptedException {
    if (exception != null) { //something went wrong on the Redis client thread
        throw exception;
    }

    for (int i = 0; i < NO_OF_ITEMS_TO_FETCH_AT_ONCE; i++) {
        ScoredValue<V> item = queue.poll(POLL_DURATION.toMillis(), TimeUnit.MILLISECONDS);
        if (item == null) {
            if (commandFuture.isDone()) {
                buffer.close();
            }
            return;
        } else {
            buffer.add(item);
        }
    }

}
Example 3
Source File: TwitterSources.java    From hazelcast-jet-contrib with Apache License 2.0 5 votes vote down vote up 
private void fillBuffer(SourceBuilder.SourceBuffer<String> sourceBuffer) {
    queue.drainTo(buffer, MAX_FILL_ELEMENTS);
    for (String item : buffer) {
        sourceBuffer.add(item);
    }
    buffer.clear();
}
Example 4
Source File: TwitterSources.java    From hazelcast-jet-contrib with Apache License 2.0 5 votes vote down vote up 
private void fillBuffer(SourceBuilder.SourceBuffer<Status> sourceBuffer) throws TwitterException {
    if (searchResult != null) {
        List<Status> tweets = searchResult.getTweets();
        for (Status tweet : tweets) {
            sourceBuffer.add(tweet);
        }
        searchResult = searchResult.nextQuery() != null ? twitter4JClient.search(searchResult.nextQuery()) : null;
    } else {
        sourceBuffer.close();
    }
}
Example 5
Source File: MongoDBSourceBuilder.java    From hazelcast-jet-contrib with Apache License 2.0 5 votes vote down vote up 
void fillBuffer(SourceBuilder.SourceBuffer<U> buffer) {
    for (int i = 0; i < BATCH_SIZE; i++) {
        if (cursor.hasNext()) {
            U item = mapFn.apply(cursor.next());
            if (item != null) {
                buffer.add(item);
            }
        } else {
            buffer.close();
        }
    }
}