Java Code Examples for org.apache.samza.system.IncomingMessageEnvelope#getSystemStreamPartition()

The following examples show how to use org.apache.samza.system.IncomingMessageEnvelope#getSystemStreamPartition() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: TaskSideInputHandler.java    From samza with Apache License 2.0 6 votes vote down vote up 
/**
 * Processes the incoming side input message envelope and updates the last processed offset for its SSP.
 * Synchronized inorder to be exclusive with flush().
 *
 * @param envelope incoming envelope to be processed
 */
public synchronized void process(IncomingMessageEnvelope envelope) {
  SystemStreamPartition envelopeSSP = envelope.getSystemStreamPartition();
  String envelopeOffset = envelope.getOffset();

  for (String store: this.sspToStores.get(envelopeSSP)) {
    SideInputsProcessor storeProcessor = this.storeToProcessor.get(store);
    KeyValueStore keyValueStore = (KeyValueStore) this.taskSideInputStorageManager.getStore(store);
    Collection<Entry<?, ?>> entriesToBeWritten = storeProcessor.process(envelope, keyValueStore);

    // TODO: SAMZA-2255: optimize writes to side input stores
    for (Entry entry : entriesToBeWritten) {
      // If the key is null we ignore, if the value is null, we issue a delete, else we issue a put
      if (entry.getKey() != null) {
        if (entry.getValue() != null) {
          keyValueStore.put(entry.getKey(), entry.getValue());
        } else {
          keyValueStore.delete(entry.getKey());
        }
      }
    }
  }

  this.lastProcessedOffsets.put(envelopeSSP, envelopeOffset);
}
Example 2
Source File: MultiFileHdfsReader.java    From samza with Apache License 2.0 5 votes vote down vote up 
public IncomingMessageEnvelope readNext() {
  if (!hasNext()) {
    LOG.warn("Attempting to read more data when there aren't any. ssp=" + systemStreamPartition);
    return null;
  }
  // record the next offset before we read, so when the read fails and we reconnect,
  // we seek to the same offset that we try below
  curSingleFileOffset = curReader.nextOffset();
  IncomingMessageEnvelope messageEnvelope = curReader.readNext();
  // Copy everything except for the offset. Turn the single-file style offset into a multi-file one
  return new IncomingMessageEnvelope(messageEnvelope.getSystemStreamPartition(), getCurOffset(),
    messageEnvelope.getKey(), messageEnvelope.getMessage(), messageEnvelope.getSize(),
    messageEnvelope.getEventTime(), messageEnvelope.getArrivalTime());
}
Example 3
Source File: RunLoop.java    From samza with Apache License 2.0 5 votes vote down vote up 
private boolean checkEndOfStream() {
  if (pendingEnvelopeQueue.size() == 1) {
    PendingEnvelope pendingEnvelope = pendingEnvelopeQueue.peek();
    IncomingMessageEnvelope envelope = pendingEnvelope.envelope;

    if (envelope.isEndOfStream()) {
      SystemStreamPartition ssp = envelope.getSystemStreamPartition();
      processingSspSet.remove(ssp);
      if (!hasIntermediateStreams) {
        pendingEnvelopeQueue.remove();
      }
    }
  }
  return processingSspSet.isEmpty();
}
Example 4
Source File: FanOutTask.java    From newsfeed with MIT License 5 votes vote down vote up 
@Override
@SuppressWarnings("unchecked")
public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) {
  String incomingStream = envelope.getSystemStreamPartition().getStream();
  if (incomingStream.equals(NewsfeedConfig.FOLLOWS_STREAM.getStream())) {
    processFollowsEvent((Map<String, Object>) envelope.getMessage());
  } else if (incomingStream.equals(NewsfeedConfig.MESSAGES_STREAM.getStream())) {
    processMessageEvent((Map<String, Object>) envelope.getMessage(), collector);
  } else {
    throw new IllegalStateException("Unexpected input stream: " + envelope.getSystemStreamPartition());
  }
}