Java Code Examples for com.twitter.hbc.core.endpoint.StreamingEndpoint

The following examples show how to use com.twitter.hbc.core.endpoint.StreamingEndpoint. These examples are extracted from open source projects. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source Project: hazelcast-jet-contrib   Author: hazelcast   File: TwitterSources.java    License: Apache License 2.0 6 votes vote down vote up
private TwitterStreamSourceContext(
        @Nonnull ILogger logger,
        @Nonnull Properties credentials,
        @Nonnull String host,
        @Nonnull SupplierEx<? extends StreamingEndpoint> endpointSupplier
) {
    this.logger = logger;
    checkTwitterCredentials(credentials);
    String consumerKey = credentials.getProperty("consumerKey");
    String consumerSecret = credentials.getProperty("consumerSecret");
    String token = credentials.getProperty("token");
    String tokenSecret = credentials.getProperty("tokenSecret");

    Authentication auth = new OAuth1(consumerKey, consumerSecret, token, tokenSecret);
    client = new ClientBuilder()
            .hosts(host)
            .endpoint(endpointSupplier.get())
            .authentication(auth)
            .processor(new StringDelimitedProcessor(queue))
            .build();
    client.connect();
}
 
Example #2
Source Project: pulsar   Author: apache   File: TwitterFireHose.java    License: Apache License 2.0 6 votes vote down vote up
private StreamingEndpoint getEndpoint(TwitterFireHoseConfig config) {
    List<Long> followings = config.getFollowings();
    List<String> terms = config.getTrackTerms();

    if (CollectionUtils.isEmpty(followings) && CollectionUtils.isEmpty(terms)) {
        return new SampleStatusesEndpoint().createEndpoint();
    } else {
        StatusesFilterEndpoint hosebirdEndpoint = new StatusesFilterEndpoint();

        if (CollectionUtils.isNotEmpty(followings)) {
           hosebirdEndpoint.followings(followings);
        }

        if (CollectionUtils.isNotEmpty(terms)) {
           hosebirdEndpoint.trackTerms(terms);
        }

        return hosebirdEndpoint;
    }
}
 
Example #3
Source Project: hbc   Author: twitter   File: BasicClient.java    License: Apache License 2.0 6 votes vote down vote up
public BasicClient(String name, Hosts hosts, StreamingEndpoint endpoint, Authentication auth, boolean enableGZip, HosebirdMessageProcessor processor,
                   ReconnectionManager reconnectionManager, RateTracker rateTracker, ExecutorService executorService,
                   @Nullable BlockingQueue<Event> eventsQueue, HttpParams params, SchemeRegistry schemeRegistry) {
  Preconditions.checkNotNull(auth);
  HttpClient client;
  if (enableGZip) {
    client = new RestartableHttpClient(auth, enableGZip, params, schemeRegistry);
  } else {
    DefaultHttpClient defaultClient = new DefaultHttpClient(new PoolingClientConnectionManager(schemeRegistry), params);

    /** Set auth **/
    auth.setupConnection(defaultClient);
    client = defaultClient;
  }

  this.canRun = new AtomicBoolean(true);
  this.executorService = executorService;
  this.clientBase = new ClientBase(name, client, hosts, endpoint, auth, processor, reconnectionManager, rateTracker, eventsQueue);
}
 
Example #4
Source Project: hbc   Author: twitter   File: ClientBase.java    License: Apache License 2.0 6 votes vote down vote up
ClientBase(String name, HttpClient client, Hosts hosts, StreamingEndpoint endpoint, Authentication auth,
           HosebirdMessageProcessor processor, ReconnectionManager manager, RateTracker rateTracker,
           @Nullable BlockingQueue<Event> eventsQueue) {
  this.client = Preconditions.checkNotNull(client);
  this.name = Preconditions.checkNotNull(name);

  this.endpoint = Preconditions.checkNotNull(endpoint);
  this.hosts = Preconditions.checkNotNull(hosts);
  this.auth = Preconditions.checkNotNull(auth);

  this.processor = Preconditions.checkNotNull(processor);
  this.reconnectionManager = Preconditions.checkNotNull(manager);
  this.rateTracker = Preconditions.checkNotNull(rateTracker);

  this.eventsQueue = eventsQueue;

  this.exitEvent = new AtomicReference<Event>();

  this.isRunning = new CountDownLatch(1);
  this.statsReporter = new StatsReporter();

  this.connectionEstablished = new AtomicBoolean(false);
  this.reconnect = new AtomicBoolean(false);
}
 
Example #5
Source Project: Flink-CEPplus   Author: ljygz   File: TwitterSource.java    License: Apache License 2.0 5 votes vote down vote up
@Override
public StreamingEndpoint createEndpoint() {
	// this default endpoint initializer returns the sample endpoint: Returning a sample from the firehose (all tweets)
	StatusesSampleEndpoint endpoint = new StatusesSampleEndpoint();
	endpoint.stallWarnings(false);
	endpoint.delimited(false);
	return endpoint;
}
 
Example #6
Source Project: hazelcast-jet-contrib   Author: hazelcast   File: TwitterSources.java    License: Apache License 2.0 5 votes vote down vote up
/**
 * Equivalent of {@link #stream(Properties, SupplierEx)} with the
 * additional option to specify the Twitter {@code host}.
 *
 * @param host a Twitter host URL to connect to. These hosts are defined in
 *          {@link com.twitter.hbc.core.Constants}.
 */
@Nonnull
public static StreamSource<String> stream(
        @Nonnull Properties credentials,
        @Nonnull String host,
        @Nonnull SupplierEx<? extends StreamingEndpoint> endpointSupplier
) {
    return SourceBuilder.stream("twitter-stream-source",
            ctx -> new TwitterStreamSourceContext(ctx.logger(), credentials, host, endpointSupplier))
                        .fillBufferFn(TwitterStreamSourceContext::fillBuffer)
                        .destroyFn(TwitterStreamSourceContext::close)
                        .build();
}
 
Example #7
Source Project: hazelcast-jet-contrib   Author: hazelcast   File: TwitterSources.java    License: Apache License 2.0 5 votes vote down vote up
/**
 * Equivalent for {@link #timestampedStream(Properties, SupplierEx)} with
 * the additional option to specify the Twitter {@code host}.
 *
 * @param host a Twitter host URL to connect to. These hosts are defined in
 *          {@link com.twitter.hbc.core.Constants}.
 */
@Nonnull
public static StreamSource<String> timestampedStream(
        @Nonnull Properties credentials,
        @Nonnull String host,
        @Nonnull SupplierEx<? extends StreamingEndpoint> endpointSupplier
) {
    return SourceBuilder.timestampedStream("twitter-timestamped-stream-source",
            ctx -> new TwitterStreamSourceContext(ctx.logger(), credentials, host, endpointSupplier))
                        .fillBufferFn(TwitterStreamSourceContext::fillTimestampedBuffer)
                        .destroyFn(TwitterStreamSourceContext::close)
                        .build();
}
 
Example #8
Source Project: flink   Author: flink-tpc-ds   File: TwitterSource.java    License: Apache License 2.0 5 votes vote down vote up
@Override
public StreamingEndpoint createEndpoint() {
	// this default endpoint initializer returns the sample endpoint: Returning a sample from the firehose (all tweets)
	StatusesSampleEndpoint endpoint = new StatusesSampleEndpoint();
	endpoint.stallWarnings(false);
	endpoint.delimited(false);
	return endpoint;
}
 
Example #9
Source Project: pulsar   Author: apache   File: SampleStatusesEndpoint.java    License: Apache License 2.0 5 votes vote down vote up
public StreamingEndpoint createEndpoint() {
    // Returns the sample endpoint: Returning a sample from the firehose (all tweets)
    StatusesSampleEndpoint endpoint = new StatusesSampleEndpoint();
    endpoint.stallWarnings(false);
    endpoint.delimited(false);
    return endpoint;
}
 
Example #10
Source Project: flink   Author: apache   File: TwitterSource.java    License: Apache License 2.0 5 votes vote down vote up
@Override
public StreamingEndpoint createEndpoint() {
	// this default endpoint initializer returns the sample endpoint: Returning a sample from the firehose (all tweets)
	StatusesSampleEndpoint endpoint = new StatusesSampleEndpoint();
	endpoint.stallWarnings(false);
	endpoint.delimited(false);
	return endpoint;
}
 
Example #11
Source Project: hbc   Author: twitter   File: BasicClient.java    License: Apache License 2.0 4 votes vote down vote up
@Override
public StreamingEndpoint getEndpoint() {
  return clientBase.getEndpoint();
}
 
Example #12
Source Project: hbc   Author: twitter   File: ClientBase.java    License: Apache License 2.0 4 votes vote down vote up
ClientBase(String name, HttpClient client, Hosts hosts, StreamingEndpoint endpoint, Authentication auth,
           HosebirdMessageProcessor processor, ReconnectionManager manager, RateTracker rateTracker) {
  this(name, client, hosts, endpoint, auth, processor, manager, rateTracker, null);
}
 
Example #13
Source Project: hbc   Author: twitter   File: ClientBase.java    License: Apache License 2.0 4 votes vote down vote up
public StreamingEndpoint getEndpoint() {
  return endpoint;
}
 
Example #14
Source Project: hbc   Author: twitter   File: BaseTwitter4jClient.java    License: Apache License 2.0 4 votes vote down vote up
@Override
public StreamingEndpoint getEndpoint() {
  return client.getEndpoint();
}
 
Example #15
Source Project: hazelcast-jet-contrib   Author: hazelcast   File: TwitterSources.java    License: Apache License 2.0 3 votes vote down vote up
/**
 * Creates a {@link StreamSource} which reads tweets using Twitter's
 * Streaming API for data ingestion to Jet pipelines. This method uses the
 * {@link com.twitter.hbc.core.Constants#STREAM_HOST} as a Twitter
 * Streaming API host.
 * <p>
 * Example usage:
 * <pre>{@code
 * Properties credentials = loadTwitterCredentials();
 * List<String> terms = new ArrayList<>(Arrays.asList("BTC", "ETH"));
 * StreamSource<String> streamSource =
 *              TwitterSources.stream(
 *                      credentials,
 *                      () -> new StatusesFilterEndpoint().trackTerms(terms)
 *              );
 * Pipeline p = Pipeline.create();
 * StreamSourceStage<String> srcStage = p.readFrom(streamSource);
 * }</pre>
 *
 * @param credentials      a Twitter OAuth1 credentials that consist of
 *                         "consumerKey", "consumerSecret", "token",
 *                         "tokenSecret" keys.
 * @param endpointSupplier a supplier function that supplies a Twitter
 *                         StreamingEndpoint to connect to
 * @return a stream source to use in {@link Pipeline#readFrom}
 */
@Nonnull
public static StreamSource<String> stream(
        @Nonnull Properties credentials,
        @Nonnull SupplierEx<? extends StreamingEndpoint> endpointSupplier
) {
    return stream(credentials, Constants.STREAM_HOST, endpointSupplier);
}
 
Example #16
Source Project: hazelcast-jet-contrib   Author: hazelcast   File: TwitterSources.java    License: Apache License 2.0 3 votes vote down vote up
/**
 * Variant of {@link #stream(Properties, SupplierEx)} which parses the
 * messages and used the {@code timestamp_ms} field as a native timestamp.
 * If you don't need this timestamp, prefer to use the other variant.
 * <p>
 * For parameter documentation {@linkplain #stream(Properties,
 * SupplierEx) see here}.
 */
@Nonnull
public static StreamSource<String> timestampedStream(
        @Nonnull Properties credentials,
        @Nonnull SupplierEx<? extends StreamingEndpoint> endpointSupplier
) {
    return timestampedStream(credentials, Constants.STREAM_HOST, endpointSupplier);
}
 
Example #17
Source Project: Flink-CEPplus   Author: ljygz   File: TwitterSource.java    License: Apache License 2.0 votes vote down vote up
StreamingEndpoint createEndpoint(); 
Example #18
Source Project: flink   Author: flink-tpc-ds   File: TwitterSource.java    License: Apache License 2.0 votes vote down vote up
StreamingEndpoint createEndpoint(); 
Example #19
Source Project: flink   Author: apache   File: TwitterSource.java    License: Apache License 2.0 votes vote down vote up
StreamingEndpoint createEndpoint(); 
Example #20
Source Project: hbc   Author: twitter   File: Client.java    License: Apache License 2.0 votes vote down vote up
public StreamingEndpoint getEndpoint();