cascading.scheme.Scheme Java Examples

The following examples show how to use cascading.scheme.Scheme. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.

Example #1

Source File: FlinkConnector.java From cascading-flink with Apache License 2.0

4 votes

@Override
protected Class<? extends Scheme> getDefaultIntermediateSchemeClass() {
	return null; // not required for Flink
}

Example #2

Source File: FlinkTestPlatform.java From cascading-flink with Apache License 2.0

4 votes

@Override
public Tap getTap(Scheme scheme, String filename, SinkMode mode) {
	return new Hfs(scheme, filename, mode);
}

Example #3

Source File: FlinkTestPlatform.java From cascading-flink with Apache License 2.0

4 votes

@Override
public Scheme getTestConfigDefScheme() {
	return new FlinkConfigDefScheme( new Fields( "line" ));
}

Example #4

Source File: FlinkTestPlatform.java From cascading-flink with Apache License 2.0

4 votes

@Override
public Scheme getTestFailScheme() {
	return new HadoopFailScheme( new Fields( "line" ) );
}

Example #5

Source File: TapTypeUtilTest.java From plunger with Apache License 2.0

4 votes

private TestHfs(@SuppressWarnings("rawtypes") Scheme<Configuration, RecordReader, OutputCollector, ?, ?> scheme,
    String stringPath) {
  super(scheme, stringPath);
}

Example #6

Source File: BigDiffCascading.java From secure-data-service with Apache License 2.0

4 votes

public void execute(String inputPath1, String inputPath2, String outputPath) {

        // define source and sink Taps.
        Scheme sourceScheme = new TextLine( new Fields( "line" ) );
        Tap lhSource = new Hfs( sourceScheme, inputPath1 );
        Tap rhSource = new Hfs( sourceScheme, inputPath2 );

        Scheme sinkScheme = new TextLine( new Fields( "source", "line" ) );
        Tap sink = new Hfs( sinkScheme, outputPath, SinkMode.REPLACE );

        /*
        Pipe lhs = new Pipe("lhs");
        lhs = new Each(lhs, new Fields("source", "line"), )
        
        String regex = "(left|right)[\\s]+(.*)";
        Function function = new RegexGenerator( new Fields( "source", "line" ), regex );
        assembly = new Each( assembly, new Fields( "line" ), function );

        // group the Tuple stream by the "word" value
        assembly = new GroupBy( assembly, new Fields( "source" ) );

        // For every Tuple group
        // count the number of occurrences of "word" and store result in
        // a field named "count"
        Aggregator count = new Count( new Fields( "count" ) );
        assembly = new Every( assembly, count );

        // initialize app properties, tell Hadoop which jar file to use
        Properties properties = new Properties();
        FlowConnector.setApplicationJarClass( properties, WordCountCascading.class );

        // plan a new Flow from the assembly using the source and sink Taps
        // with the above properties
        FlowConnector flowConnector = new FlowConnector( properties );
        Flow flow = flowConnector.connect( "word-count", source, sink, assembly );


        // execute the flow, block until complete
        flow.complete();    
        */    
    }