Project: kettle-beam (GitHub Link)

kettle-beam-master
- src
  - main
    - resources
      - beam-bq-output.svg
      - Trans-subclassing-idea
      - beam-kafka.svg
      - beam-window.svg
      - beam-input.svg
      - beam_transgraph_overlays.xul
      - beam-kafka-output.svg
      - beam-logo.svg
      - beam-timestamp.svg
      - beam-output.svg
      - beam_spoon_overlays.xul
      - beam-bq-input.svg
      - beam-gcp-pubsub.svg
      - beam-gcp-pubsub-subscribe.svg
      - beam_perspective.xul
      - beam-kafka-input.svg
      - beam-gcp-pubsub-publish.svg
    - java
      - org
        kettle
        beam
        metastore
        BeamJobConfig.java
        BeamJobConfigDialog.java
        messages
        messages_en_US.properties
        FileDefinition.java
        FileDefinitionDialog.java
        FieldDefinition.java
        JobParameter.java
        RunnerType.java
        xp
        BeamTrans.java
        BeamDummyTrans.java
        RunBeamTransExecutionPoint.java
        perspective
        BeamHelper.java
        BeamController.java
        BeamSpoonPlugin.java
        TransDrawExtensionPoint.java
        messages
        messages_en_US.properties
        BeamPerspective.java
        BeamMenuController.java
        util
        BeamConst.java
        steps
        kafka
        BeamConsumeMeta.java
        ConfigOption.java
        BeamConsumeDialog.java
        BeamProduceDialog.java
        BeamConsume.java
        BeamProduceMeta.java
        messages
        messages_en_US.properties
        BeamProduce.java
        pubsub
        BeamSubscribeMeta.java
        BeamPublishMeta.java
        BeamSubscribe.java
        BeamPublishData.java
        BeamSubscribeDialog.java
        BeamSubscribeData.java
        messages
        messages_en_US.properties
        BeamPublish.java
        BeamPublishDialog.java
        window
        BeamTimestampMeta.java
        BeamTimestampDialog.java
        messages
        messages_en_US.properties
        BeamWindowMeta.java
        BeamWindowDialog.java
        io
        BeamOutputMeta.java
        messages
        messages_en_US.properties
        BeamInputMeta.java
        BeamOutput.java
        BeamOutputData.java
        BeamOutputDialog.java
        BeamInputData.java
        BeamInputDialog.java
        BeamInput.java
        bq
        BeamBQOutputMeta.java
        BeamBQInputDialog.java
        messages
        messages_en_US.properties
        BeamBQOutputDialog.java
        BeamBQInputMeta.java
        BQField.java
        core
        metastore
        SerializableMetaStore.java
        BeamDefaults.java
        partition
        SinglePartitionFn.java
        fn
        BQSchemaAndRecordToKettleFn.java
        KettleToStringFn.java
        KVStringStringToKettleRowFn.java
        AssemblerFn.java
        PubsubMessageToKettleRowFn.java
        KettleKeyValueFn.java
        TimestampFn.java
        StringToKettleRowFn.java
        KettleToBQTableRowFn.java
        KettleRowToKVStringStringFn.java
        StringToKettleFn.java
        WindowInfoFn.java
        KVLongStringToKettleRowFn.java
        PublishStringsFn.java
        PublishMessagesFn.java
        GroupByFn.java
        util
        JsonRowMeta.java
        KettleBeamUtil.java
        Failure.java
        coder
        KettleRowCoder.java
        KettleRow.java
        shared
        SimpleVariableSpace.java
        VariableValue.java
        AggregationType.java
        transform
        BeamKafkaInputTransform.java
        BeamKafkaOutputTransform.java
        GroupByTransform.java
        BeamInputTransform.java
        BeamBQInputTransform.java
        BeamOutputTransform.java
        StepBatchTransform.java
        BeamPublishTransform.java
        StepTransform.java
        BeamBQOutputTransform.java
        BeamSubscribeTransform.java
        BeamKettle.java
        pipeline
        main
        MainBeam.java
        TransMetaPipelineConverter.java
        fatjar
        FatJarBuilder.java
        flink
        MainFlink.java
        spark
        MainSpark.java
        KettleBeamPipelineExecutor.java
        BeamMetricsUpdatedListener.java
        handler
        BeamBigQueryOutputStepHandler.java
        BeamKafkaInputStepHandler.java
        BeamStepHandler.java
        BeamBigQueryInputStepHandler.java
        BeamInputStepHandler.java
        BeamTimestampStepHandler.java
        BeamPublisherStepHandler.java
        BeamKafkaOutputStepHandler.java
        BeamMergeJoinStepHandler.java
        BeamGroupByStepHandler.java
        BeamGenericStepHandler.java
        BeamSubscriberStepHandler.java
        BeamBaseStepHandler.java
        BeamWindowStepHandler.java
        BeamOutputStepHandler.java
        carte
        MetricsRegistrationQueue.java
        RegisterBeamMetrics.java
        BeamMetricsEntry.java
  - test
    - resources
      - customers
        customers-100.txt
        state-data.txt
    - java
      - core
        metastore
        SerializableMetaStoreTest.java
        KettleRowTest.java
        coder
        KettleRowCoderTest.java
      - org
        kettle
        beam
        util
        BeamTransMetaUtil.java
        transform
        PipelineTestBase.java
        StreamLookupPipelineTest.java
        SwitchCasePipelineTest.java
        BasePipelineTest.java
        MergeJoinPipelineTest.java
        FilterPipelineTest.java
        GroupByPipelineTest.java
- pom.xml
- LICENSE
- README.md
- .idea
  - codeStyles
    - codeStyleConfig.xml
- flink-notes-matt.txt
- .gitignore
- spark-notes-matt.txt

kettle-beam

Kettle plugins for Apache Beam

First

build/install project kettle-beam-core

https://github.com/mattcasters/kettle-beam-core

Build

mvn clean install

Note you need the Pentaho settings.xml in your ~/.m2 : https://github.com/pentaho/maven-parent-poms/blob/master/maven-support-files/settings.xml

Install

Create a new directory called kettle-beam in \<PDI-DIR>plugins/
Copy target/kettle-beam-.jar to \<PDI-DIR>/plugins/kettle-beam/
Copy the other jar files in target/lib to \<PDI-DIR>/plugins/kettle-beam/lib/

Configure

File Definitions

Describe the file layout for the input and output of your pipeline using :

Spoon menu Beam / Create a file definition

Specify this file layout in your "Beam Input" and "Beam Output" steps. If you do not specify the file definition in the "Beam Output" step, all fields arriving at the step will be written with comma for separator and double quotes as enclosure. The formatting in the fields will be used.

Beam Job Configurations

A Beam Job configuration is needed to run your transformation on Apache Beam. Specify which Runner to use (Direct and Dataflow are supported).
You can use the variables to make your transformations completely generic. For example you can set an INPUT_LOCATION location variable

/some/folder/* for a Direct execution during testing
gs://mybucket/input/* for an execution on GCP Dataflow

Supported

Input: Beam Input, Google Pub/Sub Subscribe and Google BigQuery Input
Output: Beam Output, Google Pub/Sub Publish and Google BigQuery Output
Windowing with the Beam Window step and adding timestamps to bounded data for streaming (Beam Timestamp)
Sort rows is not yet supported and will never be supported in a generic sense like in Kettle.
Group By step : experimental, SUM (Integer, Number), COUNT, MIN, MAX, FIRST (throws errors for not-supported stuff)
Merge Join
Stream Lookup (side loading data)
Filter rows (including targeting steps for true/false)
Switch/Case
Plugin support through the Beam Job Configuration: specify which plugins to include in the runtime

Runners

Beam Direct : working
Google Cloud DataFlow : working
Apache Spark : mostly untested, configurable (feedback welcome)
Apache Flink : not started yet, stubbed out code
Aache Apex : not started yet, stubbed out code
JStorm : not started yet

More information

http://diethardsteiner.github.io/pdi/2018/12/01/Kettle-Beam.html