I've been using Apache Beam for many years to process big data, Apache Beam support lots of runtime under the hood, e.g Apache Flink, Apache Spark, Google Cloud Dataflow, I'm using Apache Flink as beam runtime , I'm also using kubernetes to create and manage flink cluster in kubernetes cluster ,this is running very well, but from Beam to Flink there is a translation phase to translate Beam API to Flink executable code, the overhead is minimum, but I'm always curious about how to do it directly via Flink API directly. the other reason why I want to try Flink directly is that Beam support Java/Go/Python but it does not support Scala out of box, on the other hand Flink support both Java and Scala out of box, I also want to play with Scala, this is why the Flinker project is created.

flink

Flink have a template to bootstrap sample flink project out of box, we could use maven archetype to generate it easily.

mvn archetype:generate \
    -DarchetypeGroupId=org.apache.flink \
    -DarchetypeArtifactId=flink-quickstart-scala \
    -DarchetypeVersion=1.9.1 \
    -DgroupId=me.vipmind.flinker \
    -DartifactId=flinker-scala \
    -Dversion=1.0-SNAPSHOT \
    -Dpackage=me.vipmind.flinker \
    -DinteractiveMode=false

This command will generate a project named flinker-scala it will use maven to build the project, by default the archetype plugin will generate flink project based on scala 2.11, I'm more prefer to use scala 2.12, so I change could just change the version in the pom.xml from 2.11 to 2.12 with this commit. to import the scala project into IntelliJ Idea, we could import the maven project as normal, but we need to enable maven profile add-dependencies-for-IDEA, so that IDEA could find Flink dependencies when comiple code.

Flink also could generate java based project by the command below for flinker-java project

mvn archetype:generate \
    -DarchetypeGroupId=org.apache.flink \
    -DarchetypeArtifactId=flink-quickstart-java \
    -DarchetypeVersion=1.9.1 \
    -DgroupId=me.vipmind.flinker \
    -DartifactId=flinker-java \
    -Dversion=1.0-SNAPSHOT \
    -Dpackage=me.vipmind.flinker \
    -DinteractiveMode=false

I'm going to use flinker-scala as a base project to evaluate all the Flink features via Scala API in the following months.