Skip to main content

Jobs

Jobs represent Apache Flink jobs: the runtime representation of a Flink program.

job.yaml

_17
apiVersion: core/v1alpha1
_17
kind: Job
_17
zone: shared-aws-eu-west-1
_17
metadata:
_17
name: top-speed-windowing-v1
_17
project: default
_17
spec:
_17
flinkVersion: 1.16
_17
artifactRef:
_17
name: top-speed-windowing-v1
_17
jarOptions:
_17
entryClass: org.apache.flink.streaming.examples.windowing.TopSpeedwindowing
_17
mainArgs: ["--topic", "input-topic"]
_17
resources:
_17
rpus: 5
_17
configuration:
_17
execution.checkpointing.interval: 30s

Like Apache Flink jobs, a Job on Immerok Cloud can either be executed in execution mode STREAMING (a streaming job) or in execution mode BATCH (a batch job). At this point, only streaming jobs are supported, though.

Like Apache Flink jobs, Immerok Cloud Jobs are immutable except for resources, which can be changed during runtime to rescale your Job.

The Flink Version determines the major and minor version of Apache Flink for your Job.

Immerok Cloud will ensure that your Jobs always run on the most secure and robust version of Apache Flink that is compatible with the configured Flink version. Runtime Updates - in particular security updates and upgrades to the latest patch version of Apache Flink - happen automatically in the background for all your running Jobs.

Immerok Cloud currently only supports flinkVersion: 1.16.

Resources​

Rok Processing Units (RPU) are a horizontal unit of scaling for Jobs on Immerok Cloud. This means if you increase the resources of a Job from rpus: 2 instead of rpus: 1, the Job will receive twice as many resources as before. Provided your Flink program itself scales perfectly, the Job will be able to process twice as many records/s.

The number of RPUs are the primary dimension of billing on Immerok Cloud.

Implementation Detail

The number of RPUs currently corresponds to the number of "Slots", which corresponds to the parallelism of the Flink job unless you configure SlotSharingGroups.

Configuration​

The configuration section allows you to pass certain Flink configuration options to your Jobs on Immerok Cloud.

All of these configurations have sane defaults, so that you should not be required to set any of them unless you like to manually tune your Job. In general, Immerok Cloud tries to expose as few knobs as possible and as many as necessary.

Please see the configuration reference for a full list of all available configuration options.

Lifecycle and Status​

A Job is created in phase Pending. Once all prerequisites are fulfilled, e.g. the reference Artifact exists and is Ready, and the Flink REST API is available the Job transitions into Ready and stays in Ready until it is deleted.

job.yaml

_6
kind: Job
_6
apiVersion: core/v1alpha1
_6
...
_6
status:
_6
phase: Pending
_6
flinkJobState: ""

status.phase

flowchart TD; Pending-->Ready; Ready-->Deleting;

While the Job is in phase Ready, status.flinkJobState reflects the status of the underlying Flink job one-to-one. Due to how Flink jobs are deployed and configured on Immerok Cloud, the state diagram of a Flink job becomes slightly simpler compared to a generic Flink job. For example, a Job on Immerok Cloud never reaches the Failed state. Instead, Jobs are restarted indefinitely until they either reach Finished or Cancelled.

  • Finished is reached if the Job runs out of data or if it is stopped.
  • Cancelled is reached if the Job is cancelled, which corresponds to a deletion of the Job resource in Immerok Cloud.
job.yaml

_6
kind: Job
_6
apiVersion: core/v1alpha1
_6
...
_6
status:
_6
phase: Ready
_6
flinkJobState: Running

status.flinkJobState

flowchart TD; Initializing-->Created; Created-->Running; Running-->Failing; Running-->Rescaling; Failing-->Restarting; Rescaling-->Restarting; Restarting-->Running; Running-->Finished; Running-->Cancelled;