Skip to main content

Integrating with GitHub Actions

GitHub Actions is a popular CI/CD solution. In this guide you will learn how you can automatically deploy Flink jobs to Immerok Cloud using a custom workflow.

Prerequisites

This guide uses the immerok/examples repository. To follow along, fork the repository and create a local checkout. You can also use your own Flink job as long as you have a build process that can generate a versioned JAR artifact.

In order to authenticate with Immerok Cloud, you will need an API access token. You can generate this using the following command. Do note that the access tokens have a limited TTL (time-to-live), and you can use --expires-at to define a custom TTL.


_2
# Keep this token safe!
_2
rok auth generate

As a best practice, head over to the repository settings of your repository and go to Security > Secrets > Actions. Create a new secret named ROK_ACCESS_TOKEN with the access token you have generated above as the value.

Creating a workflow

To use GitHub Actions, create a file at .github/workflows/deploy-window-aggregation.yaml relative to the repository root with the following content:

.github/workflows/deploy-window-aggregation.yaml

_21
name: Deploy window-aggregation to Immerok Cloud
_21
on:
_21
push:
_21
branches:
_21
- main
_21
jobs:
_21
deploy:
_21
runs-on: ubuntu-latest
_21
steps:
_21
- name: Checkout
_21
uses: actions/checkout@v3
_21
_21
- name: Install rok CLI
_21
uses: immerok/setup-rok@v1
_21
with:
_21
version: latest
_21
accessToken: "${{ secrets.ROK_ACCESS_TOKEN }}"
_21
_21
- name: Test CLI
_21
shell: bash
_21
run: rok get projects

This workflow installs the rok CLI using the immerok/setup-rok Action, and authenticates using the secret you created earlier.

Tip

This example uses the most recent version of the rok CLI. For a production setup we suggest pinning the version to ensure consistent runs, and periodically updating your workflow to the latest version.

You can now commit and push these changes. Open the "Actions" tab on GitHub and verify that the workflow is running and finishes successfully. Finally, verify that the authentication has worked by making sure that the output of the "Test CLI" looks like this:


_2
NAME AGE PHASE
_2
default 1d Active

Deploying the Job

In order to deploy to Immerok Cloud, you need to define what the Job resource should look like. Create a new folder named immerok-cloud/window-aggregation in the repository root, and add a window-aggregation-0.1.yaml which defines the template for the Job.

immerok-cloud/window-aggregation/window-aggregation-0.1.yaml

_12
apiVersion: core/v1alpha1
_12
kind: Job
_12
metadata:
_12
name: window-aggregation-0.1
_12
project: default
_12
zone: shared-aws-eu-west-1
_12
spec:
_12
artifactRef:
_12
name: window-aggregation-0.1
_12
flinkVersion: "1.16"
_12
resources:
_12
rpus: 2

Now that you have a workflow with working access to Immerok Cloud, it is time to build, package, and deploy your Flink job. Start by changing the workflow to only run on tags and removing the "Test CLI" step (you will no longer need it). Instead, add the following steps:

.github/workflows/deploy-window-aggregation.yaml

_33
name: Deploy window-aggregation to Immerok Cloud
_33
on:
_33
push:
_33
tags:
_33
- 'window-aggregation/v*.*'
_33
jobs:
_33
deploy:
_33
runs-on: ubuntu-latest
_33
steps:
_33
- name: Checkout
_33
uses: actions/checkout@v3
_33
_33
- name: Install rok CLI
_33
uses: immerok/setup-rok@v1
_33
with:
_33
version: latest
_33
accessToken: "${{ secrets.ROK_ACCESS_TOKEN }}"
_33
_33
- name: Set up JDK 11
_33
uses: actions/setup-java@v3
_33
with:
_33
java-version: "11"
_33
distribution: "adopt"
_33
_33
- name: Build Flink job
_33
working-directory: window-aggregation
_33
run: mvn package
_33
_33
- name: Deploy to Immerok Cloud
_33
run: |
_33
ARTIFACT_JAR="$(find window-aggregation/target/ -name "example-window-aggregation-*.jar")"
_33
rok create artifact "window-aggregation-0.1" -f "${ARTIFACT_JAR}"
_33
rok apply --recursive -f immerok-cloud/window-aggregation/

The first two of the three new steps set up a JDK and then build and package your Flink application. The final step then creates an Artifact for the produced JAR file, and finally applies the folder containing the Job resource definition.

You are now ready to commit and push these changes again. For the workflow to run, you need to also create a tag this time. Do note that the tag must follow the naming pattern used in the workflow.


_2
git tag -a window-aggregation/v0.1 -m""
_2
git push origin window-aggregation/v0.1

As before, you will see that the workflow is being executed and succeeds. But this time, the workflow will have actually deployed the Flink job to Immerok Cloud. Use the following command to verify that the Job has been created, and that it eventually transitions into the Running state (this may take a minute):


_4
$ rok get jobs
_4
_4
NAME AGE ZONE PHASE FLINK JOB STATE FLINK VERSION
_4
window-aggregation-0.1 1m shared-aws-eu-west-1 Ready Running 1.16

Upgrading the Job

While you have deployed the Flink job to Immerok Cloud now (🎉), the workflow will fail if run again. This is because the resources already exist, but Artifacts and Jobs are (mostly) immutable. Say you want to release a new 0.2 version of the Flink job, how can this be done?

Note

In the future, we will provide higher-level, mutable APIs that can be updated. Immerok Cloud will then take care of common upgrade scenarios like the one discussed here.

The idea is that you will create a new resource file for a new version (window-aggregation-0.2.yaml). This file will not only create a new Job, but also a final Savepoint for the existing one which it references as its initial state:

immerok-cloud/window-aggregation/window-aggregation-0.2.yaml

_27
apiVersion: core/v1alpha1
_27
kind: Savepoint
_27
metadata:
_27
name: window-aggregation-state-0.1
_27
project: default
_27
zone: shared-aws-eu-west-1
_27
spec:
_27
type: Final
_27
jobRef:
_27
name: window-aggregation-0.1
_27
finalOptions:
_27
shutdownMethod: Graceful
_27
---
_27
apiVersion: core/v1alpha1
_27
kind: Job
_27
metadata:
_27
name: window-aggregation-0.2
_27
project: default
_27
zone: shared-aws-eu-west-1
_27
spec:
_27
artifactRef:
_27
name: window-aggregation-0.2
_27
initialStateRef:
_27
name: window-aggregation-state-0.1
_27
flinkVersion: "1.16"
_27
resources:
_27
rpus: 2

For this to work, the workflow must create the Artifact with the correct version automatically. You can extract the version from the tag which triggered the workflow. Make the following changes to implement this idea:

.github/workflows/deploy-window-aggregation.yaml

_7
- name: Deploy to Immerok Cloud
_7
run: |
_7
ARTIFACT_JAR="$(find window-aggregation/target/ -name "example-window-aggregation-*.jar")"
_7
ARTIFACT_VERSION="${GITHUB_REF##*/v}"
_7
_7
rok create artifact "window-aggregation-${ARTIFACT_VERSION}" -f "${ARTIFACT_JAR}"
_7
rok apply --recursive -f immerok-cloud/window-aggregation/

As you might have guessed, it is time to commit and push these changes again. Don't forget to create a new tag as well:


_2
git tag -a window-aggregation/v0.2 -m""
_2
git push origin window-aggregation/v0.2

After verifying that the workflow has finished successfully, verify that the new Job has been created:


_5
$ rok get jobs
_5
_5
NAME AGE ZONE PHASE FLINK JOB STATE FLINK VERSION
_5
window-aggregation-0.1 5m shared-aws-eu-west-1 Completed Finished 1.16
_5
window-aggregation-0.2 1m shared-aws-eu-west-1 Ready Running 1.16

Conclusion

In this guide you have learned how to set up the rok CLI in a GitHub Actions workflow, and how to create a custom workflow that can deploy and upgrade a Flink job automatically.