Integrating with GitHub Actions

GitHub Actions is a popular CI/CD solution. In this guide you will learn how you can automatically deploy Flink jobs to Immerok Cloud using a custom workflow.

Prerequisites

This guide uses the immerok/examples repository. To follow along, fork the repository and create a local checkout. You can also use your own Flink job as long as you have a build process that can generate a versioned JAR artifact.

In order to authenticate with Immerok Cloud, you will need an API access token. You can generate this using the following command. Do note that the access tokens have a limited TTL (time-to-live), and you can use --expires-at to define a custom TTL.

# Keep this token safe!
rok auth generate

As a best practice, head over to the repository settings of your repository and go to Security > Secrets > Actions. Create a new secret named ROK_ACCESS_TOKEN with the access token you have generated above as the value.

Creating a workflow

To use GitHub Actions, create a file at .github/workflows/deploy-window-aggregation.yaml relative to the repository root with the following content:

.github/workflows/deploy-window-aggregation.yaml

name: Deploy window-aggregation to Immerok Cloud
on:
  push:
    branches:
      - main
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Install rok CLI
        uses: immerok/setup-rok@v1
        with:
          version: latest
          accessToken: "${{ secrets.ROK_ACCESS_TOKEN }}"
      - name: Test CLI
        shell: bash
        run: rok get projects

This workflow installs the rok CLI using the immerok/setup-rok Action, and authenticates using the secret you created earlier.

Tip

This example uses the most recent version of the rok CLI. For a production setup we suggest pinning the version to ensure consistent runs, and periodically updating your workflow to the latest version.

You can now commit and push these changes. Open the "Actions" tab on GitHub and verify that the workflow is running and finishes successfully. Finally, verify that the authentication has worked by making sure that the output of the "Test CLI" looks like this:

NAME      AGE   PHASE
default   1d    Active

Deploying the Job

In order to deploy to Immerok Cloud, you need to define what the Job resource should look like. Create a new folder named immerok-cloud/window-aggregation in the repository root, and add a window-aggregation-0.1.yaml which defines the template for the Job.

immerok-cloud/window-aggregation/window-aggregation-0.1.yaml

apiVersion: core/v1alpha1
kind: Job
metadata:
  name: window-aggregation-0.1
  project: default
zone: shared-aws-eu-west-1
spec:
  artifactRef:
    name: window-aggregation-0.1
  flinkVersion: "1.16"
  resources:
    rpus: 2

Now that you have a workflow with working access to Immerok Cloud, it is time to build, package, and deploy your Flink job. Start by changing the workflow to only run on tags and removing the "Test CLI" step (you will no longer need it). Instead, add the following steps:

.github/workflows/deploy-window-aggregation.yaml

name: Deploy window-aggregation to Immerok Cloud
on:
  push:
    tags:
      - 'window-aggregation/v*.*'
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Install rok CLI
        uses: immerok/setup-rok@v1
        with:
          version: latest
          accessToken: "${{ secrets.ROK_ACCESS_TOKEN }}"
          
      - name: Set up JDK 11
        uses: actions/setup-java@v3
        with:
          java-version: "11"
          distribution: "adopt"
      
      - name: Build Flink job
        working-directory: window-aggregation
        run: mvn package
      
      - name: Deploy to Immerok Cloud
        run: |
          ARTIFACT_JAR="$(find window-aggregation/target/ -name "example-window-aggregation-*.jar")"
          rok create artifact "window-aggregation-0.1" -f "${ARTIFACT_JAR}"
          rok apply --recursive -f immerok-cloud/window-aggregation/

The first two of the three new steps set up a JDK and then build and package your Flink application. The final step then creates an Artifact for the produced JAR file, and finally applies the folder containing the Job resource definition.

You are now ready to commit and push these changes again. For the workflow to run, you need to also create a tag this time. Do note that the tag must follow the naming pattern used in the workflow.

git tag -a window-aggregation/v0.1 -m""
git push origin window-aggregation/v0.1

As before, you will see that the workflow is being executed and succeeds. But this time, the workflow will have actually deployed the Flink job to Immerok Cloud. Use the following command to verify that the Job has been created, and that it eventually transitions into the Running state (this may take a minute):

$ rok get jobs
NAME                     AGE   ZONE                   PHASE   FLINK JOB STATE   FLINK VERSION
window-aggregation-0.1   1m    shared-aws-eu-west-1   Ready   Running           1.16

Upgrading the Job

While you have deployed the Flink job to Immerok Cloud now (🎉), the workflow will fail if run again. This is because the resources already exist, but Artifacts and Jobs are (mostly) immutable. Say you want to release a new 0.2 version of the Flink job, how can this be done?

Note

In the future, we will provide higher-level, mutable APIs that can be updated. Immerok Cloud will then take care of common upgrade scenarios like the one discussed here.

The idea is that you will create a new resource file for a new version (window-aggregation-0.2.yaml). This file will not only create a new Job, but also a final Savepoint for the existing one which it references as its initial state:

immerok-cloud/window-aggregation/window-aggregation-0.2.yaml

apiVersion: core/v1alpha1
kind: Savepoint
metadata:
  name: window-aggregation-state-0.1
  project: default
zone: shared-aws-eu-west-1
spec:
  type: Final
  jobRef:
    name: window-aggregation-0.1
  finalOptions:
    shutdownMethod: Graceful
---
apiVersion: core/v1alpha1
kind: Job
metadata:
  name: window-aggregation-0.2
  project: default
zone: shared-aws-eu-west-1
spec:
  artifactRef:
    name: window-aggregation-0.2
  initialStateRef:
    name: window-aggregation-state-0.1
  flinkVersion: "1.16"
  resources:
    rpus: 2

For this to work, the workflow must create the Artifact with the correct version automatically. You can extract the version from the tag which triggered the workflow. Make the following changes to implement this idea:

.github/workflows/deploy-window-aggregation.yaml

      - name: Deploy to Immerok Cloud
        run: |
          ARTIFACT_JAR="$(find window-aggregation/target/ -name "example-window-aggregation-*.jar")"
          ARTIFACT_VERSION="${GITHUB_REF##*/v}"
          rok create artifact "window-aggregation-${ARTIFACT_VERSION}" -f "${ARTIFACT_JAR}"       
          rok apply --recursive -f immerok-cloud/window-aggregation/

As you might have guessed, it is time to commit and push these changes again. Don't forget to create a new tag as well:

git tag -a window-aggregation/v0.2 -m""
git push origin window-aggregation/v0.2

After verifying that the workflow has finished successfully, verify that the new Job has been created:

$ rok get jobs
NAME                     AGE   ZONE                   PHASE       FLINK JOB STATE   FLINK VERSION
window-aggregation-0.1   5m    shared-aws-eu-west-1   Completed   Finished          1.16
window-aggregation-0.2   1m    shared-aws-eu-west-1   Ready       Running           1.16

Conclusion

In this guide you have learned how to set up the rok CLI in a GitHub Actions workflow, and how to create a custom workflow that can deploy and upgrade a Flink job automatically.

Integrating with GitHub Actions

Prerequisites​

Creating a workflow​

Deploying the Job​

Upgrading the Job​

Conclusion​

Prerequisites

Creating a workflow

Deploying the Job

Upgrading the Job

Conclusion