Integrating with GitHub Actions
GitHub Actions is a popular CI/CD solution. In this guide you will learn how you can automatically deploy Flink jobs to Immerok Cloud using a custom workflow.
This guide uses the immerok/examples repository. To follow along, fork the repository and create a local checkout. You can also use your own Flink job as long as you have a build process that can generate a versioned JAR artifact.
In order to authenticate with Immerok Cloud, you will need an API access token. You can generate this using the
following command. Do note that the access tokens have a limited TTL (time-to-live), and you can use
to define a custom TTL.
_2# Keep this token safe!_2rok auth generate
As a best practice, head over to the repository settings of your repository and go to Security > Secrets > Actions.
Create a new secret named
with the access token you have generated above as the value.
Creating a workflow
To use GitHub Actions, create a file at
.github/workflows/deploy-window-aggregation.yaml relative to the repository
root with the following content:
This workflow installs the
rok CLI using the immerok/setup-rok Action, and
authenticates using the secret you created earlier.
This example uses the most recent version of the
rok CLI. For a production setup we suggest pinning the version
to ensure consistent runs, and periodically updating your workflow to the latest version.
You can now commit and push these changes. Open the "Actions" tab on GitHub and verify that the workflow is running and finishes successfully. Finally, verify that the authentication has worked by making sure that the output of the "Test CLI" looks like this:
_2NAME AGE PHASE_2default 1d Active
Deploying the Job
In order to deploy to Immerok Cloud, you need to define what the Job resource should look like. Create a new folder
immerok-cloud/window-aggregation in the repository root, and add a
window-aggregation-0.1.yaml which defines
the template for the Job.
Now that you have a workflow with working access to Immerok Cloud, it is time to build, package, and deploy your Flink job. Start by changing the workflow to only run on tags and removing the "Test CLI" step (you will no longer need it). Instead, add the following steps:
The first two of the three new steps set up a JDK and then build and package your Flink application. The final step then creates an Artifact for the produced JAR file, and finally applies the folder containing the Job resource definition.
You are now ready to commit and push these changes again. For the workflow to run, you need to also create a tag this time. Do note that the tag must follow the naming pattern used in the workflow.
_2git tag -a window-aggregation/v0.1 -m""_2git push origin window-aggregation/v0.1
As before, you will see that the workflow is being executed and succeeds. But this time, the workflow will have actually
deployed the Flink job to Immerok Cloud. Use the following command to verify that the Job has been created, and that it
eventually transitions into the
Running state (this may take a minute):
_4$ rok get jobs_4_4NAME AGE ZONE PHASE FLINK JOB STATE FLINK VERSION_4window-aggregation-0.1 1m shared-aws-eu-west-1 Ready Running 1.16
Upgrading the Job
While you have deployed the Flink job to Immerok Cloud now (🎉), the workflow will fail if run again. This is
because the resources already exist, but Artifacts and Jobs are (mostly) immutable. Say you want to release a new
version of the Flink job, how can this be done?
In the future, we will provide higher-level, mutable APIs that can be updated. Immerok Cloud will then take care of common upgrade scenarios like the one discussed here.
The idea is that you will create a new resource file for a new version (
window-aggregation-0.2.yaml). This file
will not only create a new Job, but also a final Savepoint for the existing one which it references as its initial
For this to work, the workflow must create the Artifact with the correct version automatically. You can extract the version from the tag which triggered the workflow. Make the following changes to implement this idea:
As you might have guessed, it is time to commit and push these changes again. Don't forget to create a new tag as well:
_2git tag -a window-aggregation/v0.2 -m""_2git push origin window-aggregation/v0.2
After verifying that the workflow has finished successfully, verify that the new Job has been created:
_5$ rok get jobs_5_5NAME AGE ZONE PHASE FLINK JOB STATE FLINK VERSION_5window-aggregation-0.1 5m shared-aws-eu-west-1 Completed Finished 1.16_5window-aggregation-0.2 1m shared-aws-eu-west-1 Ready Running 1.16
In this guide you have learned how to set up the
rok CLI in a GitHub Actions workflow, and how to create a custom
workflow that can deploy and upgrade a Flink job automatically.