Rapid Data AppDev with Windmill (on K3s) and Serverless Aurora
Over the summer, I went down the workflow rabbit hole (starting with what I knew like Apache Airflow and Dagster) but ended up deciding on Windmill.dev as the tool of choice.
It was a compromise between pure code options vs. no code for orchestrating random scripts. What were the factors?
- Open Source and good enough feature set in the free version.
- Support for Bash, Python, Typescript, Golang, and more (Ansible has subsequently been added)
- Solid RBAC
- Written in Rust for high performance
- Async task execution triggered by webhooks
- Built in Secrets Management, often through variables and resources.
- AI-codegen (you enter your OpenAI API key when you deploy the instance) and Vim editor support (just kidding!)
- A CLI that allows you do local code development and execution.
- Easy Python dependency management, even though your jobs execute within containers, just import the libraries and they are automatically installed
Self-Hosting Windmill on K3s
Windmill is super easy to deploy with a reliable helm chart that I've used on both EKS and K3s (pictured below)
I expose the web UI with Tailscale, but like most Helm resources you can use it to deploy a load-balancer and then terminate with an ACM cert.
apiVersion: v1
kind: Service
metadata:
annotations:
meta.helm.sh/release-name: rogwindmill
meta.helm.sh/release-namespace: windmill
tailscale.com/expose: "true"
creationTimestamp: "2024-11-13T23:38:20Z"
finalizers:
- tailscale.com/finalizer
labels:
app.kubernetes.io/managed-by: Helm
name: windmill-app
namespace: windmill
resourceVersion: "8843"
uid: 0a5116b1-d48f-42d8-bc40-fd65d842350b
Creating Aurora Instance
This was the first time I've used RDS Serverless Aurora
I used Gemini to create a sample dataset
Which I entered through the RDS Query editor
Next, you add the resource, entering the credentials retrieved from AWS Secrets manager
Builders Build!
Postgres is a "script type" just like Python, Bash, or other languages and it allows you to view the data as a script.
A simple listing of the Data
When you run it you'll see the output in JSON or Table format
Next build the app and use the use the Database Studio widget
When you run the "App" you'll get the CRUD editor
That allows you to insert (and edit) data.
References
mdfranz
