https://kotlinlang.org logo
Join Slack
Powered by
I came up with a new idea for not having to bundle...
# github-workflows-kt
p
I came up with a new idea for not having to bundle action bindings with the lib. Client-side binding generation turned out to be cumbersome: it's hard to set up and work with, and there are some bugs in Kotlin and IntelliJ (e.g. poor support for 
@file:Import
) that makes working with these even more painful. An alternative idea, in some ways similar what @Vampire you proposed some time ago: have a separate Maven package for each action, but I'd like to host the packages in a way that allows me to generate the bindings on the server side, once the user requests them. A bit like https://jitpack.io/, but specialized to cover just the action bindings. The expected UX would be like:
Copy code
// foobar.main.kts
/// ...
@file:Repository("<https://bindings.typesafegithub.io|https://bindings.typesafegithub.io>")
@file:DependsOn("actions:checkout:v4")
// ... more dependencies like:
@file:DependsOn("action-owner:action-name:action-version")

import actions.Checkout

// ...
uses(action = Checkout(...))
// ...
The user would have to create no extra files like 
_used_actions.yaml
, no extra workflows like 
generate-action-bindings.main.kts
and call it either locally or in GitHub workflows. If done right, dependency updating bots like Renovate would work as well, since the 
maven-metadata.xml
would be generated according to what versions a given action publishes. For this to work kinda like JitPack, we'd have to have our own live endpoint that can react on appropriate GET requests, and generate the bindings on the fly. I'm also thinking about a variation of this idea, to not have to worry about paying for a server, just to start with something: host the pre-built packages somewhere on GitHub. If someone needs a binding for an action that is not yet supported, they would create a GitHub issue in a well-defined place and format that would later be picked up by a GitHub workflow, and it would generate and commit desired bindings. A separate workflow would periodically regenerate the bindings to catch manifest changes. Here's a little working PoC where I manually built the package and committed it to GitHub here, and show below how to consume it:
Copy code
#!/usr/bin/env kotlin
// If another repo than Maven Central is specified, we need to specify Maven Central explicitly.
@file:Repository("<https://repo1.maven.org/maven2|https://repo1.maven.org/maven2>")
@file:DependsOn("io.github.typesafegithub:github-workflows-kt:1.12.0")

// Dependency on a repo which contains Maven packages with bindings for various action.
@file:Repository("<https://raw.githubusercontent.com/typesafegithub/github-workflows-kt/poc-host-precompiled-bindings/bindings/|https://raw.githubusercontent.com/typesafegithub/github-workflows-kt/poc-host-precompiled-bindings/bindings/>")
@file:DependsOn("actions:checkout:v4")

import actions.Checkout

Checkout(
    fetchDepth = Checkout.FetchDepth.Value(10),
)
Let me know your thoughts! Any risks you see etc.
👀 1
@LeoColman also curious about what you think since you use client-side generation in Petals. I imagine you'd welcome such approach
v
While I highly dislike JitPack for anything but a quick test, I think here the approach might nicely fit and much better than the client-side generation that I never got warm with, even though I don't remember having suggested something like that. :-) There are ways to have free hosting for ad-hoc generation. Granted, most are shitty, as you get what you pay for. But there are exceptions. Unless the whole project gets very very popular very fast, the free tier of Google Cloud Platform or similar should be sufficient and work fine. This could always do the generation ad-hoc if it is fast and reliable, or maybe better trigger the GitHub workflow that generates it if not present and then deliver the cached result. Another option is the Project hosting SourceForge provides. Even though many consider SF dead, this by far is not the case, and the hosting it provides is very powerful and flexible. You can run anything on it that is supported and even have the generated files stored on disk. You cannot call out to other services anymore though, so you could not trigger a GitHub workflow from it.
p
Thanks for the feedback! I think I'll defer the way this is executed/hosted, because first I'd like to implement a module that takes generated binding code and outputs Maven artifacts. There are subproblems like calling the Kotlin compiler, generating a JAR and generating files like POM. I'd like to avoid using Gradle for all of this because most likely it would be too slow, and instead use some lower-level utils. Any thoughts here?
l
I'd totally be up for it. Not having files on my project would indeed streamline usage, and I agree it's cumbersome and hard to work with as is. This would give you a lot of work, but I think it makes sense if you're up to that 😂
p
It would also save me a lot of work in maintaining the library long-term :) So I'm pretty determined, now I just need to find some so called "spare time" :P
v
Any thoughts here?
I think where you might host it is very relevant. You might not have everything available everywhere. Generating the POM should be pretty trivial. You do not need all the fields set you need for Maven Central, and you do not need any dependencies. So the POM is practically just containing the artifacts coordinates, so any simple string-to-file will do.
p
thanks for all the thoughts! let me track this effort in Serve action bindings as Maven artifacts, see the checklist at the top. I already learned how to create the JAR (probably the most difficult part, not counting where to host this stuff) and have a piece of messy code that does it.
I got the happy path working with a local server, and specifying 
@file:Repository("<http://localhost:8080/binding/>")
success baby @LeoColman is your offer still up? 🙂 if I productize such bindings server and wrap it in a Docker image, would you be able to host it until we find a better place, just to give folks an opportunity to play with it?
❤️ 1
l
The offer is definitely up! However I must point out that for artifact hosting I cannot guarantee HA. This can be dangerous if we have too many clients (wishful thinking? xD). It would make thousands of pipelines download the artifact from the server, which could lead to a DDOS and ultimately make pipelines start to fail in a dependency-chain fashion (and annoy me for a few moments)
If it's for prototyping or for early hosting then absolutely count me in 🙂
🙌 1
p
Sounds good 🙂 definitely high availability is not something we should worry at such early stage, my goal is to present the general look-and-feel of the new solution, and gather early feedback. To protect you in some way here, we could have proper configuration like spillover on the load balancer (or equivalent) to fail fast. Some extra questions: 1. Could you share more info about your hosting, like: is it a single machine/VM that is always up, or something serverless? Or does it run on some major cloud provider, or not necessarily? What kind of non-volatile storage we can use there (e.g. for cache)? 2. Is it possible to publish and browse metrics on your hosting? It would be good to see how popular the service is. At least the number of requests to defined endpoints would be good to count, and nice to have: be able to publish extra metrics from the business logic. 3. Would it be possible to have automated deployment, so that we can set up a GitHub workflow to push the Docker image (or equivalent) to your hosting, and not have to bother you each time an update is needed?
@Vampire regarding SourceForge hosting, can you share a reference? is it this one? I'm a bit hesitant to go with GCP/AWS free tier because there's no 100% bulletproof way of ensuring that I won't exceed the free tier 😄 I just don't have to hook my credit card there just yet. I'd very much prefer a solution where I pay $X up front for a box + domain
l
@Piotr Krzemiński 1. It's a GTHost machine in Montreal. 
Intel Xeon D-1531 6c/12t 2.20-2.70GHz 1 x 16GB Micron DDR4 UDIMM 2133MHz ECC VLP 1 x 480GB SSD Intel S3510 300Mbit/s Unmetered
. It's a dedicated machine permanently up. We can allocate docker volumes to store data without any problems. On it I only have a Portainer to host multiple apps, most from LeoColman/MyStack 2. I think from the host itself no, not possible. Maybe CPU/Network data, but that would be from multiple apps. We can add metrics to the service and expose them, co-hosting with docker-compose some solution to grab the metrics. 3. Absolutely. I have Watchtower updating containers using 
latest
, pulling from DockerHub. It runs once every 24h, but we can make it shorter or even plug to some Portainer Webhook if that's possible
👍 1
Specifically, this is how I have the setup for our previous HTTP operation: https://github.com/LeoColman/MyStack/blob/main/github-workflows-kt/docker-compose.yml
Copy code
version: "3.8"

services:
  github-workflows-kts:
    image: krzema12/github-workflows-kt-script-generator-server:0.41.0
    networks:
      - reverse_proxy
    labels:
      caddy: github-workflows-kt.colman.com.br
      caddy.reverse_proxy: "{{upstreams 8080}}"

networks:
  reverse_proxy:
    external: true
We could tweak and squeeze it however we want. Adding volumes, extra services, etc
p
got it, sounds good, thanks! BTW, github-workflows-kt.colman.com.br seems to be down (it would generate code for a pretty old version of the lib, but the fact that this effort is somewhat abandoned is another problem 😄 ). Pointing it out because it might mean a potential problem with infra?
the next step is on me: dockerize a PoC version of the service
l
Ah, yeah, it's down. I've removed the service but never updated the repository 😅
👌 1
p
you can remove it in the repo as well, no one complained about it being down
👍 1
@LeoColman one more question: do you have a way of securely storing secrets on your hosting? I anticipate a feature where a GitHub token will be needed
l
Yes, it's possible to define environment secrets and password files
👍 1
v
@Vampire regarding SourceForge hosting, can you share a reference? is it this one?
Yes
I'm a bit hesitant to go with GCP/AWS free tier because there's no 100% bulletproof way of ensuring that I won't exceed the free tier 😄
If it exceeds the free tier, it should just stop serving content, it should not auto-upgrade. The credit card information is just used to ensure you are an individual, not a robot and that you do not create multiple free accounts. But you will not have to pay unless you manually activate it as far as I know.
I just don't have to hook my credit card there just yet. I'd very much prefer a solution where I pay $X up front for a box + domain
Well, you can of course do that, those options were just to get a free starter. You can anytime rent some VPS from Strato, or Hetzner, or any other VPS provider, or a root server if VPS is not powerfull enough anymore.
You could for example make a simple PHP script that checks whether the precompiled artifact is already there, and if not call a Kotlin executable that generates it and saves it on disk before then serving it.
I did a quick test and uploaded Kotlin and Java to the user web, and with this simple PHP script:
Copy code
<?
$result = `JAVA_HOME=/home/user-web/vampire0/jdk-21.0.2+13/ /home/user-web/vampire0/kotlinc/bin/kotlin -version`;
echo"
<HTML>
  <HEAD>
    <TITLE>Execute Script :-D</TITLE>
  </HEAD>
  <BODY>
";
  echo"
    result:   \"$result\"<BR>
";
echo"
  </BODY>
</HTML>
";
?>
you get this result: https://vampire0.users.sourceforge.net/kotlin-version.php
It might need a bit longer to generate a binding as the Kotlin and JVM process needs to be started first, than when a ktor server is running, but you get it for free and it should work just fine.
p
I'm allergic to PHP, I'd rather pay 😛 but thanks for the idea. I'll also double-check how the free tier in GCP works, I hope what you write is true
v
Then any other language that is supported and works. You can probably also directly start a Kotlin file, the overhead of starting a JVM on each call is most probably to high. The PHP or Perl or whatever would just be a very thin wrapper like "if file does not exist, run Kotlin to create the file (or Gradle or whatever else you like), then serve the file. This wrapper should be thin enough to not trigger your allergy to much.
That's the point of my example that shows that you can push Kotlin and Java there and execute it on request
p
sorry, I've just noticed you're showing an example application of the SourceForge hosting 🙈 makes sense, thanks! it's my option #2, let's now try Leonardo's hosting
v
You could also just serve the files statically without any dynamic interpreter and just have one URL where you can manually trigger a binding to be generated. You can probably even build some nice UI with Kotlin for that. But then of course you do not have the "it just works" experience.
👍 1
p
@LeoColman here's the Docker image, could you try to deploy it: https://hub.docker.com/repository/docker/krzema12/github-workflows-kt-jit-binding-server/general I'm a bit worried about the architecture, I built it on Apple Silicon (ARM) which in theory shouldn't work with your Intel server, but let me know
as for the domain, I think 
<http://github-workflows-kt-bindings.colman.com.br|github-workflows-kt-bindings.colman.com.br>
fits nicely
l
Up and running 😎
in theory shouldn't work with your Intel server
Docker Magic probably solves that
p
thanks! 🚀 > I have Watchtower updating containers using 
latest
, pulling from DockerHub. It runs once every 24h, but we can make it shorter could you set it to e.g. 1 hour? or if you have a webhook that I can call to refresh the image, would be cool as well, although I can live with scheduled updates. I've already pushed a new version of the image that fixes a crucial problem
l
Updated manually now. Yes, let me lower it to 1h. I'll try to see if there's a way to say "Hey, for this image, check every 5 minutes" or a webhook
🙌 1
Done. Auto-update every 5 minutes
Also sent you a private link for the webhook
I'll try to find a way to provide you with runtime logs
p
Thanks a lot!
v
I'm allergic to PHP
SF project web also supports Perl, Python, Tcl, and Shell scritps for CGI. But actually, the SF project web is unsuitable for this. I have suppressed the fact, that the generator needs to get the types and so on. Due to past abuse any outbound connection from project web is forbidden, even e-mail sending does not work directly anymore.
😬 1
👌 1
p
3 Views
Open in Slack
PreviousNext
Powered by