We have a similar use case. All Elixir code base, but need to use Python for ML libraries. We decided to use IPC. Elixir will spawn a process and communicate over stdio. https://github.com/akash-akya/ex_cmd makes it a breeze to stream stdin and stdout. This also has the added benefit of keeping the Python side completely stateless and keeping all the domain logic on the Elixir side. Spawning a process might be slower compared to enqueuing a job, but in our case the job usually takes long enough to make it irrelevant.
languagehacker 1 hours ago [-]
I feel like if you need to utilize a tool like this, odds are pretty good you may have picked the Wrong Tool For the Job, or, perhaps even worse, the wrong architecture.
This is why it's so important to do lots of engineering before writing the first line of code on a project. It helps keep you from choosing a tool set or architecture out of preference and keeps you honest about the capabilities you need and how your system should be organized.
victorbjorklund 40 minutes ago [-]
Strange opinion. Plenty of apps have more than one language. I might end up using this.
Why? Because my app is built in Elixir and right now I’m also using a python app that is open source but I really just need a small part of the python app. I don’t wanna rewrite everything in Elixir because while it’s small I expect it to change over time (basically fetching a lot of data sources) and it will be pain to keep rewriting it when data collections needs to change (over a 100 different sources). Right now I run the python app as an api but it’s just so overkill and harder to manage vs just handling everything except the actually data collection in Elixir where I am already using Oban.
Arubis 51 minutes ago [-]
It’s almost as though choosing a single-threaded, GIL-encumbered interpreted scripting language as the primary interface to an ecosystem of extremely parallelized and concurrent high-performance hardware-dependent operations wasn’t quite the right move for our industry.
geooff_ 50 minutes ago [-]
I disagree, using python for a web-server and something like celery for background work is a pretty common pattern.
My reading of this is it more or less allows you to use Postgres (which you're likely already using as your DB) for the task orchestration backend. And it comes with a cool UI.
languagehacker 30 minutes ago [-]
That's not the sort of architecture I'm referring to. I'm specifically talking about splitting your application layer between Elixir and Python.
whalesalad 1 hours ago [-]
What leads you to this conclusion
mrcwinn 16 minutes ago [-]
I absolutely love Elixir, but if this is the bridge you need to cross, just write it in Python in the first place.
cpursley 3 hours ago [-]
Very nice, Oban is great. I effectually found a similar approach with pgflow.dev (built around pgmq) - but the stateless deno "workers" are pretty unreliable and built an elixir worker (https://github.com/agoodway/pgflow) that can pick up and process jobs that were created by pgflow's supabase/typescript client. So maybe there's an opportunity also with Oban to have a TypeScript/Node client that can insert jobs that Elixir/Python Oban can pick up. Also, I wonder if another approach vs the python workers picking things up is to have elixir workers call/run python/lua, etc code or is that too limiting?
btw, a lot of postgres envs are not going to have pgmq, so just use Oban and don't reinvent the wheel like I did ;)
jongjong 35 minutes ago [-]
I don't see the point of Elixir now. LLMs work better with mainstream languages which make up a bigger portion of their training set.
I don't see the point of TypeScript either, I can make the LLM output JavaScript and the tokens saved not having to add types can be used to write additional tests...
The aesthetics or safety features of the languages no longer matter IMO. Succinctness, functionality and popularity of the language are now much more important factors.
Furthermore, it's actually kind of annoying that the LLMs are not better than us, and still benefit from having code properly typed, well-architected, and split into modules/files. I was lamenting this fact the other day; the only reason we moved away from Assembly and BASIC, using GOTOs in a single huge file was because us humans needed the organization to help us maintain context. Turns out, because of how they're trained, so do the LLMs.
So TypeScript types and tests actually do help a lot, simply because they're deterministic guardrails that the LLM can use to check its work and be steered to producing code that actually works.
cloud8421 18 minutes ago [-]
> I don't see the point of Elixir now. LLMs work better with mainstream languages which make up a bigger portion of their training set.
I can't say if it works better with other languages, but I can definitely say both Opus and Codex work really well with Elixir. I work on a fairly large application and they consistently produce well structured working code, and are able to review existing code to find issues that are very easy to miss.
The LLM needs guidance around general patterns, e.g. "Let's use a state machine to implement this functionality" but it writes code that uses language idioms, leverages immutability and concurrency, and generally speaking it's much better than any first pass that I would manually do.
I have my ethical concerns, but it would be foolish of me to state that it works poorly - if anything it makes me question my own abilities and focus in comparison (which is a whole different topic).
jakejohnson 9 minutes ago [-]
LLMs work great with Elixir. Running tsc in a loop while generating code still catches type errors introduced by an LLM and it’s faster than generating additional tests. Elixir is also succinct and highly functional. If you can’t find a specific library it’s easier than ever to build out the barebones functionality you need yourself or use NIFs, ports, etc.
This is why it's so important to do lots of engineering before writing the first line of code on a project. It helps keep you from choosing a tool set or architecture out of preference and keeps you honest about the capabilities you need and how your system should be organized.
Why? Because my app is built in Elixir and right now I’m also using a python app that is open source but I really just need a small part of the python app. I don’t wanna rewrite everything in Elixir because while it’s small I expect it to change over time (basically fetching a lot of data sources) and it will be pain to keep rewriting it when data collections needs to change (over a 100 different sources). Right now I run the python app as an api but it’s just so overkill and harder to manage vs just handling everything except the actually data collection in Elixir where I am already using Oban.
My reading of this is it more or less allows you to use Postgres (which you're likely already using as your DB) for the task orchestration backend. And it comes with a cool UI.
I don't see the point of TypeScript either, I can make the LLM output JavaScript and the tokens saved not having to add types can be used to write additional tests...
The aesthetics or safety features of the languages no longer matter IMO. Succinctness, functionality and popularity of the language are now much more important factors.
Furthermore, it's actually kind of annoying that the LLMs are not better than us, and still benefit from having code properly typed, well-architected, and split into modules/files. I was lamenting this fact the other day; the only reason we moved away from Assembly and BASIC, using GOTOs in a single huge file was because us humans needed the organization to help us maintain context. Turns out, because of how they're trained, so do the LLMs.
So TypeScript types and tests actually do help a lot, simply because they're deterministic guardrails that the LLM can use to check its work and be steered to producing code that actually works.
I can't say if it works better with other languages, but I can definitely say both Opus and Codex work really well with Elixir. I work on a fairly large application and they consistently produce well structured working code, and are able to review existing code to find issues that are very easy to miss.
The LLM needs guidance around general patterns, e.g. "Let's use a state machine to implement this functionality" but it writes code that uses language idioms, leverages immutability and concurrency, and generally speaking it's much better than any first pass that I would manually do.
I have my ethical concerns, but it would be foolish of me to state that it works poorly - if anything it makes me question my own abilities and focus in comparison (which is a whole different topic).
https://dashbit.co/blog/why-elixir-best-language-for-ai