The name gets me every single time. Always think it’s going to be about radio LoRa
papascrubs 1 days ago [-]
Not just me then. It's always the first thing that springs to mind.
apple4ever 1 days ago [-]
Nope not just you! Gets me everytime.
CaptainOfCoit 22 hours ago [-]
Microsofts inability to properly name things once again introduces more confusion than clarity, thanks Microsoft :)
At this point I think they do it on purpose, as their metrics for "people visiting the website/repository" or whatever gets increased as people thinking the repository is about the existing concept/technology.
dannyfritz07 1 days ago [-]
Dang it! Got me too! I've been wanting to hop into Meshtastic lately.
ijustlovemath 1 days ago [-]
Set up a node! Bare boards that work with the app are like $50 and take a few clicks to flash and setup. The basic antenna with no amp makes contacts up to 50mi away if the conditions are right. I have one in a window and one in a backpack at all times.
jacquesm 1 days ago [-]
It's insane how far you can go between hops, really most impressive. Where I live the mesh density is fairly high but I've also tried it in places where it was vanishingly low and yet I never completely lost contact. LoRa is very much an underappreciated technology.
wkjagt 13 hours ago [-]
I have a couple of nodes up, but not seeing a lot of traffic
halfmatthalfcat 1 days ago [-]
Same - sad it's not.
moffkalast 19 hours ago [-]
No such thing as LoRa and LoRaWAN without regret I'm afraid, all the range but no throughput.
halfmatthalfcat 16 hours ago [-]
You can do a lot with 255 bytes (SF5-8), just have to be creative :)
22 hours ago [-]
sifar 1 days ago [-]
And I thought you were going to say thinking machines :). Buy yeah LoRA trips me up too.
mrandish 1 days ago [-]
Yeah, kinda disappointed it's just more AI stuff...
dvfjsdhgfv 23 hours ago [-]
By the way, some time ago when I checked there were two cool applications of LoRa: (1) a mesh, for (hopefully) truly decentralized and more difficult to disrupt communication, (2) a gateway, so that you could get data from your sensors in remote places via standard internet protocols.
Both are very cool, but I wonder if I missed something else?
canadiantim 1 days ago [-]
I thought it was Lora the CRTD implementation, but then realized that Loro
frostyel 1 days ago [-]
[dead]
kouteiheika 1 days ago [-]
> However, the literature is unclear on how well LoRA performs relative to FullFT.
> The paper describes the finding that LoRA and full fine-tuning, with equal performance on the fine-tuning task, can have solutions with very different generalization behaviors outside the fine-tuning task distribution. We found that LoRA and full fine-tuning yield models with significant differences spectral properties of their weight matrices: LoRA models often containing “intruder dimensions”, high-ranking singular vectors approximately orthogonal to the singular vectors of pre-trained weight matrices. The existence of intruder dimensions correlates with the fine-tuned model forgetting more of the pre-training distribution as well as forgetting more when trained on tasks sequentially in a continual learning setup.
I'm surprised they didn't cite this; it's a well known paper.
adhi01 1 days ago [-]
To say that the 'literature is clear on that' while citing a single paper, which has been rejected from ICLR, is a bit of an overstatement.
kouteiheika 24 hours ago [-]
> which has been rejected from ICLR
Oh, you mean rejected just like these papers?
Efficient Estimation of Word Representations in Vector Space[1], one of the most influential papers in the space with tens of thousands of citations[2]? Or the RoBERTa[3] paper (dramatically improved upon BERT; RoBERTa and derived models currently have tens of millions of downloads on HF and still serve as a reliable industry workhorse)? Or the Mamba paper[4] (pretty much the only alternative to transformers that actually gets used)? Do you want me to keep going?
Honestly, I find that whether a paper gets rejected or not means diddly squat considering how broken the review system is, and through how much honestly terrible papers I have to wade through every time I'm looking through the conference submissions for anything good.
Even that paper itself does not provide any "clear" conclusions about which method is better.
muragekibicho 1 days ago [-]
Thanks for this comment.
lelanthran 1 days ago [-]
> I'm surprised they didn't cite this; it's a well known paper.
I'm surprised you copied and pasted all of that without explaining what it means.
Does LoRA perform worse, better or statistically insignificantly different to FullFT?
You aren't able to tell from what you pasted, are you?
cheald 15 hours ago [-]
Standard LoRA (W_delta = B@A with standard inits) generally underperforms FT, primarily because of "intruder dimensions" (new high-ranking singular vectors which misalign with the singular vectors of the underlying weights) as outlined in the paper.
There are techniques like PiCa and SVFT which can mitigate much of the loss, though.
tangjurine 8 hours ago [-]
pica came out two days ago, how did you find out about it?
I don't recall how I found out about it, but it was either paperswithcode or an LLM research session working through the intruder dimensions problem.
In my Stable Diffusion tests, it substantially improves LoRA training speed and fidelity, though I've got some experiments that seem to even further substantially improve on it by adding learnable rotations of the singular vectors.
crimsoneer 18 hours ago [-]
If you're going to be snarky, could you at least clarify what the answer is for those of us who don't stay on top of ML research...?
lelanthran 15 hours ago [-]
> If you're going to be snarky, could you at least clarify what the answer is for those of us who don't stay on top of ML research...?
The answer is "There's a difference, perhaps", but the GP appeared to imply that LoRA performed worse.
My understanding is that that paper found differences, but did not conclude that the differences were quantifiably better or worse, but this is not what GP's post implied.
p1esk 17 hours ago [-]
The paper does not make any clear conclusions about LoRA vs FullFT performance, beyond "the two methods seem to be learning different things".
richardvsu 18 hours ago [-]
Why would they cite a paper that’s not helping with their Tinker API that was released soon after? :)
mountainriver 1 days ago [-]
> LoRA works well when not capacity constrained, i.e., the number of trainable parameters exceeds the amount of information to be learned, which can be estimated in terms of dataset size
I’m shocked they didn’t look at progressive merging of LoRAs. Research shows that’s the best way of improving its ability to model higher level features.
Seems like a massive miss, not to mention there is other research that contradicts a lot of their findings. This feels a bit like a researchers first pass at learning LoRA
let_tim_cook_ 18 hours ago [-]
I'm not sure why progressive LoRa merging needs to be addressed here. They show there is a regime of problem where LoRa performs equivalently to FFT.
Progressive merging of LoRa is somewhere inbetween and categorically more complex than just LoRa so would be dominated by standard LoRa in that case.
While progressive merging could train faster as fewer params are trainable at any given time, it results in very larger adapter diffs OTO the size of the original model and doesn't retain the benefits of being able to deploy multiple adapters over the same base model idt.
Question for dudes building modern nn's... what's the thinking on estimating structural capacity for real world problem? How should I estimate how many parameters to choose for the model?
p1esk 17 hours ago [-]
You test different models on your real world problem, and pick the smallest one that works.
sgt101 14 hours ago [-]
I just think that there has to be some heuristic..
BoorishBears 12 hours ago [-]
Closest thing to a heuristic is trying the task with non fine-tuned models and building an intuition for how far off each model is, what directions it's off in, and how easily you can improve that direction via fine-tuning.
For example, for classification, if is hallucinating semantically similar, but not technically valid classes, you can probably fine-tune your way out of the gap with a smaller model.
But if your task requires world knowledge, you likely need a larger model. It's not cheap, efficient, or generally useful to fine-tune for additional world knowledge directly.
I really want something like this with flip out keyboard and could do Signal on LTE/WiFi.
markisus 1 days ago [-]
Can someone explain the bit counting argument in the reinforcement learning part?
I don’t get why a trajectory would provide only one bit of information.
Each step of the trajectory is at least giving information about what state transitions are possible.
An infinitely long trajectory can explore the whole state space if there are no absorbing states. Such a trajectory would provide a massive amount of information about the system, even if we ignored the final reward.
navar 21 hours ago [-]
I believe it's because the way you measure things in RL, each episode only tells you whether it was good (say reward +1) or bad (say 0 or negative reward), it does not tell you anything about the trace that was produced to get the outcome. This reward is the only thing measured to produce your gradients. Hence why the amount of info in it is O(1).
This is in contrast to more "supervised" forms of learning where you could get a loss for each token produced (e.g. cross entropy loss), and where you'd get, as a consequence O(number of tokens) information into your gradients.
20 hours ago [-]
mountainriver 1 days ago [-]
A fair amount of research has shown that RL doesn’t add knowledge to the base model it just optimizes paths that already exist.
Now ProRL from Nvidia showed there are ways of adding knowledge, mostly through progressive merging.
I’m still not fully convinced of the 1bit claim, they made other mistakes in the blog post
rco8786 22 hours ago [-]
I've been curious about LoRA and find a lot of these articles interesting. But I've been unable to find a good "LoRA for idiots" kind of starting point that gets me started actually doing some training with my data. Anybody know of a more practical guide I could use for that?
Be sure to validate everything you're reading though as of late I've come across more and more things that don't seem 100% accurate in their docs, seems to heavily depend on what section.
ijk 20 hours ago [-]
My sense is they need to go back and update previous docs; they release a lot of software updates and a lot of notebooks showing how to use the features, but the two might fall out of sync. Would that match your observations?
Yenrabbit 2 days ago [-]
Thinking Machines have put out a string of incredibly high-quality posts lately. Hard to oversell how much cred it's buying them with the AI research community! Keep up the great work folks
mijoharas 1 days ago [-]
What else has there been. I've only seen this one (which is great!)
joloooo 1 days ago [-]
Their Defeating Nondeterminism in LLM Inference was interesting for me. Worth reading their others!
sudohalt 1 days ago [-]
[flagged]
dang 1 days ago [-]
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
The amount of people who mistook this for long range radio and were disappointed when it isnt about it is way too damn high. (This is including me)
ineedasername 1 days ago [-]
It might be useful to use this thread in a dataset to train a LoRa so that LLM agents can more easily disambiguate the great LoRa acronym collision of ‘25. No longer will future generations suffer the indignity of either/or/both confusions.
eagsalazar2 1 days ago [-]
stupid website hijackes cmd-back-arrow.
Rendered at 10:18:22 GMT+0000 (Coordinated Universal Time) with Vercel.
At this point I think they do it on purpose, as their metrics for "people visiting the website/repository" or whatever gets increased as people thinking the repository is about the existing concept/technology.
Both are very cool, but I wonder if I missed something else?
I think the literature is clear on that?
"LoRA vs Full Fine-tuning: An Illusion of Equivalence" -- https://arxiv.org/abs/2410.21228v1
Quoting from the conclusions:
> The paper describes the finding that LoRA and full fine-tuning, with equal performance on the fine-tuning task, can have solutions with very different generalization behaviors outside the fine-tuning task distribution. We found that LoRA and full fine-tuning yield models with significant differences spectral properties of their weight matrices: LoRA models often containing “intruder dimensions”, high-ranking singular vectors approximately orthogonal to the singular vectors of pre-trained weight matrices. The existence of intruder dimensions correlates with the fine-tuned model forgetting more of the pre-training distribution as well as forgetting more when trained on tasks sequentially in a continual learning setup.
I'm surprised they didn't cite this; it's a well known paper.
Oh, you mean rejected just like these papers?
Efficient Estimation of Word Representations in Vector Space[1], one of the most influential papers in the space with tens of thousands of citations[2]? Or the RoBERTa[3] paper (dramatically improved upon BERT; RoBERTa and derived models currently have tens of millions of downloads on HF and still serve as a reliable industry workhorse)? Or the Mamba paper[4] (pretty much the only alternative to transformers that actually gets used)? Do you want me to keep going?
Honestly, I find that whether a paper gets rejected or not means diddly squat considering how broken the review system is, and through how much honestly terrible papers I have to wade through every time I'm looking through the conference submissions for anything good.
[1] -- https://openreview.net/forum?id=idpCdOWtqXd60
[2] -- https://scholar.google.com/scholar?cites=7447715766504981253
[3] -- https://openreview.net/forum?id=SyxS0T4tvS
[4] -- https://openreview.net/forum?id=AL1fq05o7H
I'm surprised you copied and pasted all of that without explaining what it means.
Does LoRA perform worse, better or statistically insignificantly different to FullFT?
You aren't able to tell from what you pasted, are you?
There are techniques like PiCa and SVFT which can mitigate much of the loss, though.
I don't recall how I found out about it, but it was either paperswithcode or an LLM research session working through the intruder dimensions problem.
In my Stable Diffusion tests, it substantially improves LoRA training speed and fidelity, though I've got some experiments that seem to even further substantially improve on it by adding learnable rotations of the singular vectors.
The answer is "There's a difference, perhaps", but the GP appeared to imply that LoRA performed worse.
My understanding is that that paper found differences, but did not conclude that the differences were quantifiably better or worse, but this is not what GP's post implied.
I’m shocked they didn’t look at progressive merging of LoRAs. Research shows that’s the best way of improving its ability to model higher level features.
Seems like a massive miss, not to mention there is other research that contradicts a lot of their findings. This feels a bit like a researchers first pass at learning LoRA
Progressive merging of LoRa is somewhere inbetween and categorically more complex than just LoRa so would be dominated by standard LoRa in that case.
While progressive merging could train faster as fewer params are trainable at any given time, it results in very larger adapter diffs OTO the size of the original model and doesn't retain the benefits of being able to deploy multiple adapters over the same base model idt.
https://arxiv.org/abs/2410.22911
https://arxiv.org/abs/2409.16167
For example, for classification, if is hallucinating semantically similar, but not technically valid classes, you can probably fine-tune your way out of the gap with a smaller model.
But if your task requires world knowledge, you likely need a larger model. It's not cheap, efficient, or generally useful to fine-tune for additional world knowledge directly.
Stumbled on this today... https://hackerpager.net/
I really want something like this with flip out keyboard and could do Signal on LTE/WiFi.
I don’t get why a trajectory would provide only one bit of information.
Each step of the trajectory is at least giving information about what state transitions are possible.
An infinitely long trajectory can explore the whole state space if there are no absorbing states. Such a trajectory would provide a massive amount of information about the system, even if we ignored the final reward.
This is in contrast to more "supervised" forms of learning where you could get a loss for each token produced (e.g. cross entropy loss), and where you'd get, as a consequence O(number of tokens) information into your gradients.
I’m still not fully convinced of the 1bit claim, they made other mistakes in the blog post
Be sure to validate everything you're reading though as of late I've come across more and more things that don't seem 100% accurate in their docs, seems to heavily depend on what section.
https://news.ycombinator.com/newsguidelines.html