I applaud the effort for the local (lo-fi) space ! Yet, reading over the example linked in the docs (which does not seem cheery-picked, kudos for that!), my impression is that the document is a rather messy outcome [1].
I think what's missing is one (or more) step in-between, possible a graph database (eg[2]), which the LLM can place all it's information in, see relevant interconnections, query to question itself, and then generate the final report.
(maybe the final report could be an interactive HTML file that the user can ask questions, or edit themselves).
There's also a similar open-deep research tool called onyx [2], with I think has better UI/UX albeit not local. Maybe the author could consider porting this to local instead of rolling and maintaining another deep-research tool themselves ?
I'm saying this, not because I think it's not a good project, but because there are a ton of open deep-research projects which I'm afraid will just fizzle out, and would be better if people would join forces working on those aspects they care most about (e.g. local aspect, or RAG strategies, etc) .
If you want to add embeddings over internet as a source, you should try out exa.ai. Includes: wikipedia, tens of thousands of news feeds, Github, 70M+ papers including all of arxiv, etc.
disclaimer: I am one of the founders (:
antonkar 3 hours ago [-]
I think the guy who’ll make the 3D game-like GUI for LLMs is the next Jobs/Gates/Musk and Nobel Prize Winner (I think it’ll solve alignment by having millions of eyes on the internals of LLMs), because computers became popular only after the OS with a GUI appeared, current chatbots are a bit like a command line in comparison. I just started ASK HN to let people and me share their AI safety ideas, both crazy and not: https://news.ycombinator.com/item?id=43332593
bravura 4 hours ago [-]
For web search, also consider the Kagi and Tavily APIs.
learningcircuit 3 hours ago [-]
Thank you I will add
throwaway24681 4 hours ago [-]
Looks very cool. How does this compare to the RAG features provided by open-webui?
There is web search and a way to embed documents, but so far it seems like the results are subpar as details are lost in embeddings. Is this much better?
learningcircuit 2 hours ago [-]
Give me a question and I can give you the output? So you can compare.
throwaway24681 48 minutes ago [-]
I tried it myself. It looks like this can do a lot more than open-webui's web search in terms of detail, which sounds useful, thanks for making it open source.
It seems to have a weird behavior of specifying a date when I didn't ask for it, is this expected? Also, I feel like searching "questions" is not optimal for most search engines, and it should instead search in terms of keywords.
Also, I wish there can be a more informative log at a slightly higher level - I don't need to see every request being made, but I do want to see a summary of what's happening at each step, like the prompt used, result, and the new search being done.
On another note, for local models, reasoning models have significant advantages over non-reasoning models. Can they be used for this?
You could be the first if you were to develop an eval (preferably automated with llm as judge) and compared local deep research with perplexity's, openai's and deepseek's implementations on high-information questions.
learningcircuit 8 hours ago [-]
How do they evaluate the quality of the report? It's one of the most important things for me.
mentalgear 4 hours ago [-]
Given a benchmark corpus, the evaluation criteria could be:
- Facts extracted: the amount of relevant facts extracted from the corpus
- Interpretations : based on the facts, % of correct interpretations made
- Correct Predictions: based on the above, % of correct extrapolations / interpolations / predictions made
The ground truth could be in JSON file per example.
(If the solution you want to benchmark uses a graph db, you could compare these aspects with a LLM as judge.)
---
The actual writing is more about formal/business/academic style, and I find less relevant for a benchmark.
However I would find it crucial to run a "reverse RAG" over the generated report to ensure each claim has a source. [0]
Is anyone using (local) LLMs to directly search for (by scanning over) relevant materials from a corpus rather than relying on vector search?
suprjami 6 hours ago [-]
Generally this fails.
Most LLMs lose the ability to track facts over about 20k words of content, the best can manage maybe 40k words.
Look for "needle" benchmark tests, as in needle-in-haystack.
Not to mention the memory requirements of such a huge context like 128k or 1M tokens. Only people with enterprise servers at home could run that locally.
learningcircuit 5 hours ago [-]
Very good answer. It is very hard with small LLM.
alchemist1e9 5 hours ago [-]
Nice work!
I’ve been thinking recently that a local collection of pre-processed for RAG using curated focused structured information might be a good complement to this dynamic searching approach.
I see this used LangChain, might be worth checking into txtai.
I think what's missing is one (or more) step in-between, possible a graph database (eg[2]), which the LLM can place all it's information in, see relevant interconnections, query to question itself, and then generate the final report.
(maybe the final report could be an interactive HTML file that the user can ask questions, or edit themselves).
There's also a similar open-deep research tool called onyx [2], with I think has better UI/UX albeit not local. Maybe the author could consider porting this to local instead of rolling and maintaining another deep-research tool themselves ?
I'm saying this, not because I think it's not a good project, but because there are a ton of open deep-research projects which I'm afraid will just fizzle out, and would be better if people would join forces working on those aspects they care most about (e.g. local aspect, or RAG strategies, etc) .
[1] https://github.com/LearningCircuit/local-deep-research/blob/...
[2] "In-Browser Graph RAG with Kuzu-WASM and WebLLM" https://news.ycombinator.com/item?id=43321523
[3] https://github.com/onyx-dot-app/onyx
If you want to add embeddings over internet as a source, you should try out exa.ai. Includes: wikipedia, tens of thousands of news feeds, Github, 70M+ papers including all of arxiv, etc.
disclaimer: I am one of the founders (:
There is web search and a way to embed documents, but so far it seems like the results are subpar as details are lost in embeddings. Is this much better?
It seems to have a weird behavior of specifying a date when I didn't ask for it, is this expected? Also, I feel like searching "questions" is not optimal for most search engines, and it should instead search in terms of keywords.
Also, I wish there can be a more informative log at a slightly higher level - I don't need to see every request being made, but I do want to see a summary of what's happening at each step, like the prompt used, result, and the new search being done.
On another note, for local models, reasoning models have significant advantages over non-reasoning models. Can they be used for this?
- Facts extracted: the amount of relevant facts extracted from the corpus
- Interpretations : based on the facts, % of correct interpretations made
- Correct Predictions: based on the above, % of correct extrapolations / interpolations / predictions made
The ground truth could be in JSON file per example. (If the solution you want to benchmark uses a graph db, you could compare these aspects with a LLM as judge.)
---
The actual writing is more about formal/business/academic style, and I find less relevant for a benchmark.
However I would find it crucial to run a "reverse RAG" over the generated report to ensure each claim has a source. [0]
[0] https://venturebeat.com/ai/mayo-clinic-secret-weapon-against...
Most LLMs lose the ability to track facts over about 20k words of content, the best can manage maybe 40k words.
Look for "needle" benchmark tests, as in needle-in-haystack.
Not to mention the memory requirements of such a huge context like 128k or 1M tokens. Only people with enterprise servers at home could run that locally.
I’ve been thinking recently that a local collection of pre-processed for RAG using curated focused structured information might be a good complement to this dynamic searching approach.
I see this used LangChain, might be worth checking into txtai.
https://neuml.github.io/txtai/examples/