I just did a spot check, I think searchthearxiv search results are superior.
0101111101 9 hours ago [-]
Looks cool!
You can input either a search query or a paper URL on arxiv xplorer. You can even combine paper URLs to search for combinations of ideas by putting + or - before the URL, like `+ 2501.12948 + 1712.01815`
masterjack 8 hours ago [-]
There’s also the search and browsing on https://sugaku.net, it’s more focused on math but does also have all of the arxiv on it
nblgbg 7 hours ago [-]
Just curious, are there any techniques other than using embeddings, computing cosine similarity, and sorting the results based on that? RRF could be used but again its very simple as well.
forrestp 1 hours ago [-]
My understanding is that your levers are roughly better / more diverse embeddings or computing more embeddings (embed chunks / groups / etc) + aggregating more cosine similarities / scores. More flops = better search w/ steep diminishing returns
Colbert being a good google-able application of utilizing more embeddings.
Search ends up often being a funnel of techniques. Cheap and high recall for phase 1 and ratchet up the flops and precision in
subsequent passes on the previous result set.
elliotec 10 hours ago [-]
This is really cool, and very relevant to something I'm working on. Would you be willing to do a quick explanation of the build?
0101111101 9 hours ago [-]
Sure! I first used openai embeddings on all the paper titles, abstracts and authors. When a user submits a search query, I embed the query, find the closest matching papers and return those results. Nothing too fancy involved!
That method can break when author names and subject matter collide.
heisenburgzero 6 hours ago [-]
So did you just combine Title+Abstracts+Authors into a single chunk and embed them or embedded them individually?
synctext 4 hours ago [-]
Impressive!
Will you parse the papers in the future? Without citations this is not that usable for professors or scientists in general. The relevance ranking largely depends on showing these older, prominent papers.
(from our lab experience building decentralised search using transformers)
elliotec 3 hours ago [-]
Thank you!!
madars 10 hours ago [-]
Looks great! Could you add eprint.iacr.org (Cryptology ePrint Archive)?
medrxiv was very useful for keeping the various COVID-19 related preprints from completely swamping biorxiv, especially once biorxiv started aggressively rejecting them.
0101111101 9 hours ago [-]
Sadly I couldn't find a public API for chemrxiv, but would be happy to be proven wrong!
https://news.ycombinator.com/item?id=42519487
I just did a spot check, I think searchthearxiv search results are superior.
Colbert being a good google-able application of utilizing more embeddings.
Search ends up often being a funnel of techniques. Cheap and high recall for phase 1 and ratchet up the flops and precision in subsequent passes on the previous result set.
I'm also maintaining a dataset of all the embeddings on kaggle if you want to use them yourself: https://www.kaggle.com/datasets/tomtum/openai-arxiv-embeddin...
Don't forget chemrXiv!
https://chemrxiv.org/engage/chemrxiv/public-api/documentatio...