The organisation of knowledge has been a fascination for me since I first met my partner of over a decade, who is a librarian. In fact, search and retrieval have become my core interest as a developer. Pons is a search engine that I created to handle the vast repository of information that is 'academia'. I will preface this to say that it isn't really a project that I expect to find many users, and currently exists as a curiosity rather than a feature complete product. Its aim is to retrieve 'bridges' between disciplines. It ranks according to the reach that a paper has. A sort of inverse Rao-Stirling score is attached to the data for this purpose.
Pons exists as a sort of Bataillan project, quite unneeded and conceived for the sole pleasure of embedding and navigating the entire base of human knowledge (or more humbly the 124M subset of papers that adequately suited the parameters!)
Technically it is a hybrid search engine over 124M academic papers. It combines dense vector retrieval (FAISS) with BM25 sparse retrieval (Tantivy), fused via Reciprocal Rank Fusion. The FAISS index is topic-sharded into 4,514 partitions with query routing via cosine similarity to topic centroids. This achieves p50 83ms, p95 110ms search latency at 9× throughput under 16 perfectly concurrent users. The embedding pipeline uses INT8-quantised ONNX models via Python port workers, supervised by an Elixir/OTP pool. This means no GPU dependency, CPU-efficient inference only. This is supported by O(1) lookups from a msgpack-compressed RocksDB metadata store. And this performance occurs on a 17EUR/m VPS. There are massive horizontal and vertical scaling possibilities due to the use of Elixir and the performance is really the tip of the iceberg (given this is funded from my own bank account with no desire to monetise it...)
To further detail the ranking metric it is specifically a normalised inverse Rao-Stirling diversity score. Standard Rao-Stirling measures cross-disciplinary citation breadth (how far a paper's references reach across fields.) I inverted and normalised it to produce a 0–1 score where higher values indicate papers that draw from more diverse disciplinary sources. The target of this was to turn the search engine on its head, and provide surprising results one might not expect as an engine of oblique research and brainstorming.
Embedding model selection was evaluated across MiniLM and Mixedbread variants using MTEB benchmarks. All data maintained by Pons is GDPR compliant as I removed all PII from the set. Essentially the results just return to OpenAlex rather than self-hosting them. However, in the service of creating something instead of drilling so deeply I never returned, dozens of plausible and available features were dropped. It is an underestimate to say that the OpenAlex dataset contains a frightening amount of untapped knowledge.
Unfortunately, it has fallen somewhat to the wayside in favour of Tesserarum. A search product that serves my needs very directly focusing on art-historical data. But I leave it active as a testament of how much data can actually be tamed, organised, and provided by a single machine, given that many apps are currently shipping you a full Chrome runtime to edit your text or code. It is rather directed as a bit of a middle finger to that kind of lazy, inattentive 'professional' engineering that I have seen too much of.