eXplorer web corner

On AI, LLM personal use, Khoj - a RANT

Well, time ago I've tested a new project (back than) Khoj, it was a 0.something release. I've suffered a bit, like most ML pythonic stuff, to deploy it and tested on my org-mode notes, org-roam managed inside of Emacs which is essentially my Desktop environment (thanks also to EXWM), where notes are most of my digital information and 99% of my meta-information, since I have anything in notes, from the OS (NixOS, with it's config tangle-ed from org-mode notes) to Emacs itself, my files as org-attachments etc. Long story short my note-base should be an easy candidate for a personal LLM tool. since it's vast enough for having info to craft responses… Well… No, it does not work. direct org-roam-node-find or counsel-rg (ripgrep Emacs wrapper with live results) is not only far simpler and faster, but offer far more reliable responses.

I've tried now, a formally far more mature project and results are still very disappointing, most of the changes are in a far more demanding setup and more disk space…

the NixOS setup

Since the packaged version is really old, I do not feel confident enough with Nix language and I just want a quick test, instead of writing a derivation I choose the pip way, witch as regularly for most ML stuff with compiled deps does not work alone, the basic setup is

  • choose where to put khoj, like mkdir /tmp/khoj
  • touch there a default.nix file with
{ pkgs ? import <nixpkgs> {} }:

let fhs = pkgs.buildFHSUserEnv {
  name = "fhs-env";
  targetPkgs = pkgs: with pkgs; [
    stdenv.cc.cc.lib gcc gcc_multi
  multiPkgs = pkgs: with pkgs;
  runScript = "bash";
  profile = ''
    export FSH_ENV='1'
in pkgs.stdenv.mkDerivation {
  name = "fhs-env-shell";
  nativeBuildInputs = [ fhs ];
  shellHook = "exec fhs-env";

to create an FHS nix-shell environment with some "problematic" deps already packaged. Inside the shell we can create the pyenv and proceed

python -m venv venv ; cd venv; source bin/activate
pip install --upgrade pip
CMAKE_ARGS="DLLAMA_CUDA=on" FORCE_CMAKE=1 pip install khoj-assistant

the added vars are to use CUDA with Meta LLAMA LLM since I have a decent NVidia card. Oh, now Khoj want also a PostreSQL DB, yep, no SQLite. Not only, it want a PostreSQL extension: pgvector, well, no issue with NixOS, it's just a simple

postgresql = {
  enable = true;
  extraPlugins = with pkgs.postgresql_15.pkgs; [ pgvector ];
  package = pkgs.postgresql_15;
}; # postgresql

than a quick

su - postgres
 create user khoj with password 'ChooseSomething';
 create database khoj;
 grant all privileges on database khoj to khoj;

Oh, yup, no I forget the pgvector extension…

psql khoj
create extension vector;
alter role khoj superuser; # just for the setup

now, in the nix-shell + pyenv created

export POSTGRES_USER=khoj
export POSTGRES_PASSWORD='ChooseSomething'
export POSTGRES_DB=khoj

# these will be the admin WebUI credentials
export KHOJ_ADMIN_EMAIL="khojadmin@invalid.tld"
export KHOJ_ADMIN_PASSWORD="grumbleMumble"

# --anonymous-mode to avoid being foced to log-in with Google!
khoj --anonymous-mode

Now another unpleasant discovery, the new WebUI does not work with my files alone, it want a mandatory client, Emacs one or desktop one. There is no more a khoj.yml for a simple (albeit obscene, as all YAML stuff) config…

As a quick solution, since the WebUI allow for a single file at a time upload, I've made a quick

cat ~/org/**/*.org >>| /tmp/khoj/allnotes.org

than upload the mega-file. Than made some test queries. Well… After a Hermes-2-Pro-Mistral-7B.Q4_K_M.gguf autodownload (4.37Gb, pushing the total Khoj deploy to 4.4Gb + 15Mb of PostreSQL database just for a test) and some CUDA issues (probably solvable using conda instead of pip) what I get?


Limited results, slow of course, even if on an Nvidia RTX 3060 with 12Gb o videoram and 64Gb of system ram, giving essentially not better result than a ripgrep or a direct heading access to my notes.

Trying some docs (.pdf etc) it's no better, yes, it offer a kind of access you can't have on them without org-mode, meaning with a classic file+folders home taxonomy as still most people have and do not even know something better exists… Well… Something STILL not better than manual access with Recoll (a Xapian frontend) or even an rga search that have the benefit of zero-setup and zero-a-priori indexing to give results.

Of course an LLM can "answer", while classic tools can only query, meaning for instance you can't look for some concept, ignoring possible synonyms and so on. But seen the quality of the results… It's really no better AND you still do not know what happen to your data, since it's impossible to skim the entire codebase, not of khoj alone but all the deps.

Having tried, on their own model, some services, like ChatGPT, my conclusion so far is: nice for the wow effect, nice for those who do not know how to make a damn search alone, but almost useful for real world serious usage.

Sorry devs, it's not against you, LLMs are impressive for what they are, but the reality is that we are FAR behind the current state of advertisement.