On AI, LLM personal use, Khoj - a RANT

Since the packaged version is really old, I do not feel confident enough with Nix language and I just want a quick test, instead of writing a derivation I choose the pip way, witch as regularly for most ML stuff with compiled deps does not work alone, the basic setup is

• choose where to put khoj, like mkdir /tmp/khoj
• touch there a default.nix file with

{ pkgs ? import <nixpkgs> {} }:

let fhs = pkgs.buildFHSUserEnv {
  name = "fhs-env";
  targetPkgs = pkgs: with pkgs; [
    stdenv.cc.cc.lib gcc gcc_multi
    ];
  multiPkgs = pkgs: with pkgs;
    [
      cmake
      libgcc
      llama-cpp
      python311Packages.cmake
      python311Packages.ninja
      python311Packages.pytz
      python311Packages.wheel
    ];
  runScript = "bash";
  profile = ''
    export FSH_ENV='1'
  '';
};
in pkgs.stdenv.mkDerivation {
  name = "fhs-env-shell";
  nativeBuildInputs = [ fhs ];
  shellHook = "exec fhs-env";
}

to create an FHS nix-shell environment with some "problematic" deps already packaged. Inside the shell we can create the pyenv and proceed

python -m venv venv ; cd venv; source bin/activate
pip install --upgrade pip
CMAKE_ARGS="DLLAMA_CUDA=on" FORCE_CMAKE=1 pip install khoj-assistant

the added vars are to use CUDA with Meta LLAMA LLM since I have a decent NVidia card. Oh, now Khoj want also a PostreSQL DB, yep, no SQLite. Not only, it want a PostreSQL extension: pgvector, well, no issue with NixOS, it's just a simple

postgresql = {
  enable = true;
  extraPlugins = with pkgs.postgresql_15.pkgs; [ pgvector ];
  package = pkgs.postgresql_15;
}; # postgresql

than a quick

su - postgres
psql
 create user khoj with password 'ChooseSomething';
 create database khoj;
 grant all privileges on database khoj to khoj;
 ALTER DATABASE khoj OWNER TO khoj;
 exit

Oh, yup, no I forget the pgvector extension…

psql khoj
create extension vector;
alter role khoj superuser; # just for the setup
exit

now, in the nix-shell + pyenv created

export POSTGRES_USER=khoj
export POSTGRES_PASSWORD='ChooseSomething'
export POSTGRES_DB=khoj

# these will be the admin WebUI credentials
export KHOJ_ADMIN_EMAIL="khojadmin@invalid.tld"
export KHOJ_ADMIN_PASSWORD="grumbleMumble"

# --anonymous-mode to avoid being foced to log-in with Google!
khoj --anonymous-mode

Now another unpleasant discovery, the new WebUI does not work with my files alone, it want a mandatory client, Emacs one or desktop one. There is no more a khoj.yml for a simple (albeit obscene, as all YAML stuff) config…

As a quick solution, since the WebUI allow for a single file at a time upload, I've made a quick

cat ~/org/**/*.org >>| /tmp/khoj/allnotes.org

than upload the mega-file. Than made some test queries. Well… After a Hermes-2-Pro-Mistral-7B.Q4_K_M.gguf autodownload (4.37Gb, pushing the total Khoj deploy to 4.4Gb + 15Mb of PostreSQL database just for a test) and some CUDA issues (probably solvable using conda instead of pip) what I get?

Limited results, slow of course, even if on an Nvidia RTX 3060 with 12Gb o videoram and 64Gb of system ram, giving essentially not better result than a ripgrep or a direct heading access to my notes.

Trying some docs (.pdf etc) it's no better, yes, it offer a kind of access you can't have on them without org-mode, meaning with a classic file+folders home taxonomy as still most people have and do not even know something better exists… Well… Something STILL not better than manual access with Recoll (a Xapian frontend) or even an rga search that have the benefit of zero-setup and zero-a-priori indexing to give results.

Of course an LLM can "answer", while classic tools can only query, meaning for instance you can't look for some concept, ignoring possible synonyms and so on. But seen the quality of the results… It's really no better AND you still do not know what happen to your data, since it's impossible to skim the entire codebase, not of khoj alone but all the deps.

Having tried, on their own model, some services, like ChatGPT, my conclusion so far is: nice for the wow effect, nice for those who do not know how to make a damn search alone, but almost useful for real world serious usage.

Sorry devs, it's not against you, LLMs are impressive for what they are, but the reality is that we are FAR behind the current state of advertisement.