nix-doom-emacs-unstraightened

nix-doom-emacs-unstraightened (referred to as "Unstraightened" below) builds Doom Emacs using Nix bundling a user configuration directory and the dependencies specified by it. It is very similar to nix-doom-emacs, but is implemented differently.

Status

Tested and working on Linux, with emacs-overlay and Doom inputs updated automatically. If you're reading this on GitHub, there should be a CI status badge above: if CI is passing, Unstraightened installs an up-to-date version of Doom Emacs and (almost) all module dependencies.

macOS is covered by CI and reported as working by users, but not used by this project's author. Please file an issue if it does not work.

You may encounter "Cannot find Git revision" errors on Nix versions newer than 2.18.x (see #14). Try enabling experimentalFetchTree to work around this (see below).

Please report any issues.

Pinned sources are fixed-output derivations

Note

This is the one behavioural difference between this fork and upstream marienz/nix-doom-emacs-unstraightened.

Upstream fetches every pinned package with an eval-time fetcher primop (builtins.fetchGit, or builtins.fetchTree with experimentalFetchTree). That has three costs which only bite once you run more than one machine:

1. It can never be served by a binary cache. An eval-time fetch has no .drv. Nix resolves a cache hit for one with store.isValidPath() — a purely local check (src/libfetchers/fetch-to-store.cc:56), which never consults a substituter. Nix does have substitution machinery for trees (src/libfetchers/fetchers.cc:329, store.ensurePath(…)), but it is gated on isFinal(), and primops/fetchTree.cc:220 rejects __final from any user-supplied input; the only primop that sets it is fetchFinalTree, which is .internal = true and reachable only from the flake locker. So your flake inputs are substitutable and these package sources structurally cannot be.
2. It is serial. Nix evaluation is single-threaded, so N pins are N sequential fetches.
3. It happens during nix eval, so merely checking that a configuration evaluates drags the network.

This fork fetches every pin listed in pins.lock.json with fetchgit instead — an ordinary fixed-output derivation. Those are substitutable from any binary cache, realised in parallel across max-jobs, and need no network at evaluation time at all.

Upgrading re-downloads nothing. fetchgit { name = "source"; … } produces the byte-identical store path the eval-time fetchers produced for the same rev: both are fixed:r:sha256 content-addressed over the NAR of the same git tree, and a content-addressed path is a pure function of (name, method, hash). Any machine that already fetched a source the old way already satisfies the new FOD.

Pins not in the lock fall back to the original eval-time fetcher with a warning, so a stale lock degrades to upstream's behaviour rather than breaking. That fallback is load-bearing: your own DOOMDIR can package! anything with its own :pin, and no lock shipped here can know about it.

Regenerating the lock

After bumping the doomemacs or doomemacs-modules inputs:

nix flake update          # moves the Doom inputs
nix run .#update-pins     # refetches every pin, rewrites pins.lock.json
git add -- flake.lock pins.lock.json    # commit BOTH, always together

Important

A flake.lock bump committed without a regenerated pins.lock.json is the one state that quietly undoes this whole mechanism. Moving the Doom inputs moves Doom's :pins to revs the old lock has never seen, and every one of those falls back to the eval-time fetcher — which is a warning, not an error. Consumers keep building; they just go back to serially cloning hundreds of repositories on every machine. Verify with:

nix eval .#packages.x86_64-linux.doom-emacs.drvPath 2>&1 | grep -c 'not in pins.lock'

Zero is the healthy baseline. Anything else means either a stale lock, or a package pinned by your own DOOMDIR — see below.

This fetches every pin and rewrites pins.lock.json. It is deliberately the slow path.

The lock is the union of three whole-Doom configurations — all modules with no flags, with every flag, and with every flag except +eglot / +flymake. No single one is a superset, because a flag frequently selects an alternative package rather than an additional one: with every flag enabled :emacs (undo +tree) pulls in undo-tree and drops vundo and undo-fu-session, and with no flags the reverse. Merging all three covers both sides of each either/or.

Pins from your own DOOMDIR

update-pins generates the lock from Doom's module tree, so it can never see a package! you declared yourself. Such a pin would fall back to a serial eval-time fetch on every machine, forever, and no amount of regenerating the lock will change that.

Cover it with extraPinHashes, which is merged over the committed lock:

programs.doom-emacs.extraPinHashes = {
  # must equal the `:pin` in your packages.el
  "a9485f766ea69f6cb3a3f08dea20d44fd6596673" = {
    # exactly what the recipe derives: https://github.com/<repo>, no .git
    url = "https://github.com/manzaltu/claude-code-ide.el";
    submodules = false;
    hash = "sha256-6kaTPI2CCsdxxiCpZ7qqciv/HJCQRsJ8084+SqW8Idc=";
  };
};

Get a hash with nix flake prefetch --json "git+<url>?rev=<rev>". Bump the hash whenever you bump the :pin; if you forget, the fallback warning names the package.

How to use

(optional) Enable Cachix

If you use nixpkgs-unstable, adding our Cachix cache should speed things up. See https://app.cachix.org/cache/doom-emacs-unstraightened or the Cachix documentation for instructions.

If you do not use nixpkgs-unstable, or see slow build times even with the cache, please file an issue mentioning which version of nixpkgs and Emacs you use and I will see what I can do.

Test run

nix run github:marienz/nix-doom-emacs-unstraightened#doom-emacs \
  --override-input doomdir ~/.config/doom \
  --override-input nixpkgs nixpkgs

If you do not already have Doom Emacs set up, omit --override-input doomdir ~/.config/doom to use a simple example configuration. (Or if your configuration is in a different location or a git repository, pass its path or URL here.)

Note

If the path to your doomdir is in a Git repository but not yet checked in, you will need to git add it for this to work. (Your doomdir does not need to be in a Git repository for testing, but if it is in one, it needs to be tracked by Git.)

Note

If the path to your doomdir is in a subdirectory of a Git repository, you may need to use --override-input doomdir path:/path/to/doomdir (see this issue).

--override-input nixpkgs nixpkgs is optional. If nixpkgs is in the system registry (which it is by default on current NixOS) this allows Unstraightened to use more dependencies (particularly Emacs) already on your system. If omitted, current nixpkgs-unstable is used.

If nix run github:marienz/nix-doom-emacs-unstraightened#doom-emacs does not work, the "with flakes" setup below is unlikely to work either. Please file an issue. If this does work, but passing in your doomdir does not, please file an issue if you cannot work out the problem (and you're willing to share at least part of your configuration).

With flakes

Add this flake as an input in flake.nix:

inputs = {
  nix-doom-emacs-unstraightened = {
    url = "github:marienz/nix-doom-emacs-unstraightened";
    inputs = {
      # If you use the home-manager module, you can set your doomdir here.
      #
      # If your Doom configuration is in this flake:
      doomdir.url = "./doom.d";
      # If your Doom configuration is in a different repository:
      doomdir.url = "git+https://example.com/git/doomdir";
      # Or leave `doomdir` unset and set `programs.doom-emacs.doomDir` in your
      # home-manager configuration (see below).

      # Optional, to download less. Neither the module nor the overlay uses this input.
      nixpkgs.follows = "";
    }
  };
};

Home Manager

If you use Home Manager, add Unstraightened's Home Manager module in flake.nix:

outputs = inputs @ { nixpkgs, home-manager, ... }: {
  homeConfigurations."username" = home-manager.lib.homeManagerConfiguration {
    modules = [
      inputs.nix-doom-emacs-unstraightened.homeModule
      ./home.nix
    ];
    extraSpecialArgs = { inherit inputs; };
  };
};

Configure it in home.nix:

  programs.doom-emacs = {
    enable = true;
  };

There are a few other configurable options, see below.

If you set services.emacs.enable = true, that will run Unstraightened as well (Unstraightened sets itself as services.emacs.package). Set programs.doom-emacs.provideEmacs = false or override services.emacs.package if you want a vanilla Emacs daemon instead.

Warning

Using the overlay described below with programs.emacs.package will not work correctly (see HACKING for details).

Overlay

If you don't use Home Manager or prefer not to use Unstraightened's Home Manager module, add Unstraightened's overlay. Typically that means adding:

nixpkgs.overlays = [ inputs.nix-doom-emacs-unstraightened.overlays.default ];

to a Home Manager or NixOS module.

The overlay adds two packages:

• To install Unstraightened in parallel with a normal Emacs, add:

  (pkgs.doomEmacs {
    # Path to your doom configuration.
    # Required (does not default to the doomdir flake input).
    doomDir = ./path/to/your/doom/config;
    doomLocalDir = "~/.local/share/nix-doom";
  })
  

  to your installed packages (see below for what doomLocalDir is for). This installs a binary named doom-emacs.

• To install Unstraightened as your default Emacs, use pkgs.emacsWithDoom instead of pkgs.doomEmacs. This installs a binary named emacs as well as emacsclient and other helpers (similar to emacsWithPackages in nixpkgs).

Without flakes

This is currently not explicitly supported, but should be possible (use pkgs.callPackages ./nix-doom-emacs-unstraightened). PRs extending this part of the documentation are welcome, as are (within reason) changes necessary to support use without flakes.

Options

doomEmacs and emacsWithDoom support the following options:

• doomDir: your configuration directory (also known as DOOMDIR, Doom private directory / module). Required when not using the home-manager module, which defaults it to the doomdir flake input.

• doomLocalDir: value Doom should use as DOOMLOCALDIR. Required, because by default Doom would use its source directory, which is read-only.

Note

This supports ~ expansion but does not support shell variable expansion. Using $XDG_DATA_HOME will not work.

Note

Because Unstraightened uses Doom's profile system, using the same value you used with vanilla Doom will not result in Unstraightened finding your files. See below.

• emacs: Emacs package to use. Defaults to pkgs.emacs. Use this to select different Emacs variants like pkgs.emacs-pgtk.

• doomSource: Doom source tree. Defaults to a flake input: overriding that input is probably easier than passing this.

• doomModules: like doomSource, but for the Doom modules source tree.

• extraBinPackages: packages to add to $PATH. Defaults to Git, ripgrep and fd.

• extraPackages: Specify extra Emacs packages from nixpkgs to be available to Doom Emacs. Defaults to this function epkgs: [ ] (no extra packages). For example to include Emacs package treesit-grammars.with-all-grammars: extraPackages = epkgs: [ epkgs.treesit-grammars.with-all-grammars ];.

• experimentalFetchTree: fetch packages using fetchTree, which is more efficient but considered experimental in Nix (subject to changes which might break fetches). Only applies to pins not covered by pins.lock.json — see Pinned sources are fixed-output derivations.

• pinsLock: the rev → { url, submodules, hash } map used to fetch pinned packages as fixed-output derivations. Defaults to the pins.lock.json committed in this repository. Pass { } to disable it and use the eval-time fetchers everywhere, which is upstream Unstraightened's behaviour.

• tangleArgs: When set, Unstraightened runs doom +org tangle in doomDir. See doom +org tangle --help for the arguments you can use here, and see the Org-mode manual for more information.

• emacsPackageOverrides: Function passed to (emacsPackagesFor emacs).overrideScope (see Nixpkgs manual). Runs after Unstraightened's own overrides have been applied. That means it can be used to fix problems with Unstraightened's automatically-generated derivations, by using esuper.<name>.overrideAttrs.

Note

The :config literate module has no effect when using Unstraightened. Use tangleArgs = "--all config.org" instead.

Note

Currently, this feature uses the version of Org-mode that comes with Emacs when tangling, not the version installed by Doom. This limitation could be lifted: file an issue if this limitation is a problem for you.

There are a few other settings but they are not typically useful. See the source.

The Home Manager module supports the same options, as well as:

• provideEmacs: disable this to only provide a doom-emacs binary, not an emacs binary (that is: it switches from emacsWithDoom to doomEmacs). Use this if you want to install vanilla Emacs in parallel.

Comparison to "normal" Doom Emacs

• Unstraightened updates Doom and its dependencies along with the rest of your Nix packages, removing the need to run doom sync and similar Doom-specific commands.

• Doom pins its direct dependencies, but still pulls the live version of some packages from MELPA or other repositories. Its pins are also applied to build recipes whose source is not pinned. This makes Doom installs not fully reproducible and can cause intermittent breakage.

  Unstraightened pulls these dependencies from nixpkgs or emacs-overlay. Pinning emacs-overlay pins all build recipes and packages not already pinned by Doom.

• Unstraightened stores your Doom configuration (~/.doom.d/~/.config/doom/$DOOMDIR) in the Nix store. This has advantages (the configuration's enabled modules always match available dependencies), but also some disadvantages (see known problems below).

• Unstraightened uses Doom's profiles under the hood. This affects where Doom stores local state:

  Variable	Doom	Unstraightened
  doom-cache-dir	$DOOMLOCALDIR/cache	~/.cache/doom
  doom-data-dir	$DOOMLOCALDIR/etc	~/.local/share/doom
  doom-state-dir	$DOOMLOCALDIR/state	~/.local/state/doom

  (Doom also stores some things in per-profile subdirectories below the above directories: the default profile name used by Unstraightened is nix, resulting in paths like ~/.cache/doom/nix. All of these also respect the usual XDG_*_DIR environment variables.)

  When migrating from "normal" Doom, you may need to move some files around.

  If this bothers you, you can try setting profileName = "". This makes Unstraightened use the usual paths (relative to doomLocalDir), but uses somewhat of a hack to do so (please report any issues observed).

Comparison to nix-doom-emacs

nix-doom-emacs was an earlier solution to the same problem.

• Unstraightened does not attempt to use straight.el at all. Instead, it uses Doom's CLI to make Doom export its dependencies, then uses Nix's emacsWithPackages to install them all, then configures Doom to use the "built-in" version for all its dependencies.

• Unstraightened respects Doom's pins. This was not practical to implement in nix-doom-emacs, but seems to be necessary to avoid breakage caused by divergence between Doom's pinned packages and emacs-overlay (or nixpkgs).

Bugs

Do not report bugs upstream. If you think it's a bug in Doom, reproduce it without Unstraightened first, or report it here first.

There are a few known current bugs and likely future bugs in Unstraightened:

Pins can break

The way Unstraightened applies Doom's pins to Nix instead of straight.el build recipes is a hack. Although it seems to work fairly well (better than I expected), it will break at times.

If it breaks, it has (so far) only broken at build time.

There are two common failure modes. You should not see either of them for Doom's packages (CI catches them and prevents Unstraightened's pinned versions of Doom and emacs-overlay from updating until I work around the problem), but you may see them for packages you pin in your own packages.el.

Git commit not found

Errors about Git commits not being present in the upstream repository can occur because Unstraightened applies pins to build recipes from emacs-overlay, while Doom applies them to straight.el build recipes. These two sources sometimes diverge.

To work around this for a pin in packages.el, add a :recipe.

Missing dependencies

Unstraightened currently uses dependencies from the original (emacs-overlay) package. This is a performance optimization: auto-detecting dependencies uses IFD, which would be very slow if used for all packages.

The usual way in which this breaks is: a package dropped a dependency upstream, and emacs-overlay picked up a revision with this change, but it is still pinned to an earlier version.

To work around this for a pin in packages.el, you can use emacsPackageOverrides to add the missing dependency back, like this). But in most cases bumping the package will be simpler.

Saving Custom changes fails

Saving changes through Custom will not work, because custom-file is read-only. I am open to suggestions for how this should work:

• Currently, DOOMDIR/custom.el is loaded, but changes need to be applied manually.
• If we set custom-file to a writable location, that fixes saving but breaks loading. If the user copies their custom-file out of their DOOMDIR to this location once, they are not alerted to changes they may want to copy back.
• If we try to use Home Manager, I would expect to hit the same problems and/or collisions on activation, but I have not experimented with this.

Flag-controlled packages may be broken

To catch problems like the ones involving pins, Unstraightened tries to build-test all packages used by Doom. But just enabling all of Doom's modules is not sufficient to pull in all packages because some dependencies are controlled by module flags, and enabling all flags on all modules is still not sufficient because some dependencies are controlled by specific combinations of enabled and disabled flags...

Unstraightened currently approximates this by building with all modules enabled (but flags disabled), all modules and all flags enabled, and a few manually identified combinations of enabled and disabled flags (example: covering language modules with +lsp but without +eglot).

I may have missed a corner case: if you notice one, please file an issue.

doom doctor fails with / complains about...

"Checking for stale elc files... File is missing"

> Checking for stale elc files...
x There was an unexpected runtime error
  Message: File is missing
  Details: ("Opening directory" "No such file or directory" "/home/marienz/.local/share/nix-doom/straight/build-30.1")

For now, just create the directory.

I would like to fix this but have not thought of the least messy way yet.

"Doom is installed in a non-standard location"

Ignore it.

Unstraightened uses --init-directory, as the doctor recommends.

"Found another Emacs config:"

Safe to ignore, for the same reason as the previous warning.

tree-sitter grammars are not installed automatically

Install them manually by setting

  extraPackages = epkgs: [ epkgs.treesit-grammars.with-all-grammars ];

(Or specify just the grammars you need by following the example in nixpkgs, but these are tiny so installing all of them is usually fine.)

This may be automated in the future (see this issue).

Frequently Anticipated Questions

How do I add more packages?

Add (package! foo) to packages.el.

Do not wrap emacsWithDoom in emacsWithPackages. See HACKING for why this will not work.

The Home Manager option extraPackages is available to add extra Emacs packages from nixpkgs to Doom Emacs. If this is not sufficient, please file an issue.

How do I add packages not in Emacs overlay?

Add (package! foo :recipe ...) to packages.el.

If there is a derivation in emacs-overlay or nixpkgs, it will use the same dependencies as the derivation (which won't always work if you pin to a sufficiently different version). If not, the dependencies are auto-detected from headers in the package.

If the detected dependencies are not sufficient, you will have to add the package in Nix directly, with extraPackages:

              extraPackages = (
                epkgs: [
                  (epkgs.melpaBuild {
                    pname = <package-name>;
                    version = "9999snapshot1";
                    packageRequires = [ <dependencies> ];
                    src = fetchTree {...};
                  })
                ]
              );

See also #70.

If this is not sufficient, file an issue explaining what you're trying to do.

What's wrong with straight.el?

straight.el is great, but its features are somewhat at odds with Nix:

• straight.el can fetch package build recipes and packages. We cannot use this from within Nix's build sandbox: we would need to build a system similar to how emacs-overlay updates elpa / melpa and get straight.el to use it.
• straight.el maintains a tree of mutable Git checkouts: you can edit these directly or use straight.el to maintain them. The Nix store is immutable so none of this will work.
• straight.el can build packages, but so can nixpkgs / emacs-overlay.

Doom heavily uses straight.el during doom sync, but it does not use it at all at startup and barely uses it after that. Since we're replacing doom sync in its entirety, bypassing straight.el seems simpler than trying to use it just for package builds.

Unstraightened seems to use package.el. Isn't that bad?

Doom's FAQ offers several arguments against package.el. They boil down to two problems, neither of which applies to Unstraightened:

• package.el always builds from head: no rollback, no pinning, no reproducibility, no way to override the package source used. Unstraightened does not use package.el to fetch packages: it leaves that to Nix. We can handle pinning there, and Nix flakes add further reproducibility and rollback beyond what Doom's pins offer.
• package.el can be slow to initialize. Doom normally speeds up startup by combining autoloads from all installed packages into one file. Because package.el produces autoload files much like straight.el does, and we're loading everything from the immutable Nix store, we can apply exactly the same approach to package.el. Unstraightened startup performance should be about the same as vanilla Doom.

It's so slow to build!

Parallel builds should help (set Nix's max-jobs to something greater than 1), but it is a bit slow.

There are a few issues:

• Unstraightened uses IFD to determine packages to install and to determine package dependencies for packages not in emacs-overlay. Especially the latter is slow.

• Doom (currently) does not native-compile ahead of time, but Unstraightened (or nixpkgs, really), does.

  • You can try to disable this through this trick, (set emacs passed to Unstraightened to that), but it relies on nixpkgs implementation details and might break at some point.

  • https://cachix.org/ may help (see next point).

• Unstraightened's packages are cached on cachix.org, but this is not perfect.

  • Unstraightened's generated derivations for elisp packages do not depend on the exact Doom source or configuration they're generated from, making them somewhat cacheable. But they do do depend on the pinned version and specific Emacs binary: we do not build and cache all possible combinations.

Required disclaimer

This is not an officially supported Google product. It is a personal side project.