A little earlier this evening I got a new issue raised against boxquote.el. Apparently Emacs 29 (I'm running 28.1 as of the time of writing) is moaning about the likes of:

(setf (point) some-location-or-other)

and

(setf (buffer-string) "")

There's a whole background to why I've tended to code like that, that stems from enjoying Common Lisp, my days reading (and sometimes posting to) comp.lang.lisp, and I think some of the stuff Erik Naggum wrote back in the day. I won't get into it all now; I'm not sure I can even remember a lot of how I got there given how far back it was.

But...

Wanting to quickly get to the bottom of why the above was suddenly an issue, I dived into the NEWS file and found the following:

** Many seldom-used generalized variables have been made obsolete.
Emacs has a number of rather obscure generalized variables defined,
that, for instance, allowed you to say things like:

   (setf (point-min) 4)

These never caught on and have been made obsolete.  The form above,
for instance, is the same as saying

   (narrow-to-region 4 (point-max))

The following generalized variables have been made obsolete:
'buffer-file-name', 'buffer-local-value', 'buffer-modified-p',
'buffer-name', 'buffer-string', 'buffer-substring', 'current-buffer',
'current-column', 'current-global-map', 'current-input-mode',
'current-local-map', 'current-window-configuration',
'default-file-modes', 'documentation-property', 'frame-height',
'frame-visible-p', 'global-key-binding', 'local-key-binding', 'mark',
'mark-marker', 'marker-position', 'mouse-position', 'point',
'point-marker', 'point-max', 'point-min', 'read-mouse-position',
'screen-height', 'screen-width', 'selected-frame', 'selected-screen',
'selected-window', 'standard-case-table', 'syntax-table',
'visited-file-modtime', 'window-height', 'window-width', and
'x-get-secondary-selection'.

As suggested above... this is my thing, this is how I coded some Elisp stuff. Look through much of my Emacs Lisp code and you'll find me setfing stuff all over the place.

Apparently my style is "obscure". Actually, I'm kinda okay with that if I'm honest.

This is going to be a bit of a pain in the arse; I'm going to have to go through a whole bunch of code and make it "less obscure", at some point.

But...

This isn't the part that had me thinking I must be getting old. Oh no. The NEWS file had another little surprise in store:

** The quickurl.el library is now obsolete.
Use 'abbrev', 'skeleton' or 'tempo' instead.

That.... that's me that is. Well, it's one of the me things. If you run about-emacs, dive into Authors, and search for my name, in any copy of GNU Emacs from the last decade or two, you'll find this:

Dave Pearson: wrote 5x5.el quickurl.el

quickurl.el was a package I wrote back in the late 1990s, back when I was a very heavy user of Usenet, and often found myself posting the same URLs in posts again and again; especially in comp.lang.clipper. As a fairly quick hack I wrote the code so that I could very quickly insert often-used URLs.

Some time later, I got an email from the FSF (I actually think it was from RMS -- but that's an mbox I've long ago lost -- or a backup of it might be in storage back in England, on a DVD), asking if I wanted to contribute it to Emacs proper. This seemed like an odd thing to add to Emacs but, sure, why the hell not?

And so I had my second contribution to a body of code I used a lot (the first being 5x5.el -- which itself was my first ever attempt at writing some non-trivial Elisp code).

So... yeah... here we are. I'm now old enough to have written some Emacs Lisp code, had it required by the FSF for inclusion in Emacs, had it live in there for something like two decades, and then become obselete!

When I started with this version of the blog on Hashnode¹, my plan was to try and write something at least once a week. I managed to do that as far as January this year, give or take, but then things got a little busy, the world got trickier, and... Well, you don't need me to tell you about that. Anyway, I'm back to write some more and, hopefully, try and keep up with writing. I feel it'll do me good.

I have been busy since January. Work has, as far as possible, carried on as normal. The only big difference being I'm back working from home. It's an odd turn of events personally given that I'd worked from home for 21 years, then (with some trepidation) went back to working in an office. Having got used to being in an office and being around people, it took some adjusting to working alone again.

So, on to the main point of this post... During that period I've spent a lot more time at my machine than I would normally, and so it was inevitable that I'd end up tinkering with my Emacs configuration. While it's true that there's hardly a week goes by where I don't tinker with some small thing, I try not to make huge changes too often. Huge changes do happen though.

A handful of weeks back was another one of those times where I made a big change.

After I burnt my original ~/.emacs file back in 2016 I've had an approach where I've tried to make things as modular as possible, and as easy to clone down to a new machine as possible. The design I came up with, especially once I moved over to using use-package, has served me well. But there was always one thing that bothered me: the files that handled the loading and configuration of packages were, in effect, still three large monolithic files. While this was better than a single monolithic ~/.emacs, it still didn't feel quite right.

Giving it a bit of thought, I decided that what I really wanted was a single directory in which I could drop lots of small .el files that would handle the loading and configuration of all sorts of packages. While it didn't gain anything concrete, the idea felt tidier. Tidy is good. Feeling like the code smell is good, is good too.

First though I needed a way to load multiple files, ideally within a whole directory hierarchy, without needing to know in advance what files would be there. After a little bit of tinkering I settled on this bit of code that revolves around the use of directory-files-recursively:

(let ((source (expand-file-name "init.d/packages.d/" user-emacs-directory)))
  (when (file-exists-p source)
    (cl-loop for use in (directory-files-recursively source (rx ".el" eol))
             do (load (file-name-sans-extension use)))))

Simply put, this code finds every file in and below ~/.emacs.d/init.d/packages.d/ whose name ends in .el (note the use of a regular expression, using rx to create it), removes the extension from the name, and loads it with load (dropping the extension means load can decide if it wants to load the compiled version of the code, or the source, depending on what's available).

After writing that I could then start to populate ~/.emacs.d/init.d/packages.d/ with lots of smaller files that handled the loading of packages. In some cases there's a file just for one package, in other cases there's a file that handles a logical grouping of packages (where there are obvious direct dependencies, or where one package is designed to extend the other, etc). For now I've decided to break things down into three directories that map to the three monolithic files I had before:

However, I'm not totally wedded to this design and I may change this as time goes on.

While I've not spent any time properly testing it out, I've also not really noticed any obvious impact on startup speed. However, this tends not to be a real concern for me. I seldom start Emacs anyway. I always have it running and use emacsclient as my editor in most places so files open instantly in the running version of Emacs.

Late on last year I stumbled on an article about dnote. Annoyingly, I can't recall now where I saw it, but I made a reminder to look at it over my Christmas break.

Dnote looked like a tool that would fill a hole I had in how I work. When it comes to making notes about things, and keeping things for future reference, I use a few tools, each one being just right (for me) for the job in question. I use Evernote to track documents and other household type things. I use Keep to make notes about stuff I need to remember short-term (say, the size of a space in my bedroom that I want furniture to go in) and also to record notes while in meetings at work. I use Journey to keep a journal about... anything, really. Finally, I use Pinboard to keep hold of URLs I might want to go back to (I also use it to create a to-read list).

Amongst all of this, however, I felt I was missing something for keeping track of things that don't really fall into any of the categories above. Mostly this would be work-based or hacking-based things that are short and sweet but I don't always use enough to easily remember. I wanted just the right tool that would let me ferret away useful one-liners, remind myself of obscure switches that get used once or twice a year, etc.

After reading up on dnote it seemed pretty clear that this was just such a tool.

After getting back to the office at the start of this month I decided to make use of it and see how it went. My idea was simple: I'd record any "TIL" stuff that I might want to remember in the future, as well as recording things I need now and again but can't always remember.

So far it's working quite well. I like that it has a simple CLI. I like that it's got a backend that you can use to sync between different machines. I like that it's got a web interface that's mobile-friendly. I like that it's Free Software and so you can host your own server if you wish.

I found I liked it enough that, of course, I felt the need to start a simple Emacs wrapper for the CLI.

At the moment dnote.el is designed to be a simple capture system. There are commands for capturing a one-liner (entered in the Emacs mini-buffer), for capturing the content of the current buffer, and for capturing freshly-entered multi-line text, entered in a buffer that uses markdown-mode. There's also a command for syncing notes if you have configured dnote to talk to a backend.

What I don't have right now is the ability to navigate and view notes. So far I've not really felt the need for that because the CLI approach works so well. Longer-term though I can see my tweaking this and adding in commands for searching, viewing, editing and deleting notes.

But, for now, if you've not had a look at dnote, I'd highly recommend having a play and seeing if it makes sense for you. And, if it does, and you're an Emacs user, perhaps dnote.el will be useful too?

Both at work and at home I use Emacs by keeping a copy running all the time, and use emacsclient to open files inside it (including on remote machines thanks to a bit of ssh and heavy use of tramp -- I might write this up at some point). This works really well, but does mean I tend to build up a lot of buffers over time.

Having lots of buffers open isn't generally an issue, and if I'm working on lots of different files in a project during the course of a hacking session it's actually a good thing. But, quite often, I want to tidy up the buffer list, clearing it back to near-zero buffers open. Many years ago, when I had a "proper" tower system running 24/7, with Emacs open all the time, I'd use clean-buffer-list as part of midnight. Along the way that fell out of favour with me, likely because I drifted into using machines that had Emacs open all the time but where the machine wasn't awake all the time.

Eventually I decided to have some fun rolling my own solution, and nuke-buffers was born.

Rather than try and do things in an automated way, this was designed to be bound to a key (or two) and then be run when I wanted, being as harsh as possible about cleaning up the buffer list. Since first writing it it's worked well for me.

These days I tend to let the buffer list build up as I work on a new feature, or chase down a bug, etc. Then, once I've made the final commit for that period of focus, I'll hit the nuke-buffer key combo as the final act of confirming that I've done the job. So not only does this help tidy my Emacs session a bit, it also feels like a physical form of punctuation -- back in less sensible days, when I had some terrible habits, it would have been when I'd reach for the celebratory cigarette; buffer-tidying feels far more wholesome. ;)

The way the code works is, of course, mostly directed at how I work -- it's highly likely it wouldn't make sense for many other people. The main aim is to kill as many buffers as possible, but without disturbing anything else. The list of buffers it gathers for nuking avoids buffers that are visiting files but have unsaved content, avoids the minibuffer (obviously), avoids any "special" buffer (one that starts with a space then an asterisk), avoids the current buffer and also avoids any buffer in a list of names to avoid.

I've being using this on a daily basis for around 2.5 years now and it's done the job without ever losing me any work.

While it does seem that they're a little out of fashion these days, in some circles anyway, I'm still an avid fan of make and make files. Even in environments where I don't need a Makefile to actually build anything, I'll use one (or more) to help create handy shortcuts for getting stuff done.

Looking at the main Makefile for one of my major work projects, there's 45 targets that help fire off various jobs (all of them self-documenting using a variation on an approach I read a while back).

In most cases the targets aren't real targets. That's to say, they don't build the thing they're called. They are phony targets. So, as makes sense, I make a point of marking them all as such. I follow the convention that has the .PHONY marker appear on the line before the target; this feels cleaner to me and easier to follow and maintain.

But.... I'm lazy. And I use Emacs. Typing out .PHONY foo all the time feels like far too much work. So, some time ago, I quickly threw together make-phony.el.

With this I could be really lazy. I could type out the Makefile target and then, with my cursor on it, press a key combination and have the .PHONY marker put in place.

Does it save much time? Yeah, probably not really. But it was a fun little exercise and an excuse to write a little bit of Emacs Lisp.

There's one thing I made a point of doing in the heart of this too: using rx. For anyone who doesn't know of it, think of it as a very Lispy way of writing regular expressions. I won't even try and explain it all here because others have done an excellent job already. What I will do is say this: if you're in the habit of writing some Emacs Lisp, or even tinkering with your configuration, and you find yourself writing a regular expression, consider looking at rx -- it's well worth the time to get to know it.

Slowly, as time goes on, I'm weeding out "vanilla" regular expressions from my config and code and moving over to using rx. I feel, quite rightly I think, that something like this:

(rx
 (or
  ;; Ignore hidden files.
  (group bol ".")
  ;; I never want to edit the desktop.
  (group "Desktop/" eol)
  ;; Ignore compiled files.
  (group "." (or "pyc" "elc") eol)
  (group ".egg-info/" eol)))

is much easier to write, read and maintain, than this:

"\\(^\\.\\)\\|\\(Desktop/$\\)\\|\\(\\.\\(?:\\(?:\\(?:el\\|py\\)c\\)\\)$\\)\\|\\(\\.egg-info/$\\)"

I mean, even if the regular expression above can be written in a more efficient way (and I imagine it can), as someone working in a Lisp environment, I'd much sooner write and work with the rx version.

As detailed in a blog post I wrote back in 2016, I first got into using Emacs in the mid 1990s, starting with it on OS/2 and then moving over to GNU/Linux. It's been my often-used and much-loved development environment for most of those years (I even have a couple of packages that are part of Emacs itself).

For most of that time my configuration was a single ~/.emacs file, which was around 1,000 lines in length (including comments and whitespace). It'd grown over the years, having special configuration sections for versions of Emacs I didn't use any more, and operating systems I didn't work on any more (yes, really, there were things in there specific to MS-DOS, for example). On top of that I always hand-installed packages I used -- Emacs' package management system having turned up long after I first got into using Emacs.

Then, in early 2016, I decided to nuke the whole thing and start from scratch. As mentioned above, the start of this is detailed in an older post. Another big round of changes happened round a year later -- which included the birth of delpa to manage my personal packages. A couple or so months later there was one last big round of changes, mostly killing off my enthusiastic embracing of customize and instead going back to hand-set settings, only this time done via use-package.

The full history of this can be found over on GitHub, starting with the first "throw everything away and start again" process and all the steps between then and where my Emacs configuration is now.

Which brings me to the fun part of this blog post. Earlier this week I stumbled on Gource. It's a tool that's primarily designed to visualise changes in repositories, although it can be used to visualise anything that has a tree structure and changes over time (this week I produced a video of the growth of my employer's electronic lab notebook by hooking up the Benchling API with Gource, for example). So I got curious. What did it look like as I reworked and tweaked and changed and tinkered with my Emacs configuration?

This is what it looked like:

One of the things I really like about coding with Emacs is how I can easily identify a repeated task and turn it into a command in my environment, saving me a load of work down the line.

pypath.el is one such example.

In my day job I write a lot of Django code. As part of that, I write a good number of unit tests too. Sometimes I'll write the tests as I'm writing the code they test, other times I'm writing them afterwards; it all really depends on where my head's at and how the code is flowing.

When I'm writing those tests, I often want to test them as I go. Given that starting up a test session can take a while, and given that running all the tests in the system can take a while, it's really handy if I can run that single test I'm working on.

This is easy enough with Django. In my work environment it's normally something like:

$ pipenv run ./manage.py test -v 2 app.test.some.sub.module.TestClass.test_method

Only... typing out the:

app.test.some.sub.module.TestClass.test_method

part is a bit of a pain. Sure, once you've typed it the once you can use your shell of choice (mine being fish and on occasion eshell) to recall it from history, but typing it out the first time is the annoying part.

So this was the point where I took 1/2 hour or so to code up pypath.el to solve the problem for me. It gives me two commands:

• pypath: which simply places the dotted path of the current "defun", within the context of being part of a Django system, into the clipboard buffer.
• pypath-django-test: which works similar to the above but places the whole Django testing command into the clipboard.

With the above, I can work on a test, hit the latter command above, flip to my command line, paste and I'm running the test.

Of course, I'm sure there's plenty of other handy ways to do this. Doubtless there's work environments where the test can be run right there, in the edit buffer, without flipping away, and which takes into account the fact that there's a pipenv-managed virtual environment involved, etc. If there is, that's great, but I don't think it'd work with how I work.

And that's one of the things I really love about Emacs, and why it's still my work environment after almost 25 years of on and off use: with very little work on my part I can create a couple of commands that work exactly how I need them to. While it's great to create generally-useful code for Emacs that lots of people benefit from, sometimes the real value is that you can code up your own particular quirk and just get on with stuff.

To conclude: this post isn't to show off pypath.el; really this post is to sing the praises of Emacs and why it still works so well for me after all these years.

I've been pretty quiet on here since I last wrote about how I'd done a further revamp of my Emacs config, so I thought that subject would be a good reason to write another blog post.

It'll be a mostly short one, and one to muse over something that's been bugging me for a while now: my decision to lean heavily on customize to set all sorts of settings.

Initially, when I nuked my original config over a year ago, it seemed to make a lot of sense. Let all the tweaks and set values "hide" in a file of their own and try and keep the hand-edited config files as small and as clean as possible. Recently though I've got to thinking that this obscures too much, hides too much detail, and removes the ability to actually document what I'm doing and why. It also does make it tricky to adapt some settings to different platforms or even environments on a single platform.

Another problem I've run into is this: when I made the second round of changes and decided to lean heavily on use-package, I soon ran into the minor issue of some packages not making sense, or even being needed, on some platforms (stuff that's useful on my macOS machines isn't always useful on my Windows machines, that sort of thing). While use-package can handle this easily thanks to the :if keyword, I'm still left with the fact that package-selected-packages still gets populated.

Having package-selected-packages contain a list of installed packages likely makes sense if you're using just the Emacs package system and you're not doing the installing with use-package and :ensure. But with use-package and :ensure I feel like I've got far more control over things and can adapt what gets installed when depending on which Emacs I'm running where.

But, because I'm syncing my ~/.emacs.d/.custom.el to all my machines too, any use-package that has a :if to not bother using a package has little effect because the package still ends up being listed/loaded/seen as part of the installation.

Ideally, I think, I'd like to be able to have package-selected-packages held in its own file, or I'd only ever use ~/.emacs.d/.custom.el for local stuff (and so stop syncing it).

Starting today I'm going about a process of moving as much as I can out of ~/.emacs.d/.custom.el and into hand-edited files. In some respects I guess I am going back to how I used to manage Emacs configuration, but this time it's not a massive monolithic file-of-Lisp, it's neatly broken down into sensible sections and it's also biased towards a "grab and config this package" approach.

Meanwhile, I've not seen any good discussions online about customize vs "hand-edit", which strikes me as a little odd as it feels like the perfect "religious issue" for people to have endless disagreements over. I guess, over the next couple or so weeks, I'll find out if switching back was a good idea.

Just under a year ago I decided to totally rewrite my GNU emacs config. As I wrote at the time, it'd been following me around all sorts of machines since the early 1990s, starting life on an OS/2 Warp machine and travelling via MS-DOS, GNU/Linux, Windows and, these days, macOS.

The changes I made last year have served me really well, but there were two related issues with it that bothered me a little: the fact that I was maintaining a local library of elisp code in the repository and, worse still, I was storing the packages I'd installed from elpa and melpa in the repository as well.

While this did mean it was pretty easy for me to start up a new installation of emacs on a machine -- all I had to do was clone the repo and run up emacs -- I wasn't happy with the duplication involved. I didn't like holding code in my .emacs.d repo that was already held in package archives.

The solution I saw was in two parts:

1. Get some of my code, that might be useful to others, into melpa.
2. Somehow sort my own package archive for my personal code.

Over the past week or so I've worked on this approach. It initially started with me tackling item 1 above: I tidied up and submitted obfusurl.el, protocols.el, services.el, thinks.el and uptimes.el. This was a really helpful process in that it allowed me to brush up on my elisp and emacs knowledge. It's a good 15+ years since I last wrote any significant elisp code and things have moved on a little in that time.

Having done that I'd managed to move a handful of my own packages out of my local library of code, and so out of my .emacs.d repo, but it left me with the problem of what to do with the rest of it.

That's when I discovered package-x and:

,----[ C-h f package-upload-buffer RET ]
| package-upload-buffer is an interactive compiled Lisp function in
| ‘package-x.el’.
|
| (package-upload-buffer)
|
| Upload the current buffer as a single-file Emacs Lisp package.
| If ‘package-archive-upload-base’ does not specify a valid upload
| destination, prompt for one.
`----

(plus package-upload-file too, of course). This meant I could, in effect, start my own personal package archive and look at tackling issue 2 above.

This did give me one small problem though: how and where would I host the archive? I did consider hosting it on a DigitalOcean droplet, but that felt a little like overkill for something so simple. And then I realised: GitHub Pages! All I needed to do was keep the package archive in its own repo (which I would have done anyway) and then make the whole repo the source for a GitHub Pages site. A quick test later and... it worked!

So, by this point, I'd farmed some of my code off to melpa, and now had the rest of it in "delpa" (which I'd called my personal archive). I could now use the emacs package management system to install third party packages and also my own.

But I was still left with one issue: I was still holding the installed packages inside my .emacs.d repo by way of ensuring that all machines were in sync in terms of what was installed. Now I needed to work out how to solve that.

Around this time, as luck would have it, @tarsius had suggested I look at a package called use-package by @jwiegley. This was the bit I was missing.

With use-package I would be able to declare which packages I needed, how they'd be installed and, most important of all, it could be set to handle the fact that the package wasn't even installed. If a package is requested and there is no local install use-package is smart enough to get the emacs package system to install it.

So, given that, all I need to do was create a startup file that would declare the packages I use and I'd have a setup that should, once I'd cloned .emacs.d, self-install.

Except... yeah, one more issue. use-package isn't part of GNU emacs yet so I'd need a method of getting it to auto-install so it could then handle everything else. As it was that was as easy as adding this to the start of my init.el.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Make sure the package system is up and running early on.
(require 'package)
(add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/"))
(add-to-list 'package-archives '("delpa" . "http://blog.davep.org/delpa/"))
(package-initialize)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Bootstrap `use-package'
(unless (package-installed-p 'use-package)
  (package-refresh-contents)
  (package-install 'use-package))

With that in place I was able to nuke all my config on a machine, clone a fresh copy of .emacs.d (having now ceased tracking and storing the installed packages in that repo), run up emacs, wait a few moments and then find that everything was installed and ready to use.

Perfect!

My .emacs.d is now a lot smaller than it was before and, I think, even easier to maintain. Right now I think I'm very close to the ideal emacs config that I wanted to create when I did the complete rewrite a year ago.