After having knocked up obs2nlm the other day I realised there was another source of sources available to me that might be fun to throw into NotebookLM: one or more of the Norton Guide files in my collection!

And so, after a wee bit of hacking on a wet Sunday afternoon, ng2nlm was born.

In terms of what it does, it's pretty much the same as obs2nlm, only it uses ngdb to read the contents of a Norton Guide and turn it into a single Markdown file which can then be used as a source in NotebookLM.

So far it seems to be working a treat.

Here I grabbed the guide to assembly language Norton Guide, and the DOS interrupts Norton Guide, turned them into sources, created a notebook and let it do its thing.

I can't vouch for how valid the answer to my question is -- it's a long time since I wrote any 8086 assembler code and it's a moment since I last had to dig into DOS interrupts and the like -- but I like the result.

Probably the biggest issue I see at the moment is with the code examples: it seems to have a habit of adding some sort of citation marker that links to nowhere when it's emitting code. I think this is down to the instructions I've given it in the source I output, at least that's what it's suggesting when I ask it:

Me: What are all the numbers in square brackets for?

NotebookLM: The numbers in square brackets are passage indices used to cite which specific part of the sources supports each statement [Instruction]. Within the source files themselves, these markers identify distinct entries or sections within the Norton Guide databases.

Given I am trying to encourage it to cite its sources, I think this is where the confusion comes from. In "normal" paragraphs it does do a good job of citing its source and linking to it:

so presumably, when it "thinks" it's outputting code it doesn't do the markup to actually do the linking, and so things end up looking a little confused.

If I ask that citations aren't included in the code, this does seem to have the desired effect:

So, given this, perhaps I can use the --additional-instructions switch for the particular source to encourage it to not add citation links to code? Or of course I could tweak ng2nlm itself to include that instruction to start with. While it's a little specific to one use case, Norton Guide files do tend to be coding-related so it could make sense.

Anyway, in the very unlikely event that you have a need to turn one or more Norton Guide files into sources to throw at NotebookLM or similar tools: ng2nlm exists.

I'm sure I've mentioned a couple of times before that I've become quite the fan of Obsidian. For the past few years, at any given point, I've had a couple of vaults on the go. Generally I find such vaults a really useful place to record things I'd otherwise forget, and of course as a place to go back and look things up.

But... even then, it's easy enough to forget what you might have recorded and know that you can even go back and look things up. Also I tend to find that I can't quite figure out a good way of getting a good overview of what I've recorded, over time.

Meanwhile: I've been playing around with Google's NotebookLM as a tool to help research and understand various topics. After doing this with my recent winter break coding project (more on that in the future) I realised I really should get serious about taking this approach with my Obsidian Vaults.

I'm sure this is far from novel, I'm sure lots of people have done similar things already; in fact I'd quickly dabbled with the idea a few months ago, had a bit of a laugh at some of the things the "studio" made of a vault, and promptly forgot about it.

This time though I got to thinking that I should try and take it a little more seriously.

And so obs2nlm was born.

The idea is simple enough: enumerate all the Markdown files in the vault, wrap them in boundary markers, add some instructions to the start of the file to help NotebookLM "comprehend" the content better, throw in a table of contents to give clues to the structure of the vault, and see what happens when you use the resulting file as a source.

So far it's actually turning out to be really helpful. I've been using it to get summaries regarding my work over the past 18 months or so and it's helped me to consolidate my thoughts on all sorts of issues and subjects.

It's not perfect, however. I've had it "hallucinate" some stuff when making things in the studio (most notably in the slide deck facility); for me though I find this an acceptable use of an LLM. I know the subject it's talking to me about and I know when it's making stuff up. This, in turn, makes for a useful lesson in how and when to not trust the output of a tool like this.

Having tested it out with a currently-active vault, I'm now interested to find out what it'll make of some of the archived vaults I have. Back in 2024 I wrote a couple or so tools for making vaults from other things and so I have a vault of a journal I kept in Journey for a number of years, a vault of a journal I'd kept before then in Evernote, and I also have a vault of all the tweets I wrote before I abandoned Twitter. I also have a vault that was dedicated to recording the daily events and thoughts of my time working at Textualize. It's going to be fun seeing what NotebookLM makes of each of those; especially the last one.

Anyway, if Obsidian is your thing, and if you are dabbling with or fancy dabbling with NotebookLM, perhaps obs2nlm will be handy for you.

As I've mentioned a few times on this blog, I've long had a bit of a thing for writing tools for reading the content of Norton Guide files. I first used Norton Guides back in the early 1990s thanks to the release of Clipper 5, and later on in that decade I wrote my first couple of tools to turn guides into HTML (and also wrote a Windows-based reader, then rewrote it, wrote one for OS/2, wrote one for GNU/Linux, and so on).

One tool (ng2html) got used by a few sites on the 'net to publish all sorts of guides, but it's not something I ever got into doing myself. Amusingly, from time to time, because I had a credit on those sites as the author of the conversion tool, I'd get random emails from people hoping I could help them with the topic of whatever guide they'd been reading. Sometimes I could help, often not.

From what I've recently been told two of the biggest sites for this sort of thing (they might even have been the same site, or one a copy of the other, I didn't really dive into them too much and wasn't sure who was behind them anyway) have long since gone offline. This means that, as far as I can tell, a huge collection of knowledge from the DOS days is a lot harder to get to, if it hasn't disappeared altogether.

This makes me kind of sad.

Edit to add: digging a little, one of the sites was called www.clipx.net and it looks to have long-since gone offline. It is on archive.org though. The other was x-hacker.org which, digging a wee bit more, seems to have been a copy of what was on clipx.net.

So I had an idea: having recently polished up my replacement for ng2html, why not use that to build my own site that publishes the guides I have? So I set about it.

There's one wrinkle to this though. While the other sites seemed to just publish every NG file they got their hands on, I'd prefer to try and do it like this: publish every guide I have in my collection that I have a licence or permission to publish; or as near as possible¹

Given all of this, norton-guides.davep.dev has been born. The repository that drives it is on GitHub, and I have a wiki page that lists all the guides I have that I could possibly publish, showing what I know about the copyright/licence of each one and what the publishing state is.

So with this, I'm putting out a call for help: if you remember the days of Norton Guide help files, if you have Norton Guide help files I don't have, and especially if you are the copyright-holder of any of these files and you can extend me the permission to open them up, or if you know the right people and can get me in touch with them, DROP ME A LINE!

I'd also love to have others join me in this... quest. So if you want to contribute to the repository and help build it up I'd also love to hear from you.

Back in the very early days of my involvement with Textualize, while looking for fun things to build to test out the framework and find problems with it, I created textual-astview. The idea was pretty simple: exercise Textual's Tree widget by using it to display the output of Python's abstract syntax tree module.

While the code still works, Textual has moved on a lot, as has my approach to building applications with Textual, and so I've been wanting to do a ground-up rewrite of it. At the same time I was also thinking that it might be interesting to build a tool that provides other ways of understanding how your Python source gets turned into runnable code; with this in mind I've ended up building a terminal-based application called DHV.

The idea is pretty straightforward: you type in, paste in, or load up, Python code, and you get a real-time display of what the resulting bytecode and abstract syntax tree would be.

If you've ever wondered what a particular bit of code looks like under the hood, or wondered if one particular approach to a problem is "more efficient"¹ than another, or just wondered to yourself if 1+1 ends up being a complex operation or simply gets turned into 2, this tool might be useful to experiment and see.

As of now DHV only works with Python 3.13. The main reason for this is that the Python dis module is a bit of a moving target and has had some noticeable interface changes over the past few versions. When I find some time I might work on making DHV a little more backward-compatible. But for now keep this in mind: when you're looking at the results for some code you're looking at what Python 3.13 (or later) would do, earlier Pythons may differ.

DHV is licensed GPL-3.0 and available via GitHub and also via PyPI. If you have an environment that has pipx installed you should be able to get up and going with:

pipx install dhv

If you're a fan of uv and friends you can install it like this:

uv tool install --python 3.13 dhv

Much like Norton Guide readers or the 5x5 puzzle, code that has fun with the Mandelbrot set is another one of my goto exercises. I've written versions in many languages, and messed with plots in some different environments, as varied as in VR on the web to wearable items.

Back in the early days of my involvement with Textualize I wrote a deliberately worst-approach version using that framework. The whole thing was about taking a really silly approach while also stress-testing Textual itself. It did the job.

Later on I did a second version that targets Textual. This time it did a better job and was the catalyst for building textual-canvas. This version was intended more to be a widget that happened to come with an example application, and while it was far more better than the on-purpose-terrible version mentioned above, I still wasn't 100% happy with the way it worked.

Recently I did some maintenance work on textual-canvas, cleaning up the repository and bringing it in line with how I like to maintain my Python projects these days, and this prompted me to look back at textual-mandelbrot and rework it too. Quickly I realised it wasn't really sensible to rewrite it in a way that it would be backward compatible (not that I think anyone has ever used the widget) and instead I decided to kick off a fresh stand-alone application.

Complexitty is the result.

Right now the application has all the same features as the mandelexp application that came with textual-mandelbrot, plus a couple more. Also it's built on top of the common core library I've been putting together for all my own terminal-based Python applications. As time goes on I'll add more features.

As with most of my recent TUI-based projects, the application is built with comprehensive help for commands and key bindings.

and there's also a command palette that helps you discover (and run) commands and their keyboard bindings.

Complexitty is licensed GPL-3.0 and available via GitHub and also via PyPI. If you have an environment that has pipx installed you should be able to get up and going with:

pipx install complexitty

It can also be installed with Homebrew by tapping davep/homebrew and then installing complexitty:

brew tap davep/homebrew
brew install complexitty

It pretty much all started with this:

 * Revision 1.1  1996/02/15 18:57:13  davep
 * Initial revision

That's from the rcs log for the source for w3ng, a tool I wrote so I could read Norton Guide files in my web browser, served by Apache, running on my GNU/Linux server in my office. The tool itself was written as a CGI tool (remember them?).

I believe I posted about this to comp.lang.clipper and pretty quickly some folk asked if it might be possible to do a version that would write the whole guide as a collection of HTML files for static hosting, rather than serving them from a cgi-bin utility. That seemed like a sensible idea and so:

 * Revision 1.1  1996/03/16 09:49:00  davep
 * Initial revision

ng2html was born.

Fast forward around a quarter of a century and I decided it would be fun to write a library for Python that reads Norton Guide files, and a tool called ng2web was the first test I wrote of it, designed as a more flexible replacement for ng2html. I've tweaked and tinkered with the tool since I first created it, but never actually "finished" it.

That's changed today. I've just released v1.0.0 of ng2web.

If turning one or more Norton Guide files into static websites seems like the sort of thing you want to be doing, take a look at the documentation.

ng2web is licensed GPL-3.0 and available via GitHub and also via PyPi. If you have an environment that has pipx installed you should be able to get up and going with:

$ pipx install ng2web

It can also be installed with Homebrew by tapping davep/homebrew and then installing ng2web:

$ brew tap davep/homebrew
$ brew install ng2web

I've just released AgiNG v0.3.0. The main focus of this release was to get some searching added to the application. Similar to what I added to WEG back in the day, I wanted three types of searching:

• Current entry search.
• Current guide-wide search.
• All registered guides-wide search.

The current entry search is done with a simple modal input, and for now the searching is always case-insensitive (I was going to add a switch for this but it sort of felt unnecessary and I liked how clean the input is).

The search is started by pressing /, and if a hit is found n will take you through all subsequent matches.

As always, if you're not sure of the keys, you'll find them in the help screen or via the command palette:

Guide-wide and all-guide searching is done in the same dialog. To search guide-wide you enter what you want to find and untick "All Guides".

With that, the search will stick to the current guide.

As will be obvious, searching all guides that have been registered with AgiNG is as simple as ticking "All Guides". Then when you search it'll take a walk through every entry of every guide you've added to the guide directory in the application.

Global searching is accessed with Ctrl+/ or via the command palette.

With this added, I think that's most of the major functionality I wanted for AgiNG. I imagine there's a few more tweaks I'll think of (for example: I think adding regex search to the global search screen could be handy), but I don't think there's any more big features it needs.

AgiNG can be installed with pip or (ideally) pipx from PyPi. It can also be installed with Homebrew by tapping davep/homebrew and then installing aging:

$ brew tap davep/homebrew
$ brew install aging

The source is available on GitHub.

It seems I really do still have this need to create new terminal-based projects at the moment. There's been Braindrop, then Peplum, then after that came Hike. While I'm still tweaking and adding to them, and also using them to refine a wee library I'm building up that forms the core of my latest apps, I felt I still had this one app that I needed to finally build.

Since the 1990s I've had this mild obsession with building tools for maintaining access to Norton Guide files. I've written readers for GNU/Linux (which also works on macOS too), OS/2, Windows, GNU Emacs, and also or the web (in multiple incarnations). Those builds have covered a few languages, including C, C++, Pascal, JavaScript and Emacs Lisp.

I'd never written a Python library or application for it though.

So when I first saw Textual mentioned in passing on Twitter a few years back, way back in the 0.1 days, I thought that could be the thing that would push me over the edge. In anticipation of that, back in 2021, I initially developed ngdb.py. This is a library that provides the core Norton Guide reading code for Python applications and could form the basis for other tools.

As a test for this I then wrote ng2web (which works, but I think still needs a bit of tidying up -- something I'm aiming to do in the next few weeks).

Meanwhile, the journey with Textual itself kicked off, happened, and came to an end; and yet somehow I'd never got round to building the thing I'd initially looked at Textual for: a terminal-based Norton Guide reader that looked nice and modern (by terminal standards). When I initially joined Textualize the owner had actually said they wanted me to build this as test of the framework, to essentially start out by employing me to create some Free Software that would help dogfood the library, but that seemed to get forgotten.

Fast forward to the start of this month and I finally felt it was time to tackle this. The result is AgiNG¹.

As of v0.1.0 it has most of the features you'd expect from a usable Norton Guide reader, including:

• An ability to add guide files to an in-application directory.
• The ability open and navigate a guide.
• Full see-also support, etc.
• Full translation of characters as were under MS-DOS into the terminal.
• The ability to copy entry text or source to the clipboard.
• The ability to copy save entry text or source to a file.
• Access to a guide's credits.

I still need to write some proper documentation for the application, but meanwhile all commands and key shortcuts can be discovered either via the help screen:

or by pulling up the command palette:

Hopefully the workings of the application will be fairly obvious to anyone who is familiar with Norton Guide files; if anything isn't making sense I'm more than happy to answer questions or take suggestions for improvements.

One wee feature I want to call out, that I felt was important to add, was a "classic view" facility. The thing with Norton Guide files is they were mostly created in the very late 1980s and early-to-mid 1990s. People would often get creative with the colouring within them, but in many cases the colouring assumed the default Norton Guide application. Its colours were white text on a blue background. So sometimes other colouring was done assuming that background.

You can see an example of this here, with an entry in a guide being viewed using the default textual-dark theme:

Notice the colouring in the syntax section. This is more obvious if the application is switched to one of the light themes:

With a nod to this issue in mind, I added the "classic view" for entries (which is a sticky setting -- turn it on and it stays on until you turn it off):

A little hard on the eyes, I think, but also filled with nostalgia!

Talking of themes, all the usual application themes are available, here's a wee selection:

AgiNG is licensed GPL-3.0 and available via GitHub and also via PyPi. If you have an environment that has pipx installed you should be able to get up and going with:

$ pipx install aging

It can also be installed with Homebrew by tapping davep/homebrew and then installing aging:

$ brew tap davep/homebrew
$ brew install aging

Expect to see more updates in the near future; as with other recent projects this is very much something I'm going to be dabbling with and improving as time goes on.

Following on from my post this morning, regarding the problem I was having with _mkdocstrings.css being 404 any time I deployed by documentation, build with mkdocs/mkdocstrings, to GitHub Pages...

It's come to light that I was doing this on hard mode, pretty much.

While trying to figure out the best way of deploying the docs, I'd stumbled on ghp-import and had been using that. On the other hand, MkDocs has it's own command for doing the same thing: mkdocs gh-deploy.

Timothée pointed out to me that he never runs into this problem, but he used this command. As it turns out, if you use mkdocs gh-deploy it creates the .nojekyll file by default.

And how does it do this? It uses the ghp-import code and uses a switch it has to achieve exactly this. In fact... the command line version even has a switch for it!

-n, --no-jekyll      Include a .nojekyll file in the branch.

This is off by default, when you run the command itself, but I wish I'd noticed this when I was first experimenting. O_o

Anyway, thanks to Timothée's pointers, I've now managed to simplify how I build and publish the docs from textual-fspicker, and I'll apply this to other projects too.

I've just made a wee update to textual-fspicker, my dialog library for Textual which adds FileOpen, FileSave and SelectDirectory dialogs. There's no substantial change to the workings of the library itself, but I have added something it's been lacking for a long time: documentation!

Well... that's not quite true, it's always had documentation. I'm an avid writer of Python docstrings and I make a point of always writing them for every class, function, method or global value as I write the code. As such the low-level "API" documentation has always been sat there ready to be published somehow, eventually.

Meanwhile the description for how to use the library was mostly a pointer to some example code inside the README. Not ideal, and something I really wanted to improve at some point.

Given I'm still on a bit of a coding spree in my spare time, I finally decided to get round to using the amazing mkdocstrings, in conjunction with mkdocs, to get some better documentation up an running.

The approach I decided to take with the documentation was to have a page that gave some general information on how to use the library and then also generate low-level documentation for the all the useful content of the library from the docstrings. While latter isn't really useful to anyone wanting to use the library in their own applications, it could be useful to anyone wanting to understand how it hangs together at a lower-level, perhaps because they want to contribute to or extend the library in some way.

While writing some of the general guide took a bit of work, of course, the work to get the documentation up and running and generating was simple enough. The effort comes down to 3 rules in the Makefile for the project:

##############################################################################
# Documentation.
.PHONY: docs
docs:                    # Generate the system documentation
    $(mkdocs) build

.PHONY: rtfm
rtfm:                    # Locally read the library documentation
    $(mkdocs) serve

.PHONY: publishdocs
publishdocs: docs        # Set up the docs for publishing
    $(run) ghp-import --push site

The rtfm target is useful for locally-serving the documentation so I can live preview as I write things and update the code. The publishdocs target is used to create and push a gh-pages branch for the repository, resulting in the documentation being hosted by GitHub.

A wee problem

NOTE: I've since found out there's an easier way of fixing the issue.

This is, however, where I ran into a wee problem. I noticed that the locally-hosted version of the documentation looked great, but the version hosted on GitHub Pages was... not so great. I was seeing a load of text alignment issues, and also whole bits of text just not appearing at all.

Here's an example of what I was seeing locally:

and here's what I was seeing being served up from GitHub Pages:

As you can see, in the "bad" version the func label is missing from the header, and the Parameters and Returns tables look quite messy.

I spent a little bit of time digging and, looking in Safari's console, I then noticed that I was getting a 404 on a file called _mkdocstrings.css in the assets folder. Problem found!

Only... was it though? If I looked in the gh-pages local branch the file was there (and with fine permissions). If I looked in the remote branch, it was there too. Thinking it could be some odd browser problem I even tried to grab the file back from the command line and it came back 404 as well.

At this point it was getting kind of late so I decided I must have screwed up somehow but I should leave it for the evening and head to bed. Before doing so though I decided to drop a question into the mkdocstrings discussions to see if anyone could see where I'd messed up.

As it turns out, it looked like I hadn't messed up and the reply from the always super-helpful Timothée was, in effect, "yeah, that should work fine". At least I wasn't the only one confused.

Fast forward to this morning and, with breakfast and coffee inside me, I decided to try and methodically get to the bottom of it. I wrote up the current state of understanding and looked at what might be the common cause. The thing that stood out to me was that this was a file that started with an underscore, so I did a quick search for "github pages underscore" and right away landed on this result.

Bingo!

That had to be it!

A little bit of testing later and sure enough, the documentation hosted on GitHub Pages looked exactly like the locally-hosted version.

So, TIL: by default sites hosted by GitHub Pages will pretend that any asset that starts with an underscore doesn't exist, unless you have a .nojekyll in the root of the repository, on the gh-pages branch (or whatever branch you decide to serve from).

To make this all work I added .nojekyll to docs/source and added this to mkdocs.yml:

exclude_docs: |
  !.nojekyll

All done!

And now I've worked out a simple workflow for using mkdocs/mkdocstrings for my own Python projects, in conjunction with GitHub Pages, I guess I'll start to sprinkle it over other projects too.