Posts in category "Meta"

The job is finally done. After considering moving all the images in the blog over to WebP, and then finally getting the migration under way, I'm all done.

As I mentioned before: I've done this by hand, one post at a time, also adding missing covers as I go. The process went faster than I anticipated and I found that adding linting support to BlogMore really helped with this process. Each time I made a batch of changes I could run the linter to make sure I'd not broken any image links.

As for the result: I've brought the total size of images on the blog down from around 56MB to about 32MB, give or take (keep in mind the latter figure also includes all the WebP images I've added while blogging since I started this process). While I don't really have to worry so much about the storage costs of these images (I'm using GitHub Pages after all), overall, over time, there should be savings in the time it takes for readers to load any given page.

The new linting tool I've added to BlogMore has paid off already. While it is the case that it helped me find a couple of broken links and one or two other things to tidy, as I was working on the feature; by the time I released it, my blog was lint-free.

But last night I did a little more work on the slow migration of images over to WebP. As I've mentioned before: this is a process I'm doing by hand, one post at a time, for a couple of different reasons. The thing is, I'm in a part of my blog now where I was often posting about updates to projects I was working on (Tinboard being a good example), and the cover for all of the posts would be the same. To save having multiple copies of the cover image, all subsequent posts would point back to the first cover image¹.

So what was happening was, I'd have a cover image that got transitioned from PNG to WebP, and then the covers of a number of posts, later in time, would be broken. While I would get to them eventually, if I'd called it a day there and rebuilt my blog, those would have been published broken.

Using blogmore lint while making those changes yesterday evening alerted me to this right away.

I've finally made a proper start on the planned migration to webp for images. I did consider writing a tool that would go through and migrate the files, and update the Markdown, all in one go, but something about that makes me kind of nervous. While it wouldn't be a destructive approach (the whole blog is under version control after all), I just have this niggling feeling that I'd miss something and it would sit broken, unnoticed, for ages.

So instead I've decided to take a one-post-at-a-time approach, making the migration by hand. As well as having the benefit of letting me go slowly and check my work as I go, I can also do some tidying up of old posts. So while I do this I'm also going to tidy up obviously broken links when I notice them, and also remove embedded tweets (swapping to the simple blockquote version).

Another thing I'm doing is adding cover images where possible. I'd been running this blog for a long time before I started to use cover (it might be that I didn't start until I moved to Pelican). Since then I've tried to use it any time there's an appropriate image in a post. More recently, I added cover images to the graph view so they're even more useful now. Back-adding a cover to older posts will make them more appealing to discover in the graph because those older notes will acquire attention-grabbing thumbnails too.

One thing I wanted to do was have an easy way to keep track of where I'm up to in the migration. It's going to be a steady process that's going to take a few days, doing a few posts at a time. So to aid this I've added this to the Makefile of the blog:

cd content/extras/attachments
find -E ./ -iregex '.*\.(png|jpg|jpeg)$' | cut -d'/' -f2,3,4 | sort -u

With this I get a handy list of dates of posts that still have unconverted png or jpeg files.

Of course, for a wee while, this will not get to an empty list because I want to make sure some of the more recent posts still have their older images available as they might be in feeds out there. More recently I've only been using webp for images, so once the webp-using posts fill the main RSS and Atom feeds I can clean out the last of the bulkier images.

Ever since I kicked off the work on BlogMore I've had a renewed interest in writing on this blog (as you can probably tell from the stats and the calendar). But not just writing: also tweaking it, tidying it up, thinking about maintaining it into the future, thinking about the links and the categories and so on.

In doing so, I've also been looking at other folk who persist in keeping a blog, and especially those who maintain blogs built with static site generation tools, and in some cases I'm mildly envious of how far back some of them stretch.

When it comes to the world of blogging I was kind of late to the party. The first version was just a section of my self-developed website, hosted on www.davep.org. Don't go looking there for it now, it was long ago removed. In fact my personal website is mostly just a placeholder for what once was. The Wayback Machine still has a copy though, so I can see that the first blog post I wrote for my site was dated 2003-03-31.

I maintained this for a while, the engine for it all being some self-written PHP engine that was what could be best described as a dynamic static site (in other words it generated everything on request from underlying text files and HTML snippets because I had no wish to be faffing around with databases on a web host). Eventually though the blog side of this got to be too much trouble and I jumped over to Blogger.

I maintained that blog for quite a few years, with the first post being made in 2006 and the last in 2011. Sadly it's all quite broken now. I used to include a lot of images and, while some of them are embedded in the site itself, most were hosted on the older version of my website, as part of the photo gallery I also had there.

This all fell apart when I finally killed off the PHP version of my site and all the images were removed. Now the blog is a wasteland of broken image icons (not to mention a wasteland of broken external links -- so many of the sites I referred to back then have fallen off the net).

I hate this. I hate that thirty-something me was fired up enough to want to write stuff down and communicate to other people (and to future me) and it's all decayed. I especially dislike that the original version of my blog, now only stored on the Wayback Machine (and perhaps on a hard drive that I think is in a box somewhere in storage, perhaps also on some burnt-as-a-backup DVDs) is otherwise inaccessible. Much like I did with my original photoblog, I want to rescue this. I want to rescue all of this.

The technical challenges of teasing out the original posts from the Internet Archive and from Blogger aren't too great. Turning a bunch of HTML into Markdown isn't impossible either -- the library that I use in OldNews should do the job fine there. All that sort of work feels like a fun little challenge that will keep me amused for a few evenings.

There are two main things that cause me to pause when thinking about doing this.

The first is that some of those very old posts, as I mentioned above, link to places that don't exist and haven't existed for a long time. It raises the question: do I even care to preserve things that have no context any more?

The second is that many of the posts in the Blogger blog, as I mention, relied on images hosted on my old site. Right now I'm not actually sure where those photos are! While I took a backup of all the code and other data for www.davep.org when I did the big reboot (storing it all up on GitHub), I seem to have stripped out all of the photos. This makes sense as there was a lot of data there. Making sure I had a backup of those files feels like something I would do -- I hang on to all sorts of data -- but at the moment I can't locate them¹.

To make this work, for this to stand any chance of working, I need to pull them all back out from somewhere.

Will I do this? I don't know yet. The seed is there, the itch is waiting to be scratched. I look at the age span of this blog, and the calendar page, and think it could be really cool to really back-fill it from my older blogs. The graph might end up looking really funky.

On the other hand: am I just trying to preserve irrelevant things as a way to make work for myself (albeit "work" that is fun; after all coding is a hobby as well as a living).

On the gripping hand: if I can get the images back, a wasteland of links to sites that don't exist any more does, at the very least, provide a history of what was and is no longer.

For a good while now I've been pretty happy with the PageSpeed measurements of this blog, which in turn means I've been happy with the state of what's generated by BlogMore. I have pretty much everything that can be minified nice and minimal. At this point, the main thing that causes the speed measurement to fluctuate is image sizes.

I use a lot of PNGs on this blog. When I'm using images, they're almost always in posts that include screenshots, which in turn pretty much demand that I use a lossless format. When I take these screenshots I don't worry too much about the dimensions (within reason), and of course I don't really do anything to optimise how they'll work and appear on different display sizes. If I was to get too into that, it would add friction to writing something, and the whole point of this is to feel less friction when it comes to sitting at the keyboard.

So I've been living with the fact that some images can be pretty big. While I do make a point of using pngcrush on every image, it generally doesn't make a huge saving.

Then yesterday I read this post on Andy's blog and I suddenly realised what I had to do!

Borrowing from what Andy did, I used mogrify too, setting up this Fish abbr in my Fish configuration:

if type -q mogrify
    abbr -g mkwebp "mogrify -format webp -define webp:lossless=true -quality 100"
end

In my case, at least in the initial experiment, I decided to keep it all lossless. So far the results have been really good, cutting the image sizes down by a significant amount. For example, if I look at the images for yesterday's posts:

 90581 15 Apr 18:14 sl-overview.png
 33446 16 Apr 20:23 sl-overview.webp
392661 15 Apr 18:14 slstats-region-info.png
225392 16 Apr 20:23 slstats-region-info.webp
 36049 15 Apr 19:39 year-chart.png
 15590 16 Apr 20:23 year-chart.webp

That's a pretty reasonable saving!

So far all I've done is convert the few latest posts that make up the front page of my blog, just so I can see what impact it has. I'm getting improved load times on mobile, for sure.

There are a couple of downsides to this, of course.

1. Now I want to do the whole blog, so while I can easily go through and convert all the png files to webp, converting all the image markup in the Markdown files isn't quite so simple, and even if I do write something to automate it, I'm then going to want to review it to make 100% sure nothing has broken.
2. I can't just then remove all the png files to cut back on the space used by the generated site. The front page of the site has a feed, and all the categories have a feed each too. This means that there could be HTML out there from some of my oldest posts, referring to the png files, and just removing them will result in broken images.

Overall though, it might be worth doing at some point soon. Meanwhile, from now on, I think I'm going to replace my pngcrush step with a mkwebp step and just use webp instead of png now.

I guess I'm all modern now!

For about 2 and a half years now this blog has been built with Pelican. For the most part I've enjoyed using it, it's been easy enough to work with, although not exciting to work with (which I think is a positive thing to say about a static site generator).

There were, however, a couple or so things I didn't like about the layout I was getting out of it. One issue was the archive, which was a pretty boring list of titles of all the posts on the site. It would have been nice to have them broken down by date or something, at least.

Of course, there are lots of themes, and it also uses templates, so I could probably have tweaked it "just so"; but every time I started to look into it I found myself wanting to "fix" the issue by building my own engine from scratch.

Thankfully, every time that happened, I'd come to my senses and go off and work on some other fun personal project. Until earlier this week, that was.

The thing is... I've been looking for a project where I could dive into the world of "AI coding" and "Agents" and all that nonsense. Not because I want to abandon the absolute thrill and joy I still get from writing actual code as a human, but because I want to understand things from the point of view of people who champion these tools.

The only way I'm going to have an informed opinion is to get informed; the only way to get informed is to try this stuff out.

So, here I am, with my blog now migrated over to BlogMore; a project that gives me a blog-focused static site generator that I 100% drove the development of, but for which I wrote almost none of the code.

At the moment it's working out well, as a generator. I'm happy with how it works, I'm happy with what it generates. I also think it's 100% backwards-compatible when it comes to URLs and feeds and so on. If you do see anything odd happening, if you do see anything that looks broken, I'd love to hear about it.

As for this being a "100% AI" project, and how I found that process and how I feel about the implications and the results... that's a blog post to come.

I took lots of notes.

While I'm still in blog-tinkering mode (long may it last!), I thought it might be handy to keep a page kicking around that has links to the small collection of "dot file" repositories I have.

Like many people, I keep these in a central location (in my case up on GitHub) so that I can very quickly spin up a familiar work environment on a new machine (new machines are something that doesn't happen too often, but it's always good to be able to get going quickly when it does).

So, depending on browser type/size, either above here or off to the side, there should now be a permanent link to a page of links to those repositories.

As I look at it now it's actually surprising to me how much of my "comfortable" environment is encapsulated in so few tools, and configured with so few collections of files. There are other tools I use a lot too, but most of them either have their own sync systems, or they have so few configuration options (and are likely in a format that isn't easy to grab/store) that it's not worth the bother.

This feels like a good thing, really.

One thing that's not amongst all of this, partly because it's not that interesting, but also partly because the repository is private, is a single bash script called myenv. On a new machine, once I've got enough of a setup that I can clone from GitHub, I drag this down and run the script and most of the rest of the environment follows.

It's quite satisfying when I need to use it.

Well, it didn't take as long as I expected it to. Just yesterday morning I was giving Pelican a look over as a possible engine for generating my blog, having wanted to move away from Jekyll for a while now. Having tried it and liked what I saw to start with, I wrote about how I liked it and wondered how long it would take me to make the switch.

By the evening I was making a proper effort to get the switchover started, and just a wee while earlier, before writing this post, the switch was made!

The process of making the switch was roughly this (and keep in mind I'm coming from using Jekyll):

1. Made a branch in the repo to work in.
2. Removed all of the Jekyll-oriented files.
3. Decided to set up Pelican and related tools in a virtual environment, managed using pipenv.
4. Ran pelican-quickstart to kick things off and give me a framework to start with.
5. Renamed the old _posts directory to content.
6. Kept tweaking the hell out of the Pelican config file until it started to look "just so" (this is a process that has been ongoing, and doubtless will keep happening for some time to come).
7. Tried out a few themes and settled on Flex; while not exactly what I wanted, it was close enough to help keep me motivated (while rolling my own theme from scratch would seem fun, I just know it would mean the work would never get done, or at least finished).
8. Did a mass tidy up of all the tags in all the posts; something I'd never really paid too much attention to as the Jekyll-based blog never actually allowed for following tags.
9. Went though all the posts and removed double quotes from a lot of the titles in the frontmatter (something Jekyll seems to have stripped, but which Pelican doesn't).
10. Tweaked the FILE_METADATA to ensure that the slugs for the URLs came from the filenames -- by default Pelican seems to slugify the title of a post and this meant that some of the URLs were changing.

All in all I probably spent 6 or 7 hours on making the move; a lot of that involving reading up on how to configure Pelican and researching themes. The rest of it was a lot of repetitive work to fix or tidy things.

The most important aspect of this was keeping the post URLs the same all the way back to the first post; as best as I can tell I've managed that.

So far I'm pleased with the result. I'm going to live with the look/theme for a wee while and see how it sits for me. I'm sure I'll tweak it a bit as time goes on, but at the moment I'm comfortable with how it looks.

Since getting my blog editing environment set up on the "new" machine a couple of days back I've been thinking some more about moving away from Jekyll. Jekyll itself has served me well since I started this blog back in 2015, but I was reminded again when installing it on the Mac Mini that it's Ruby-based and I have very little understanding of how to get a good Ruby experience on macOS¹.

Having mentioned on Mastodon that I was thinking about finally looking at moving my blog management/generation to something new, and specifically something Python-based and ideally some sort of site generator, I got a few suggestions.

One that looks promising so far is Pelican. At first glance it seems to tick a few boxes for me:

• Python-based (so easy for me to grok in terms of installing, and also more chance of being hackable).
• Uses Markdown (curiously as an alternative, to reStructuredText, which looks to be the default).
• Does article-based stuff as well as page-based stuff.
• Lots of themes, and themes are Jinja2-based (I'm pretty familiar with Jinja thanks to my Django days and also using the library when kicking off ng2web).
• RSS feed generation.
• Syntax-highlighted code blocks.

While I'm not quite ready to dive in and make the move just yet (I am on a "muck about at home" holiday this week, but I've got enough planned without losing a day to rebooting my blog), I did do a quick experiment to see if Pelican would work for me.

Key to this is can I keep the URLs for all the posts the same? If I can't that's a non-starter.

Things got off to a good start with an easy install:

pipx install "pelican[markdown]"

I then used the pelican-quickstart to kick off a test site, copied in my existing Markdown files, dived into the docs and found how to configure the generated URLs and... yeah, within like 10 minutes I had a very rough version of my blog up and going.

It looked like garbage, the theme really wasn't to my taste at all, but the core of the blog was working.

I've nuked it all for now but a more considered evaluation is now on my TODO list. Things I'll need to drive into properly are:

• Find a base theme that's to my taste.
• Get Disqus working with it so that any old comments remain in place.
• Get my image/attachment layout back in place.
• Go through and tidy up all the tagging (which has been a mess with this blog because I never did get round to getting Jekyll to actually use tags).
• Figure out the best way to do the publishing to GitHub pages.
• Likely a bunch of other stuff I've not thought about yet.

But, yeah, for a brief "over first coffee of the day" tinker to see if I like it... I like!

Let's see how long it takes me to actually get around to making the switch. ;-)

So... erm... yeah... I did it again. I looked away for a moment and somehow almost 7 months passed without a post! It's so easily done too isn't it? While, when I revived this blog last year, I didn't make a point of intending to write lots and often, I had hope that I'd manage something at least once a week; perhaps at least once a month.

Ahh well.

There's been two main reasons why it's been quiet around here. The first is that my (now not so) new job keeps me busy (in a good way). It involves a reasonable amount of trekking into town and back (which I don't mind on the whole), and once I'm home in the evening I'm generally (but not always) done with the keyboard and desk.

The second reason, which is probably the dafter one, is that a bit earlier this year I finally upgraded my desktop setup from the 2019 Intel MacBook Pro I was using to a recently-released M2Pro Mac Mini (and what an upgrade!). How this plays into blogging being even more quiet is... I needed to set up jekyll again, and I'd forgotten how I got it running in the first place, so I kept putting off getting it going, and...

Well, this morning, I sat down with coffee, grepped the history on my previous machine, and got it running in like 5 minutes (of course).

So, here I am, back adding another blog post. I'm writing this as much to test that the setup works as anything else.

But also, this time, I'm going to try and make a promise to myself: I'm going to try and write more. I can and should write about anything. I can and should write short things as well as long things. I can and should remember that it's not about writing things that are going to be super important or anything like that, it's about just getting stuff down and creating and recording.

Note of course I said "try" and make a promise.

We'll see. ;-)