Posts in category "AI"

I guess it was inevitable¹, but yesterday GitHub announced a new opt-out approach to learning from people's interactions with Copilot. I don't have anything novel or insightful to say on this switch, and I'm sure folk with better-informed opinions have already rushed out posts and articles about this, but I did want to jot down just how curious I am to see this roll out.

For starters: for me this feels like one of those things that will get a lot of backlash, and in a day or so GitHub will say they're pausing rolling this out while they reevaluate this approach². Then, eventually, they roll it out anyway after a "period of consultation with the community". That sort of thing.

I've not read further this morning, but before going to bed last night it wasn't a happy time in the comments section of the FAQ. I can also see why some would be cynical about this change, given the tone of some of the questions and answers in that FAQ. I'll hand it to them: they're pretty candid and honest with the FAQs, but kinda yikes too.

Here's the key thing I'm curious about, and which I'll be thinking about and watching for movement on in the next few days: all the talk here seems to be about protecting the privacy of the proprietary code of businesses³. That... is understandable, from a business point of view, from a commercial adoption point of view, from a "we want all software engineering departments to use Copilot" point of view. But how the heck are they really going to manage that?

In the comments in the FAQ this explanation stood out:

We do not train on the contents from any paid organization’s repos, regardless of whether a user is working in that repo with a Copilot Free, Pro, or Pro+ subscription. If a user’s GitHub account is a member of or outside collaborator with a paid organization, we exclude their interaction data from model training.

This seems somewhat unclear to me. Let's walk this one through for a moment: my GitHub account is a member of a "paid organisation". My account is also my account, for my personal code, I've had it a long time and it's filled with a lot of FOSS repos and I keep adding more. So which scenario is the right one here?

1. Because I'm currently a member of at least one "paid organisation" I'm always opted-out of this training no matter how the opt in/out setting is set and no matter what code I work on?
2. Because I'm currently a member of at least one "paid organisation" I can opt in when working on code that is from a repository which is mine, but I'm opted out when I'm working on code from a repository belonging to the paid organisation?

I think it reads like it's #1. But then that seems rather odd to me because, if I go and look at my settings right now, I can elect to opt in/out of this training system. If the correct reading is #1 why not just disable that setting altogether and say below it that I'm opted-out because I'm part of a paid organisation?

Which sort of suggests we should perhaps read it as #2? If that, that raises all sorts of questions. How would Copilot know I'm working on code from such a repository? Sure, it's not impossible to infer if I am working within the context of a given repository, doing some fun stuff to work out the origin and so on, but it feels messy. It also feels like a scenario that could end up being incredibly leaky. It really would not be difficult to run into a scenario where I'm working on some non-Free code but in an environment where the licence isn't clear, or where it appears that the licence⁴ would permit such training.

Or... perhaps there's a #3, or a #4, or so on, that I've not even considered yet. The fact that software engineering departments suddenly have to start thinking about this issue (yes, I know, it's been a background issue for a while but this really drags it out into the open) is going to make for a few interesting weeks, assuming people care about where their code ends up.

Who knows. Perhaps, in some strange way, this is how all software ends up being Free.

Last night, while tinkering with another BlogMore feature request, I ran into the sudden rate limit issue again. As before, there was no warning, there was no indication I was getting close to any sort of limit, and the <duration> I was supposed to wait to let things cool down was given as <duration> rather than an actual value.

So this time I decided to actually drop a ticket on GitHub support. Around 12 hours later they got back to me:

Thanks for writing in to GitHub Support!

I completely understand your frustration with hitting rate limits.

As usage continues to grow on Copilot — particularly with our latest models — we've made deliberate adjustments to our rate limiting to protect platform stability and ensure a reliable experience for all users. As part of this work, we corrected an issue where rate limits were not being consistently enforced across all models. You may notice increased rate limiting as these changes take effect.

Our goal is always that Copilot remains a great experience, and you are not disrupted in your work. If you encounter a rate limit, we recommend switching to a different model, using Auto mode. We're also working on improvements that will give you better visibility into your usage so you're never caught off guard.

We appreciate your patience as we roll out these changes.

So, in other words: expect to be rate limited more on this product we're trying to get everyone hooked on and trying to get everyone to subscribe to.

Neat.

I especially like this part:

We're also working on improvements that will give you better visibility into your usage so you're never caught off guard.

You know, if it were me, if I wanted to build up and keep goodwill for my product, I'd probably do that part first and communicate about it earlier rather than later.

I guess this is why I don't hold the sort of position that gets to make those decisions.

I don't know if it's just that GitHub Copilot is having a bad time at the moment, or if I've run into a genuine problem, but all isn't well today. After merging last night's result I kicked off another request related to a group of changes I want to make to BlogMore. It's a little involved, and it did take it a wee while to work on, but mostly it got the work done.

Again, as I said in the earlier post, I won't get into the detail of these changes yet, but they're fairly extensive and do include some breaking changes, so it's probably going to take a wee while to have it all come together. Claude's first shot at the latest change was almost there but with the glaring bug that it did all the work but didn't actually add the part that reads the configuration file settings and uses them (yeah, that's a good one, isn't it?).

So I asked it to fix it. It went off, worked on the issue, and then suddenly...

This surprised me. After the past few weeks I've had sessions where I've requested it do things way more frequently than this morning. I'm also nowhere near out of premium requests either:

While the error, as shown, might be valid and might be down to my actions, it's massively unhelpful and doesn't really explain what I did to cause this or even how I can remedy it. This is made all the more frustrating by the fact that it seems to be saying I need to wait <duration> to try again. Yes, literally a placeholder of <duration>. O_o

One thing is for sure: this is another useful experiment in the current experiment. It's worth having the experience of having the tool screw with the flow. It doesn't come as a surprise, but it's a good reminder that using an agent that is hosted by someone else means you fully rely on their ability to keep it working, the whims of their API limits, and perhaps even your ability to pay.

Yesterday evening, after dinner, but just before loading up a game released just a few hours earlier, I decided to kick off a change to BlogMore. Part of the ongoing experiment is this convenience aspect: where I can get some work going and then go and do something entirely different.

The nature of the change itself isn't important (I'll write about it when I release an update), but something that happened is. As usual, I did the prompt as an issue, and assigned it to Copilot. The first thing that was curious was that, after around 5 minutes, despite it having added the 👀 reaction (which it seems to use to indicate the agent has seen it has work to do), nothing happened. I didn't see an agent kick off doing the work. Eventually I gave up waiting and opened a Copilot session and asked it to set about dealing with the issue I'd raised.

It did as I requested, but apparently alongside another agent which had suddenly started doing the exact same work. Thinking it was a glitch (it's not like GitHub hasn't been having some trouble of late) I stopped the newest one and let the "original" go about its work.

A wee bit later, just before I started up my game and the stream, I checked in on how it was doing. It had finished, but in a quick test, I noticed a small bug. I prompted it to fix the problem, closed the tab, and went about the real business of killing bugs.

Fast forward a couple of hours, I was done with getting my arse kicked on Klendathu, I packed away my controller and headset and opened GitHub again to see where Copilot was at. It wasn't good:

@davep The model claude-sonnet-4.6 is not available for your account. This can happen if the model was disabled by your organization's policy or if your Copilot plan doesn't include access to it.

You can try again without specifying a model (just @copilot) to use the default, or choose a different model from the model picker.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: 79b81d32-e26a-4898-bd41-4bba088d08f6

Wait... what? I've been using this for weeks now, as best as I can tell I've generally been making all the changes and improvements using Claude Sonnet 4.6; there's never been an issue. Then, suddenly, in the middle of a PR, I don't have it?

Quickly checking elsewhere, sure enough, I had access to almost no models. I can't remember what there was, but it wasn't much and all the Claude ones had disappeared. Even if I tested in the Copilot CLI¹ I saw a very limited set of models.

Around this time I had two reactions: one was something like "cool, this is an important part of the experiment, knowing how it goes if your models are taken from you", the other was a curious "but Claude and I have an understanding about this codebase I can't trust some other model I've not been using".

As it was getting late into the evening and I still wanted to watch an episode of Stargate SG-1 (yes, I'm doing a full rewatch given it's on Netflix) I closed the MacBook and decided to check in on the issue in the morning.

Fast forward one SG1 episode later and, just before I headed to bed, I decided to do a quick search. While it could be a problem with my own account, it felt like it was more of a general issue. It was more of a general issue. At that point (around 23:20 UTC), checking in the GitHub app on my phone, I could see that some, if not all, of the models I was used to seeing, had come back.

As of this morning, as of time of writing, it's all looking back to how it was.

All of which is a great reminder, and something useful in the experiment: what does happen when some third party takes away the models you're using to get your work done? In the time I've been building up BlogMore I've come to trust the quality of Claude Sonnet (in the sense that I know when and where I have to pay closer attention to what it's done, and where it'll very likely have done a pretty good job²), so finding that I'd possibly have to switch to some other model that I've had no experience with... I genuinely had a moment of concern about how and where I was going to take BlogMore.

Ultimately it's not actually a serious concern for me: while I aim to maintain it for a very long time to come (it is how I'm creating the site for this blog after all), BlogMore isn't that critical. Moreover, I know I could cease using Copilot to create and maintain it and I could tidy up the code and hand-update and hand-manage it. There's a reason why I decided to really dive into this using a language I'm extremely confident with.

But for those applications some might now be relying on, developed by someone keen but as yet unskilled, what way forward for them if such a situation were to happen and not be resolved?

Just recently I saw a post go past on Mastodon, complaining about the author's perception that there was a breakdown of trust in the FOSS world, in respect to the use of AI to work on FOSS projects, or at least the willingness to accept AI-assisted contributions. The post also highlighted the author's reliance on FOSS projects and how they're driven by ethical and financial motivations (some emphasis was placed on how they had no money to spend on these things so it wasn't even necessarily a choice to FOSS up their environment).

This was, of course, in response to the current fuss about how Vim is being developed these days.

I don't want to comment on the Vim stuff -- I've got no dog in that fight -- but something about the post I mention above got me thinking, and troubled me.

Back when I first ran into the concept of Free Software, before the concept of Open Source had ever been thought of, I can remember reading stuff opposed to the idea that mostly worked along the lines of "you get what you pay for" -- the implication being that Free Software would be bad software. I think it's fair to say that history has now shown that this isn't the case.

But... I think it's fair to say that you do get what you pay for, but in a different sense.

If your computing environment is fully reliant on the time, money and effort of others; reliant on people who are willing to give all of that without the realistic expectation of any contribution back from you; I feel it's safe to say that you are getting a bloody good deal. To then question the motivations and abilities of those people, because they are exploring and embracing other methods of working, is at best a bad-faith argument and at worst betrays a deep sense of entitlement.

What I also found wild was, the post went on to document the author's concerns that they now have to worry about the ability of FOSS project maintainers to detect bad contributions. This for me suggests a lack of understanding of how non-trivial FOSS projects have worked ever since it was a thing.

I mean, sure, there are some projects that are incredibly useful and which have a solo developer working away (sometimes because nobody else wants to contribute, but also sometimes because that solo developer doesn't play well with others -- you pick which scenario you think is more healthy), but for the most part the "important" projects have multitudes working on them, with contributions coming from many people of varying levels of ability. The point here being that, all along, you've been relying on the discernment and mentoring abilities of those maintainers.

To suggest they're suddenly unworthy of your trust because they might be "using the AIs" is... well, it feels driven by dogma and it reads like a disingenuous take.

Don't get me wrong though: you are right to be suspicious, you are right to want to question the trust you place in those who donate so much to you; almost always this is made explicit in the licence they extended to you in the first place. But to suggest that suddenly they're unworthy of your trust because they're donating so much value to you in a way you don't approve of...

...well, perhaps it's time for you to pay it back?

Yesterday, while looking at starting to post to my photoblog again, I noticed I'd missed a trick when I added the first and second sets of photos when I created seen-by.davep.dev: I'd left off any cover frontmatter from all of the posts!

While I doubt it's super important -- I can't imagine people will be sharing photos from the blog after all, especially not older ones -- it felt like I'd failed to use a useful feature that I'd made sure BlogMore had.

This was obviously easy to fix. I could just write a tool that would go through all of the Markdown files, find the image that is being displayed in the post, pull out the path to the file, and add a cover to the frontmatter. Not exactly the hardest thing to write, but kind of boring for a one-off fix.

So this felt like another good time to get Copilot to do the hard work. Liking this plan, I wrote an issue for the prompt and set it to work.

The result was unexpected, and in retrospect this is how I should have approached it in the first place; Copilot wrote the script, then ran it, and then submitted the PR including the script and all of the changes after running it! It's like it was super eager to do the fix so went ahead and just did it.

This, for me, highlights a trick/approach I'm still not fully mindful of: where I might once have written some throwaway code to do a job, run the code, and then made use of the result, when it comes to using an agent I always have the option of saying what I want done to the content of the repository and just let it do it.

When I'm next faced with a problem like this, I think this is the approach I'll take: rather than ask it to write the tool to do the work, I'll just say what the work is I want done and let it go about it. This feels like where an agent should shine and where it's really useful.

Meanwhile I can get on with the fun stuff.

While I've written a lot of documentation in my life, it's not something I enjoy. I want documentation to read well, I want documentation to be useful, I want documentation to be accurate. I also want there to be documentation at all and sometimes the other parts mean it doesn't get done for FOSS projects¹.

When I started the experiment that is BlogMore, I very quickly hashed out some ideas on how it might generate some documentation, and the result was okay. In fact, if anything, it was a bit too much and there was a lot of repeated information.

So, this morning, before I sat down for the day's work, I quickly wrote an issue that would act as a prompt to Copilot to rewrite the documentation. This time I tried to be very clear about what I wanted where, but also left it to work out all the details.

The result genuinely impressed me. While I'll admit I haven't read it all in detail (and because of this have left the same warning right at the start), on the surface it looks a lot clearer and feels like a better journey to learning how to use BlogMore.

blogmore.davep.dev hosts the result of this.

I have a plan to work through the documentation and be sure it's all correct and all makes sense, but if it's as correct and as useful as I think, I might have to consider the idea of taking this approach more often. Writing down the plan for the documentation and then letting it appear in the background while I get on with my actual day makes a lot of sense.

I fear I might be warming to these tools, a little bit. :-/

One thing I've been noticing with my current experiment with GitHub Copilot is that it seems to know Python well enough to write code that gets the job done, and sometimes it knows it well enough to write more modern idiomatic Python code, but it also seems to write the inelegant version of it.

It's hard to pin down exactly, and of course it's a matter of taste (my idea of elegant might burn someone else's eyes), but on occasion, as I review the code, I find things that make me go "ugh".

Here's an example: there's a function that Copilot wrote to extract the first non-markup paragraph of an article (so that it can be used as a page description). One thing it needs to do is skip any initial images, etc. It takes a pretty brute force approach of looking at the start of each stripped line, but it gets the job done -- I can't really argue with that.

But here's how it does it:

# Skip markdown image syntax
if stripped.startswith("!["):
    continue

# Skip markdown linked image syntax ([![alt](img)](url))
if stripped.startswith("[!["):
    continue

# Skip HTML img tags
if stripped.startswith("<img"):
    continue

Now, this is good: it's using startswith. There are less-elegant approaches it could have used so I'll give it bonus points for using that method. The thing is though, it's testing each prefix one string at a time, pretty much rolling out a load of boilerplate code.

What bothers me here is that startswith will take a tuple of strings to test for. I find it curious that the generated code is idiomatic enough to know that startswith is a sensible option here, but at the same time it still writes the list of things to test out in a long-winded way.

This is exactly the sort of thing I'd call out in a normal code review. Technically, if this wasn't mostly a "let's see how it goes about this with minimal input from me" experiment, I'd have called it out here too (as an experiment, I might go back prompt it to "think" about this).

If I ever find myself using this sort of tool for generating code in a work setting, this is exactly the sort of thing I'll be watching for.

My experiment with coding with an agent continues, and from the point of view of the usability of the results, I'd say it's going well. From the point of view of learning more about this sort of tool and the pitfalls: I'd also say it's going well.

The one thing I'm really starting to get an appreciation for is just how much it employs a sort of brute force and ignorance approach to getting to a solution for some sort of problem. I noticed this on and off as I watched the agent's chat, as it worked away.

One example I enjoyed was when it was trying to add a new dependency to BlogMore. For reasons I should probably try and understand¹ it knew it should be using uv to add a dependency and it knew how to use it to update the lock file, but it didn't have uv to hand:

The lock file (uv.lock) needs to be updated to include rcssmin. Since uv isn't available, I'll need to note this in the PR. The lock file should be generated by uv when the pyproject.toml is updated.

So it decided the best solution was probably to... "hand-edit" the lock file.

The lock file needs to be updated with rcssmin. Since uv is not available in this environment, I need to update it manually or find another way.

I need to update the uv.lock file to include rcssmin. The uv.lock file needs to include:

1. The rcssmin package entry with hashes
2. The rcssmin dependency in the blogmore package section

Let me look at the PyPI page for rcssmin 1.2.2 to get the hash.

It then grabs rcssmin from PyPI and checks out the information it wants; after which it goes:

I need to update the uv.lock file with the rcssmin package information. The uv.lock file uses SHA256 hashes. Let me look at the existing format in more detail to understand what I need to add, and then I'll query PyPI for the hashes.

Actually, let me think about this differently. The uv.lock file is a generated file, and updating it manually is error-prone. The proper way to update it is to run uv sync or uv lock. Since uv is not available in this environment, I have two options:

1. Let the CI update the lock file (risky if CI fails on the locked version)

So... around this point it has some sort of plan for hand-patching the lock file, but also recognises that this is a dodgy approach and it should be using uv to update the lock file.

I also really like the part where it says it has two options, then goes on to provide an exhaustive list of... one.

After this it runs this code:

import json, urllib.request
url = 'https://pypi.org/pypi/rcssmin/json'
try:
    with urllib.request.urlopen(url, timeout=10) as f:
        data = json.load(f)
    v = '1.2.2'
    releases = data['releases'].get(v, [])
    for r in releases:
        print(f'filename: {r[\"filename\"]}')
        print(f'  url: {r[\"url\"]}')
        print(f'  sha256: {r[\"digests\"][\"sha256\"]}')
        print(f'  size: {r[\"size\"]}')
        print(f'  upload_time: {r[\"upload_time_iso_8601\"]}')
except Exception as e:
    print(f'Error: {e}')

after which it concludes:

Now I have all the information I need to update the uv.lock file. I need to add:

1. The rcssmin package entry in the lock file
2. Add rcssmin to the blogmore package dependencies

I mean... okay, I guess. Again, I'll admit that I'm still getting used to setting this sort of thing up and working with it (and keep in mind none of this was done locally; this is all via GitHub Copilot, up on GitHub), but it's kind of funny and wild that the approach taken was to recognise that using uv to remake the lock file was the best approach so... let's hand-edit the lock file.

But that's not the best part. This is where it gets really fun. It had decided that the lock file must be in alphabetical order. Okay, sure, why not:

The lock file has packages sorted alphabetically, so rcssmin should go between pygments and python-dateutil (since "r" comes before "py" actually... no, wait - "r" comes after "p" in the alphabet. Let me check the order again).

Since "r" comes before "py."

Since "r" comes before "py."

o_O

O_o

Extra credit here for giving me a good giggle. I really appreciate the way that it catches itself mid-sentence and remembers how the alphabet actually works.

As to the outcome of all of this? Sure, the feature I wanted to add got added; it worked away and got to a working solution in the end. But the route it took was, I think it's fair to say, a "brute force and ignorance" approach.

I've not been spending too much time reading the agent's own chatter, but when I have I've found myself amused by the dead ends it'll wander down and then work its way back out. There is, without question, a recognisable process here: I believe it would be a dishonest developer who says they've never had times in the past, or just one of those off days, where they've fallen down a rabbit hole of a solution, only to realise it's the right solution implemented in the worst possible way. There's also a resemblance here to how more junior developers work a problem until they really develop their skills.

I think I'm going to keep an eye on the agent chat a bit more from now on. While I imagine things will only improve as these tools improve, for the moment it's a good source of coding comedy.

This morning, with my experiment with Copilot having settled down a little bit, I thought I might try and use the result for another little experiment. For a long time now I've maintained a (currently lapsed) photoblog. It was always very much tied to "the site formerly known as twitter" and, since I fully erased my account after the site turned into a Nazi bar, I've not done anything to update how it gets populated.

So I got to thinking: I have a full backup of all the images in a couple of places; perhaps with a bit of coding (and some help from "the AIs") I can revive it using BlogMore?

I tinkered for a wee bit and mostly got something going (albeit I'm going to have to do some work to back-port the actual dates and times of some earlier images, and there's a load of work to do to somehow pull out all the tags). But then I hit a small hitch.

When building BlogMore I made the decision to let it write both the code and the documentation. It documented a couple of features I never asked for, but which seemed sensible so I never questioned. On the other hand neither did I test them at the time (because they weren't important to what I needed).

It's the exact reason I added this warning at the start of the documentation:

With this warning in mind, and with the intention of working through the documentation and testing its claims, I decided to test out one of the features when building up the new photoblog.

Whereas with this blog I keep all the posts in a flat structure, this time around I thought I'd try out this (taken from the Copilot-generated BlogMore documentation):

BlogMore is flexible about how you organise your posts. Here are some common patterns:

Flat structure (all posts in one directory):

posts/
  ├── hello-world.md
  ├── python-tips.md
  └── web-development.md

Note: Files can be date-prefixed (e.g., 2026-02-18-hello-world.md) and BlogMore will automatically remove the date prefix from the URL slug. The post will still use the date field from frontmatter for chronological ordering.

Organised by date:

posts/
  ├── 2024/
  │   ├── 01/
  │   │   └── hello-world.md
  │   └── 02/
  │       └── python-tips.md
  └── 2025/
      └── 01/
          └── web-development.md

Organised by topic:

posts/
  ├── python/
  │   ├── decorators.md
  │   └── type-hints.md
  └── web/
      ├── css-grid.md
      └── javascript-tips.md

Using the hierarchy approach, especially with dates, seemed ideal! I'd drop the actual images in such a hierarchy, and also drop the Markdown posts in a parallel hierarchy too. Perfect!

So I set it all up to do that, fired up blogmore serve, visited the URL and... No posts yet. What the hell?

So I checked the code for BlogMore and, sure enough, despite the fact the documentation was selling me on this handy way to organise my posts, no such feature existed!

As an experiment I then asked Copilot what the heck was going on. Much as I expected, rather than coming back with an answer to the question, it went right ahead and fixed it instead. Which is fine, that's where I would have taken this, but I do wish it would answer the question first.

So, yeah, unsurprisingly Copilot flat out lied¹ in the documentation. I'm not in the least bit shocked by this and, as I said, I fully expected this. But it was amusing to have an example of this turn up so early in the documentation, in such a glaring way, and in a way that was so easily fixed (really, it was just a swap of Path.glob to Path.rglob).

As I play with this more it's going to be fun to see what other bold claims turn out to not be true; or perhaps even the reverse: what neat features lurk in the code that haven't been documented.