That didn't go to plan¶

So... yeah... the dogfooding... When I wrote my previous post I had wanted to try and do a post towards the end of each week, highlighting what I'd done on the "dogfooding" front. Life kinda had other plans. Not in a terrible way, but it turns out that getting both flu and Covid jabs (AKA "jags" as they tend to say in my adopted home) on the same day doesn't really agree with me too well.

I have been working, but there's been some odd moments in the past week and a bit and, last week, once I got to the end, I was glad for it to end. So no blog post happened.

Anyway...

What have I been up to?¶

While mostly sat feeling sorry for myself on my sofa, I have been coding. Rather than list all the different things here in detail, I'll quickly mention them with links to where to find them and play with them if you want:

FivePyFive¶

While my Textual 5x5 puzzle is one of the examples in the Textual repo, I wanted to make it more widely available so people can download it with pip or pipx. See over on PyPI and see if you can solve it. ;-)

textual-qrcode¶

I wanted to put together a very small example of how someone may put together a third party widget library, and in doing so selected what I thought was going to be a mostly-useless example: a wrapper around a text-based QR code generator website. Weirdly I've had a couple of people express a need for QR codes in the terminal since publishing that!

PISpy¶

PISpy is a very simple terminal-based client for the PyPI API. Mostly it provides a hypertext interface to Python package details, letting you look up a package and then follow its dependency links. It's very simple at the moment, but I think more fun things can be done with this.

OIDIA¶

I'm a big fan of the use of streak-tracking in one form or another. Personally I use a streak-tracking app for keeping tabs of all sorts of good (and bad) habits, and as a heavy user of all things Apple I make a lot of use of the Fitness rings, etc. So I got to thinking it might be fun to do a really simple, no shaming, no counting, just recording, streak app for the Terminal. OIDIA is the result.

As of the time of writing I only finished the first version of this yesterday evening, so there are plenty of rough edges; but having got it to a point where it performed the basic tasks I wanted from it, that seemed like a good time to publish.

Expect to see this getting more updates and polish.

Wait, what about this Keymaster thing?¶

Ahh, yes, about that... So one of the handy things I'm finding about Textual is its key binding system. The more I build Textual apps, the more I appreciate the bindings, how they can be associated with specific widgets, the use of actions (which can be used from other places too), etc.

But... (there's always a "but" right -- I mean, there'd be no blog post to be had here otherwise).

The terminal doesn't have access to all the key combinations you may want to use, and also, because some keys can't necessarily be "typed", at least not easily (think about it: there's no F1 character, you have to type F1), many keys and key combinations need to be bound with specific names.

So there's two problems here: how do I discover what keys even turn up in my application, and when they do, what should I call them when I pass them to Binding?

That felt like a "well Dave just build an app for it!" problem. So I did:

If you're building apps with Textual and you want to discover what keys turn up from your terminal and are available to your application, you can:

pipx install textual-keys

and then just run textual-keys and start mashing the keyboard to find out.

There's a good chance that this app, or at least a version of it, will make it into Textual itself (very likely as one of the devtools). But for now it's just an easy install away.

I think there's a call to be made here too: have you built anything to help speed up how you work with Textual, or just make the development experience "just so"? If so, do let folk know, and come yell about it on the #show-and-tell channel in the Discord server.

An update¶

So, it's fast approaching 2 months now since I started the new thing and it's been a busy time. I've had to adjust to a quite a few new things, not least of which has been a longer and more involved commute. I'm actually mostly enjoying it too. While having to contend with buses isn't the best thing to be doing with my day, I do have a very fond spot for Edinburgh and it's nice to be in there most days of the week.

Part of the fallout from the new job has been that, in the last couple of weeks, I've thrown some more stuff up on PyPi. This comes about as part of a bit of a dog-fooding campaign we're on at the moment. While they have been, and will continue to be, mentioned on the Textualize blog, I thought I'd give a brief mention of them here on my own blog too given they are, essentially, personal projects.

gridinfo¶

This is one I'd like to tweak some more and improve on if possible. It is, in essence, a Python-coded terminal tool that does more or less the same as slstats.el. It started out as a rather silly quick hack, designed to do something different with rich-pixels.

Here's the finished version (as of the time of writing) being put through its paces:

Download from here, or install and play with it with a quick pipx install gridinfo.

unbored¶

The next experiment with Textual was to write a terminal-based client for the Bored-API. My initial plan for this was to just have a button or two that the user could mash on and they'd get an activity suggestion dropped into the middle of the terminal; but really that seemed a bit boring. Then I realised that it'd be a bit more silly and a bit more fun if I did it as a sort of TODO app. Bored? Run it up and use one of the activities you'd generated before. Don't like any of them? Ignore them and generate some more! Feeling bad that you've got such a backlog of reasons to not be bored? Delete a bunch!

And so on.

Here's a short video of it in action:

Download from here, or install and play with it with a quick pipx install unbored.

textual-qrcode¶

This one... this one I'm going to blame on the brain fog that followed flu and Covid jabs that happened the day before (and which are still kicking my arse 4 days later). Monday morning, at my desk, and I'm wondering what to next write to experiment with Textual, and I realised it would be interesting to write something that would show off that it's easy to make a third party widget library.

And... yeah, I don't know why, but I remembered qrencode.el and so textual-qrcode was born!

I think the most amusing part about this is that I did it in full knowledge that it would be useless; the idea being it would be a daft way of showing off how you could build a widget library as an add-on for Textual. But... more than one person actually ended up saying "yeah hold up there this could actually be handy!"

If you're one of those people... you'll find it here.

FivePyFive¶

While I was on a roll putting stuff up on PyPi, I also decided to tweak up my Textual-based 5x5 and throw that up too. This was my first app built with Textual, initially written before I'd even spoken to Will about the position here. At one point I even did a version in Lisp.

It's since gone on to become one of the example apps in Textual itself but I felt it deserved being made available to the world via an easy(ish) install. So, if you fancy trying to crack the puzzle in your terminal, just do a quick:

pipx install fivepyfive

and click away.

You can find it over here.

PISpy¶

Finally... for this week anyway, is a tool I've called PISpy. It's designed as a simple terminal client for looking up package information on PyPi. As of right now it's pretty straightforward, but I'd like to add more to it over time. Here's an example of it in action:

One small wrinkle with publishing it to PyPi was the fact that, once I'd chosen the name, I checked that it hadn't been used on PyPi (it hadn't) but when it came to publishing the package it got rejected because the name was too similar to another package! I don't know which, it wouldn't say, but that was a problem. So in the end the published name ended up having to be slightly different from the actual tool's name.

See over here for the package, and you can install it with a:

pipx install pispy-client

and then just run pispy in the terminal.

Conclusion¶

It's been a fun couple of weeks coming up with stuff to help exercise Textual, and there's more to come. Personally I've found the process really helpful in that it's help me learn more about the framework and also figure out my own approach to working with it. Each thing I've built so far has been a small step in evolution on from what I did in the previous thing. I doubt I've arrived at a plateau of understanding just yet.

Introduction¶

Cutler, armed with a schedule, was urging the team to "eat its own dog food". Part macho stunt and part common sense, the "dog food diet" was the cornerstone of Cutler’s philosophy.

G. Pascal Zachary — Show-Stopper!

I can't remember exactly when it was -- it was likely late in 1994 or some time in 1995 -- when I first came across the concept of, or rather the name for the concept of, "eating your own dog food". The idea and the name played a huge part in the book Show-Stopper! by G. Pascal Zachary. The idea wasn't new to me of course; I'd been writing code for over a decade by then and plenty of times I'd built things and then used those things to do things, but it was fascinating to a mostly-self-taught 20-something me to be reading this (excellent -- go read it if you care about the history of your craft) book and to see the idea written down and named.

While Textualize isn't (thankfully -- really, I do recommend reading the book) anything like working on the team building Windows NT, the idea of taking a little time out from working on Textual, and instead work with Textual, makes a lot of sense. It's far too easy to get focused on adding things and improving things and tweaking things while losing sight of the fact that people will want to build with your product.

So you can imagine how pleased I was when Will announced that he wanted all of us to spend a couple or so weeks building something with Textual. I had, of course, already written one small application with the library, and had plans for another (in part it's how I ended up working here), but I'd yet to really dive in and try and build something more involved.

Giving it some thought: I wasn't entirely sure what I wanted to build though. I do want to use Textual to build a brand new terminal-based Norton Guide reader (not my first, not by a long way) but I felt that was possibly a bit too niche, and actually could take a bit too long anyway. Maybe not, it remains to be seen¹.

Eventually I decided on this approach: try and do a quick prototype of some daft idea each day or each couple of days, do that for a week or so, and then finally try and settle down on something less trivial. This approach should work well in that it'll help introduce me to more of Textual, help try out a few different parts of the library, and also hopefully discover some real pain-points with working with it and highlight a list of issues we should address -- as seen from the perspective of a developer working with the library.

So, here I am, at the end of week one. What I want to try and do is briefly (yes yes, I know, this introduction is the antithesis of brief) talk about what I built and perhaps try and highlight some lessons learnt, highlight some patterns I think are useful, and generally do an end-of-week version of a TIL. TWIL?

Yeah. I guess this is a TWIL.

gridinfo¶

I started the week by digging out a quick hack I'd done a couple of weeks earlier, with a view to cleaning it up. It started out as a fun attempt to do something with Rich Pixels while also making a terminal-based take on slstats.el. I'm actually pleased with the result and how quickly it came together.

The point of the application itself is to show some general information about the current state of the Second Life grid (hello to any fellow residents of the original Metaverse!), and to also provide a simple region lookup screen that, using Rich Pixels, will display the object map (albeit in pretty low resolution -- but that's the fun of this!).

So the opening screen looks like this:

and a lookup of a region looks like this:

Here's a wee video of the whole thing in action:

Worth a highlight¶

Here's a couple of things from the code that I think are worth a highlight, as things to consider when building Textual apps:

Don't use the default screen¶

Use of the default Screen that's provided by the App is handy enough, but I feel any non-trivial application should really put as much code as possible in screens that relate to key "work". Here's the entirety of my application code:

class GridInfo( App[ None ] ):
    """TUI app for showing information about the Second Life grid."""

    CSS_PATH = "gridinfo.css"
    """The name of the CSS file for the app."""

    TITLE = "Grid Information"
    """str: The title of the application."""

    SCREENS = {
        "main": Main,
        "region": RegionInfo
    }
    """The collection of application screens."""

    def on_mount( self ) -> None:
        """Set up the application on startup."""
        self.push_screen( "main" )

You'll notice there's no work done in the app, other than to declare the screens, and to set the main screen running when the app is mounted.

Don't work hard on_mount¶

My initial version of the application had it loading up the data from the Second Life and GridSurvey APIs in Main.on_mount. This obviously wasn't a great idea as it made the startup appear slow. That's when I realised just how handy call_after_refresh is. This meant I could show some placeholder information and then fire off the requests (3 of them: one to get the main grid information, one to get the grid concurrency data, and one to get the grid size data), keeping the application looking active and updating the display when the replies came in.

Pain points¶

While building this app I think there was only really the one pain-point, and I suspect it's mostly more on me than on Textual itself: getting a good layout and playing whack-a-mole with CSS. I suspect this is going to be down to getting more and more familiar with CSS and the terminal (which is different from laying things out for the web), while also practising with various layout schemes -- which is where the revamped Placeholder class is going to be really useful.

unbored¶

The next application was initially going to be a very quick hack, but actually turned into a less-trivial build than I'd initially envisaged; not in a negative way though. The more I played with it the more I explored and I feel that this ended up being my first really good exploration of some useful (personal -- your kilometerage may vary) patterns and approaches when working with Textual.

The application itself is a terminal client for the Bored-API. I had initially intended to roll my own code for working with the API, but I noticed that someone had done a nice library for it and it seemed silly to not build on that. Not needing to faff with that, I could concentrate on the application itself.

At first I was just going to let the user click away at a button that showed a random activity, but this quickly morphed into a "why don't I make this into a sort of TODO list builder app, where you can add things to do when you are bored, and delete things you don't care for or have done" approach.

Here's a view of the main screen:

and here's a view of the filter pop-over:

Worth a highlight¶

Don't put all your BINDINGS in one place¶

This came about from me overloading the use of the escape key. I wanted it to work more or less like this:

• If you're inside an activity, move focus up to the activity type selection buttons.
• If the filter pop-over is visible, close that.
• Otherwise exit the application.

It was easy enough to do, and I had an action in the Main screen that escape was bound to (again, in the Main screen) that did all this logic with some if/elif work but it didn't feel elegant. Moreover, it meant that the Footer always displayed the same description for the key.

That's when I realised that it made way more sense to have a Binding for escape in every widget that was the actual context for escape's use. So I went from one top-level binding to...

...

class Activity( Widget ):
    """A widget that holds and displays a suggested activity."""

    BINDINGS = [
        ...
        Binding( "escape", "deselect", "Switch to Types" )
    ]

...

class Filters( Vertical ):
    """Filtering sidebar."""

    BINDINGS = [
        Binding( "escape", "close", "Close Filters" )
    ]

...

class Main( Screen ):
    """The main application screen."""

    BINDINGS = [
        Binding( "escape", "quit", "Close" )
    ]
    """The bindings for the main screen."""

This was so much cleaner and I got better Footer descriptions too. I'm going to be leaning hard on this approach from now on.

Messages are awesome¶

Until I wrote this application I hadn't really had a need to define or use my own Messages. During work on this I realised how handy they really are. In the code I have an Activity widget which takes care of the job of moving itself amongst its siblings if the user asks to move an activity up or down. When this happens I also want the Main screen to save the activities to the filesystem as things have changed.

Thing is: I don't want the screen to know what an Activity is capable of and I don't want an Activity to know what the screen is capable of; especially the latter as I really don't want a child of a screen to know what the screen can do (in this case "save stuff").

This is where messages come in. Using a message I could just set things up so that the Activity could shout out "HEY I JUST DID A THING THAT CHANGES ME" and not care who is listening and not care what they do with that information.

So, thanks to this bit of code in my Activity widget...

    class Moved( Message ):
        """A message to indicate that an activity has moved."""

    def action_move_up( self ) -> None:
        """Move this activity up one place in the list."""
        if self.parent is not None and not self.is_first:
            parent = cast( Widget, self.parent )
            parent.move_child(
                self, before=parent.children.index( self ) - 1
            )
            self.emit_no_wait( self.Moved( self ) )
            self.scroll_visible( top=True )

...the Main screen can do this:

    def on_activity_moved( self, _: Activity.Moved ) -> None:
        """React to an activity being moved."""
        self.save_activity_list()

Pain points¶

On top of the issues of getting to know terminal-based-CSS that I mentioned earlier:

• Textual currently lacks any sort of selection list or radio-set widget. This meant that I couldn't quite do the activity type picking how I would have wanted. Of course I could have rolled my own widgets for this, but I think I'd sooner wait until such things are in Textual itself.
• Similar to that, I could have used some validating Input widgets. They too are on the roadmap but I managed to cobble together fairly good working versions for my purposes. In doing so though I did further highlight that the reactive attribute facility needs a wee bit more attention as I ran into some (already-known) bugs. Thankfully in my case it was a very easy workaround.
• Scrolling in general seems a wee bit off when it comes to widgets that are more than one line tall. While there's nothing really obvious I can point my finger at, I'm finding that scrolling containers sometimes get confused about what should be in view. This becomes very obvious when forcing things to scroll from code. I feel this deserves a dedicated test application to explore this more.

Conclusion¶

The first week of "dogfooding" has been fun and I'm more convinced than ever that it's an excellent exercise for Textualize to engage in. I didn't quite manage my plan of "one silly trivial prototype per day", which means I've ended up with two (well technically one and a half I guess given that gridinfo already existed as a prototype) applications rather than four. I'm okay with that. I got a lot of utility out of this.

Now to look at the list of ideas I have going and think about what I'll kick next week off with...

Introduction

On Twitter, a few weeks back, @itsBexli asked me how I go about setting up Python for development on macOS. It's a great question and one that seems to crop up in various places, and since I first got into using macOS and then subsequently got back into coding lots in Python it's absolutely an issue I ran into.

With my previous employer, while I was the only developer, I wasn't the only person writing code and more than one other person had this issue so I eventually wrote up my approach to solving this problem. That document is on their internal GitLab, but I thought it worth me writing my personal macOS/Python "rules" up again, just in case they're useful to anyone else.

I am, of course, not the first person to tackle this, to document this, to write down a good approach to this. Before and after I settled on my approach I'd seen other people write about this. So... this post isn't here to try and replace those, it's simply to write down my own approach, so if anyone asks again I can point them here. I feel it's important to stress: this isn't the only way or thoughts on this issue, there are lots of others. Do go read them too and then settle on an approach that works for you.

One other point to note, which may or may not make a difference (and may or may not affect how this changes with time): for the past few years I've been a heavy user of pipenv to manage my virtual environments. This is very likely to change from now on, but keep in mind that what follows was arrived at from the perspective of a pipenv user.

So... with that admin aside...

The Problem

When I first got back into writing Python it was on macOS and, really early on, I ran into all the usual issues: virtual environments breaking because they were based on the system Python and it got updated, virtual environments based on the Homebrew-installed Python and it got updated, etc... Simply put, an occasional, annoying, non-show-stopping breaking of my development environment which would distract me when I'd sat down to just hack on some code, not do system admin!

My Solution

For me, what's worked for me without a problem over the past few years, in short, is this:

1. NEVER use the system version of Python. Just don't.
2. NEVER use the Homebrew's own version of Python (I'm not even sure this is an issue any more; but it used to be).
3. NEVER use a version of Python installed with Homebrew (or, more to the point, never install Python with Homebrew).
4. Manage everything with pyenv; which I do install from Homebrew.

The Detail

As mentioned earlier, what I'm writing here assumes that virtual environments are being managed with pipenv (something I still do for personal projects, for now, but this may change soon). Your choices and mileage may vary, etc... This is what works well for me.

The "one time" items¶

These are the items that need initially installing into a new macOS machine:

Homebrew¶

Unless it comes from the Mac App Store, I try and install everything via Homebrew. It's really handy for keeping track of what I've got installed, for recreating a development environment in general, and for keeping things up to date.

pyenv¶

With Homebrew installed the next step for me is to install pyenv. Doing so is as easy as:

brew install pyenv

Once installed, if it's not done it for you, you may need to make some changes to your login profile. I'm a user of fish so I have these lines in my setup. Simply put: it asks pyenv to set up my environment so my calls to Python go via its setup.

Plenty of help on how to set up pyenv can be found in its README.

Once I've done this I tend to go on and install the Python versions I'm likely to need. For me this tends to be the most recent "active" stable versions (as of the time of writing, 3.7 through 3.10; although I need to now start throwing 3.11 into the mix too).

I use this command:

pyenv install --list

to see the available versions. If I want to see what's available for a specific version I'll pipe through grep:

pyenv install --list | fgrep "  3.9"

This is handy if I want to check what the very latest release of a specific version of Python is.

The "Global" Python¶

When I'm done with the above I then tend to use pyenv to set my "global" Python. This is the version I want to get when I run python and I'm not inside a virtual environment. Doing that is as simple as:

pyenv global 3.10.7

Of course, you'd swap the version for whatever works for you.

When Stuff Breaks¶

It seems more rare these days, but on occasion I've had it such that some update somewhere still causes my environment to break. Now though I find that all it takes is a quick:

pyenv rehash

and everything is good again.

Setting Up A Repo¶

With all of the stuff above being mostly a one-off (or at least something I do once when I set up a new machine -- which isn't often), the real "work" here is when I set up a fresh repository when I start a new project. Really it's no work at all. Again, as I've said before, I've tended to use pipenv for my own work, and still do for personal stuff (but do want to change that), mileage may vary here depending on tool.

When I start a new project I think about which Python version I want to be working with, I ensure I have the latest version of it installed with pyenv, and then ask pipenv to create a new virtual environment with that:

pipenv --python 3.10.7

When you do this, you should see pipenv pulling the version of Python from the pyenv directories:

$ pipenv --python 3.10.7
Creating a virtualenv for this project...
Pipfile: /Users/davep/temp/cool-project/Pipfile
Using /Users/davep/.pyenv/versions/3.10.7/bin/python3 (3.10.7) to create virtualenv...
⠙ Creating virtual environment...created virtual environment CPython3.10.7.final.0-64 in 795ms
  creator CPython3Posix(dest=/Users/davep/temp/cool-project/.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/davep/Library/Application Support/virtualenv)
    added seed packages: pip==22.2.2, setuptools==65.3.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
✔ Successfully created virtual environment!
Virtualenv location: /Users/davep/temp/cool-project/.venv
Creating a Pipfile for this project...

The key thing here is seeing that pipenv is pulling Python from ~/.pyenv/versions/. If it isn't there's a good chance you have a Python earlier in your PATH than the pyenv one -- something you generally don't want. It will work, but it's more likely to break at some point in the future. That's the key thing I look for; if I see that I know things are going to be okay.

Conclusion¶

Since I adopted these personal rules and approaches (and really, calling them "rules" is kind of grand -- there's almost nothing to this) I've found I've had near-zero issues with the stability of my Python virtual environments (and what issues I have run into tend to be trivial and of my own doing).

As I said at the start: there are, of course, other approaches to this, but this is mine and works well for me. Do feel free to comment with your own, I'm always happy to learn about new ways!

Just over five years ago I got a message from my then employer to say I was going to be made redundant after 21 years working for them. After the 3 month notice period the final day came. Meanwhile, I found something new that looked terrifying but interesting. In the end it was less terrifying and way more interesting than I imagined it would be. It was fun too.

But... (there's always a but isn't there?)

In the four and change years I've been there the company got bought out, and then the result of that got bought up. As I've mentioned before I'm generally not a "big company" kind of person; in all my years I've found that I'm happier working in a smaller place. After a couple of buyouts my employer had gone from being 10s of people in size to 100s of people in size (and technically 10s of 1,000s of people in size depending on how you look at it).

This change in ownership and size meant the culture became... well, let's just say not as friendly as you tend to enjoy when it's a smaller group of folk. On top of that I was starting to notice that my efforts were making less of an impact as things got bigger, and I started to feel like my contributions weren't really relevant any more. There were some problematic things happening too: undermining of efforts, removal of responsibilities without consultation or communication, that sort of thing. Plus worse. There's little point in going into the detail, but it's fair to say that work wasn't as fun as it used to be.

That felt like a good time to start to look around. If work makes you feel unhappy and you can look around... look around.

Thing is, I wasn't sure what to look for. I was in the comfortable position of, unlike last time, not needing to find something, so I could take my time at least. Over the course of the last year I've spoken to many different companies and organisations, some big (yes, I know, I said I don't like big places -- sometimes what's on offer deserves a fair hearing), some small, but none of them quite said "this feels like me". In some cases the whole thing didn't have the right vibe, in others the industry either didn't interest me, or felt uncomfortable given my personal values. In one particular case a place looked interesting until I checked the CTO's socials and OMG NO NO NO AVOID AVOID (that was a fun one).

Then I saw Will McGugan saying he was hiring to expand Textualize. This caught my interest right away for two reasons.

I can't remember how long I've been following Will on Twitter; I likely stumbled on him as I got back into Python in 2018 and I also remember noting that he was a Python hacker just up the road from me. We'd vaguely chatted on Twitter, briefly, in that "Twitter acquaintance" way we all often do (I remember one brief exchange about fungus on The Meadows). A small company run by someone I was acquainted with seemed like a safe bet.

The second reason was Textual itself. I'd been watching Will develop it, in open, with great interest. I had (and still have) a plan to write a brand new CHUI-based (okay fine TUI-based as the kids like to say these days!) Norton Guide reader¹, all in Python, and Textual looked like the perfect framework to do the UI in. The chance to be involved with it sounded awesome.

Now, I said two reasons, but there's also a third I guess: Will's pitch for applying to Textualize felt so damn accessible! I'm on the older end of the age range of this industry; for much of my working life as a developer I've worked in isolation from other developers; while I first touched Python in the 90s, I've only been using it in anger since 2018 and still feel like I've got a lot to learn. Despite all these things, and more, saying "aye Dave this is beyond you" I felt comfortable dropping Will a line.

Which resulted in a chat.

Which resulted in some code tinkering and chatting.

Which resulted in...

Something new.

So, yeah, as of 2022-10-10 I'm on yet another new adventure. Time for me to really work on my Python coding as I work with Will and the rest of the team as part of Textualize.

Or, as I put it on Twitter a few days ago: "I'm going to be a Python impostor syndrome speedrunner".

I first learnt Python back in the mid-to-late 90s, used it in place of Perl once I was comfortable with it, and then we sort of drifted apart when I first met Ruby. It's only in the last couple of years that I've got back into it, and in a huge way, thanks to my (not-quite-so-) new job. Despite the quirks and oddness (as I perceive them), I actually quite like Python and it's one of those languages that just flows off my fingers. I'm sure you know the same thing, perhaps not with Python, but there will be languages that just flow for you, and those that take a bit more effort and concentration. Python... feels okay to me.

I also appreciate that there's been a long-standing style guide. I quite like PEP 8 as a read, and think there's a lot of good ideas in there; much of the content sits with how I'd approach things if I was tasked to come up with such a document. With this in mind, I'm a fairly heavy user of pylint and it in turn leans on PEP 8 (amongst other things) and I'm happy to accept most of its judgements. Not all of its judgements, but even when I disagree with it I try and keep track of how far I'm drifting.

But there is absolutely one hill I will happily die on when it comes to PEP 8: the concept of "extraneous whitespace" in lists and expressions. Just.... no! Oh gods no!

To borrow a line of code from the journey problem I dabbled with a while back, PEP 8 would have me write something like this:

def perform(commands: List[str],state: State) -> State:

Now, I'm sure plenty of people won't see a problem with this at all; but all I can see is an almost-claustrophobic parameter list. What's with the parameters being jammed up against the opening and closing parens? Why have the dinky little comma lost between two different things? Why have it look like a long stream of letters and punctuation? Why....

No.

Just no.

I can't.

Rightly or wrongly, I just need for the code to breathe a bit. When I type this:

def perform( commands: List[ str ], state: State ) -> State:

suddenly if feels like there's fresh air in the code, like it flows gently out of my head, off my fingers, through the keyboard and into the buffer.

In my head, and to my eyes, the code is.... relaxed.

Do I have a rational reason for this? Nope. Then again I don't see one for doing it the other way either; I can't think of one and I don't see one in the source document. So, that's a warning I always turn off with pylint and it's a style I carry through all my Python code; and I think that's the important point here: anyone reading and working with my code should see the same style all the way through. It might differ from PEP 8 on this point, but at least it's the same all the way.

And, really, that's okay: PEP 8 is there to be ignored. ;-)

PS: This is a small part of another blog post I was meaning to write, and might still do, about my (still ongoing) experience of getting lsp-mode up and running in Emacs and having it play nice with Python projects. I have that working, but it was a bit of a learning curve and epic battle over a couple of days, and one that had me first encounter pycodestyle. I may still tell the tale...

Earlier this week something in my development environment, relating to Homebrew, Python, pyenv and pipenv, got updated and broke a handful of repositories. Not in a way that I couldn't recover from, just in a way that was annoying, got in the way of my workflow, and needed attention. (note to self: how I set up for Python/Django development on a machine might be a good post in the future)

Once I was sure what the fix was (pretty much: nuke the virtual environment and recreate it with pipenv, being very explicit about the version of Python to use) the next step was to figure out how many repositories were affected; not all were and there wasn't an obvious pattern to it. What was obvious was that the problem came down to python in the .venv directory pointing to a binary that didn't exist any more.

So... tracking down problematic repositories would be simple enough, just look for every instance of .venv/bin/python and be sure it points to something rather than nothing; if it points to nothing I need to remake the virtual environment.

I quickly knocked up a script that was based around looking over the results of a find, and initially decided to use file to perform the test on python. It seemed to make sense, as I wrote the script I checked the man page for file(1) on macOS and sure enough, this exists:

-E On filesystem errors (file not found etc), instead of handling the error as regular output as POSIX mandates and keep going, issue an error message and exit.

Given that file dereferences links by default, that should get me an error for a broken link, right? Bit hacky I guess, but it was the first thing that came to mind for a quick bit of scripting and would do the trick. Only...

$ file -E does-not-exist
file: invalid option -- E
Usage: file [bcCdEhikLlNnprsvzZ0] [-e test] [-f namefile] [-F separator] [-m magicfiles] [-M magicfiles] file...
       file -C -m magicfiles
Try `file --help' for more information.

Wat?!? But it's right there! It says so in the manual! -E is documented right in the manual page! And yet it's not in the valid switch list as put out by the command, and it's an invalid option. The hell?

So I go back and look at the man page again and then I notice it isn't in the list of switches in the synopsis.

SYNOPSIS
file [-bcdDhiIkLnNprsvz] [--extension] [--mime-encoding] [--mime-type] [-f namefile] [-m magicfiles] [-P name=value] [-M magicfiles] file
file -C [-m magicfiles]
file [--help]

I then did the obvious tests. Did I have file aliased in some way? No. Was some other thing that works and acts like file in my path? No. Was I absolutely 100% using /usr/bin/file? Yes.

Long story short: it seems the man page for file, on macOS, fibs about what switches it supports; it says that -E is a valid option, but it's not there.

What's even odder is the man page says it documents v5.04 of the command, but --version reports v5.37. Meanwhile, if I check on a GNU/Linux box I have access to, it does support -E, reports it in the switches, documents it in the man page (in both the synopsis and in the main body of the page) and it is v5.25 (and so is its man page).

So that was something like 20 minutes lost to a very small problem, for which there was no real solution, but was time that had to be spent to get to the bottom of it.

In the end I went with what I probably should have gone with in the first place: stat -L.

for venv in $(find . -name .venv)
do
    if ! stat -L "$venv/bin/python" > /dev/null 2>&1
    then
        echo "$(dirname $venv)"
    fi
done

And now I have that script in my ~/bin directory, ready for the next time Homebrew and friends conspire to throw my day off for a while.

At the start of this year, as part of a much bigger process to review the work that had taken place over the previous 12 months, I was asked (at work) to provide some information about how much time I'd spent on various projects. Now, for me, there's really only one project, but there's lots of different tools and libraries that I've written to support the main work I do. All of these are split into different repositories in the company-internal instance of GitLab. This meant that getting a rough idea of what I was working on and when would be easy enough -- it's all there in the commit history.

Given that this information would make up a couple of slides at most during a far bigger presentation, I wanted something that would be snappy and easy for non-developers to follow and understand. I spent a bit of time pondering some options and decided that (ab)using a gantt chart layout would make sense.

That choice was made all the more easier given that GitLab supports the use of mermaid charts within its Markdown. This meant I could quickly write some code that took the git log of each repository, turned it into mermaid code, and then render it (by hand, this was all about getting things done quickly) via GitLab.

This sounded like it could be a fun personal project. The result was some Python code called git2gantt.

As mentioned above, the output isn't anything too clever, it's just code that can be used to create a plot via mermaid. For example, running git2gannt over itself:

gantt
  title git2gantt output
  dateFormat YYYY-MM-DD

  section git2gantt
  Development: devgit2gantt20190208, 2019-02-08, 2019-02-13
  Development: devgit2gantt20190214, 2019-02-14, 2019-02-15
  Development: devgit2gantt20190303, 2019-03-03, 2019-03-04
  Development: devgit2gantt20191203, 2019-12-03, 2019-12-04

Usage is pretty straightforward: As you can see, it can be run over multiple repos at once, and there's also an option to have it consider every branch within each repository. Another handy option is the ability to limit the output to just one author -- perhaps you just want to document what you've done on a repo, not the contributions of other people.

Also especially handy, if you don't want to bore people with too much detail, is the "fuzz" option. This lets you tell git2gannt how relaxed you want it to be when it tries to decide how long a run of work on a repo lasted. So, perhaps, you're working on and off on a library that supports some other system you're documenting, but you might only be making changes every other day or so. With the correct fuzz value you can make it clear you were working on the library for a couple of weeks, despite there only being a commit every other day.

An example of running the output over a handful of projects would look something like this:

This is one of those tools I knocked up quickly to get a job done, and haven't quite got round to finishing off fully. One thing I'd really like to do is add mermaid support directly within it, so that it actually has the option to emit plots, not just mermaid code (or, perhaps, drop the mermaid approach and use something else entirely).

Meanwhile though, if you're looking for something quick and dirty that will help you visualise what you've been working on and when for a good period of time... perhaps this will help.

It's hardly a revelation to say that learning a new programming language, or even learning software development at all, is even more difficult if you don't have an actual problem to solve. I know I'm not alone in having pet projects that, when faced with a new environment, I'll code up a version of that project as a way to get familiar with and understand a language's idioms while implementing something I know well.

Personally, my two favourites are a puzzle called 5x5 (here, here, here, here, here, here and here), and writing a library or even a full application to read Norton Guide database files (here, here, here, here, here, here, here and here). Both are fun to work on, have practical uses, and both have the benefit of being solved problems (for me) that let me concentrate on the "how do I do X in this language/toolkit/environment/framework/etc?".

Even with those two as my goto projects, I'm always open to new small problems that might be fun to apply to languages I do know, or languages I want to get to know (internally at work we have a fun "league" of sorts, writing a particular hamming distance calculation tool in different languages, for example).

A few days ago, via this repo on GitHub, I discovered this fun little problem. Right away I could see the benefit in it. As a "go away and code up a solution" interview question it strikes me as near-perfect. It's obviously not hard to solve, but it touches on some basic but important aspects of software development and so will allow the developer to show off how they approach things.

There's so many different approaches to it too. Even in a single language, I could imagine having some fun writing the smallest code to solve the problem, the most idiomatic code to solve the problem, the most supportable and well-documented code to solve the problem, etc. And then there's the thing I talk about above: knowing the solution and knowing it's easy, you can then use it to learn the idiomatic way of solving the problem in new languages.

Even better, the README of the original repo links to solutions others have written. Knowing the problem, and knowing the solution, you can then go and read other people's code and learn something about different styles and different languages.

Over the next few weeks, as I get free time, I think I might just do this. Take the "Journeys" problem and write versions in different languages I work with, or know, and also use it to get to know languages I've yet to know or use heavily (I'm especially keen to try a version in Julia -- a language I really like the look of and want to find a reason to use).

Meanwhile, yesterday, I had a quick go at a first version in Python (aimed at Python 3.8 or higher): https://github.com/davep/journeys.py

I set out to try and write something that was fairly idiomatic Python, which uses tools that I tend to employ when working on Python projects (pipenv, make, etc), and which also used something I've never quite found a need for so far in my usual coding, but which I can see being useful and helpful.

I even threw in a couple of uses of PEP 572!

I can see me tinkering with this some more over the next few days. I can even see me writing a very different implementation in Python, just for the fun of it.

I think that's what I like about this little problem. It's a good way to do a bit of programming exercise; it's like the perfect way to do the programming equivalent of going for a short run.

I first got into Python in the mid-to-late 1990s. It's so far back that I think the copy of Programming Python that I have (sadly in storage at the moment) might be a first edition. I probably fell out of the habit of using Python some time in the early 2000s (that was when I met Ruby). It was only 22 months ago that I started using Python a lot thanks to a change of employer.

As you might imagine, much had changed in the 15+ years since I'd last written a line of Python in anger. So, early on, I made a point of making Pylint part of my development process. All my projects have a make lint make target. All of my projects lint the code when I push to master in the company GitLab instance. These days I even use flycheck to keep me honest as I write my code; mostly gone are the days where I don't know of problems until I do a make lint.

Leaning on Pylint in the early days of my new position made for a great Python refresher for me. Now, I still lean on it to make sure I don't make daft mistakes.

But...

Pylint and I don't always agree. And that's fine. For example, I really can't stand Pylint's approach to whitespace, and that is a hill I'll happily die on. Ditto the obsession with lines being no more than 80 characters wide (120 should be fine thanks). As such any project's .pylintrc has, as a bare minimum, this:

[FORMAT]
max-line-length=120

[MESSAGES CONTROL]
disable=bad-whitespace

Beyond that though, aside from one or two extras that pertain to particular projects, I'm happy with what Pylint complains about.

There are exceptions though. There are times, simply due to the nature of the code involved, that Pylint's insistence on code purity isn't going to work. That's where I use its inline block disabling feature. It's handy and helps keep things clean (I won't deploy code that doesn't pass 10/10), but there is always this nagging doubt: if I've disabled a warning in the code, am I ever going to come back and revisit it?

To help me think about coming back to such disables now and again, I thought it might be interesting to write a tool that'll show which warnings I disable most. It resulted in this fish abbr:

abbr -g pylintshame "rg --no-messages \"pylint:disable=\" | awk 'BEGIN{FS=\"disable=\";}{print \$2}' | tr \",\" \"\n\" | sort | uniq -c | sort -hr"

The idea here being that it produces a "Pylint hall of shame", something like this:

  12 wildcard-import
  12 unused-wildcard-import
   8 no-member
   6 invalid-name
   5 no-self-use
   4 import-outside-toplevel
   4 bare-except
   2 unused-argument
   2 too-many-public-methods
   2 too-many-instance-attributes
   2 not-callable
   2 broad-except
   1 wrong-import-position
   1 wrong-import-order
   1 unused-variable
   1 unexpected-keyword-arg
   1 too-many-locals
   1 arguments-differ

To break the pipeline down:

rg --no-messages "pylint:disable="

First off, I use ripgrep (if you don't, you might want to have a good look at it -- I find it amazingly handy) to find everywhere in the code in and below the current directory (the --no-messages switch just stops any file I/O errors that might result from permission issues -- they're not interesting here) that contains a line that has a Pylint block disable (if you tend to format yours differently, you'll need to tweak the regular expression, of course).

I then pipe it through awk:

awk 'BEGIN{FS="disable=";}{print $2}'

so I can lazily extract everything after the disable=.

Next up, because it's a possible list of things that can be disabled, I use tr:

tr "," "\n"

to turn any comma-separated list into multiple lines.

Having got to this point, I sort the list, uniq the result, while prepending a count (-c), and then sort the result again, in reverse and sorting the numbers based on how a human would read the result (-hr).

sort | uniq -c | sort -hr

It's short, sweet and hacky, but does the job quite nicely. From now on, any time I get curious about which disables I'm leaning on too much, I can use this to take stock.