• New Things On PyPi

    An update

    So, it’s fast approaching 2 months now since I started the new thing and it’s been a busy time. I’ve had to adjust to a quite a few new things, not least of which has been a longer and more involved commute. I’m actually mostly enjoying it too. While having to contend with busses isn’t the best thing to be doing with my day, I do have a very fond spot for Edinburgh and it’s nice to be in there most days of the week.

    Part of the fallout from the new job has been that, in the last couple of weeks, I’ve thrown some more stuff up on PyPi. This comes about as part of a bit of a dog-fooding campaign we’re on at the moment (you can read some background to this over on the company blog). While they have been, and will continue to be, mentioned on the Textualize blog, I thought I’d give a brief mention of them here on my own blog too given they are, essentially, personal projects.

    gridinfo

    This is one I’d like to tweak some more and improve on if possible. It is, in essence, a Python-coded terminal tool that does more or less the same as slstats.el. It started out as a rather silly quick hack, designed to do something different with rich-pixels.

    Here’s the finished version (as of the time of writing) being put through its paces:

    Download from here, or install and play with it with a quick pipx install gridinfo.

    unbored

    The next experiment with Textual was to write a terminal-based client for the Bored-API. My initial plan for this was to just have a button or two that the user could mash on and they’d get an activity suggestion dropped into the middle of the terminal; but really that seemed a bit boring. Then I realised that it’d be a bit more silly and a bit more fun if I did it as a sort of TODO app. Bored? Run it up and use one of the activities you’d generated before. Don’t like any of them? Ignore them and generate some more! Feeling bad that you’ve got such a backlog of reasons to not be bored? Delete a bunch!

    And so on.

    Here’s a short video of it in action:

    Download from here, or install and play with it with a quick pipx install unbored.

    textual-qrcode

    This one… this one I’m going to blame on the brain fog that followed flu and Covid jabs that happened the day before (and which are still kicking my arse 4 days later). Monday morning, at my desk, and I’m wondering what to next write to experiment with Textual, and I realised it would be interesting to write something that would show off that it’s easy to make a third party widget library.

    And… yeah, I don’t know why, but I remembered qrencode.el and so textual-qrcode was born!

    The most useless Textal widget yet

    I think the most amusing part about this is that I did it in full knowledge that it would be useless; the idea being it would be a daft way of showing off how you could build a widget library as an add-on for Textual. But… more than one person actually ended up saying “yeah hold up there this could actually be handy!”

    If you’re one of those people… you’ll find it here.

    FivePyFive

    While I was on a roll putting stuff up on PyPi, I also decided to tweak up my Textual-based 5x5 and throw that up too. This was my first app built with Textual, initially written before I’d even spoken to Will about the position here. At one point I even did a version in Lisp.

    It’s since gone on to become one of the example apps in Textual itself but I felt it deserved being made available to the world via an easy(ish) install. So, if you fancy trying to crack the puzzle in your terminal, just do a quick:

    $ pipx install fivepyfive
    

    and click away.

    You can find it over here.

    PISpy

    Finally… for this week anyway, is a tool I’ve called PISpy. It’s designed as a simple terminal client for looking up package information on PyPi. As of right now it’s pretty straightforward, but I’d like to add more to it over time. Here’s an example of it in action:

    One small wrinkle with publishing it to PyPi was the fact that, once I’d chosen the name, I checked that it hadn’t been used on PyPi (it hadn’t) but when it came to publishing the package it got rejected because the name was too similar to another package! I don’t know which, it wouldn’t say, but that was a problem. So in the end the published name ended up having to be slightly different from the actual tool’s name.

    See over here for the package, and you can install it with a:

    $ pipx install pispy-client
    

    and then just run pispy in the terminal.

    Conclusion

    It’s been a fun couple of weeks coming up with stuff to help exercise Textual, and there’s more to come. Personally I’ve found the process really helpful in that it’s help me learn more about the framework and also figure out my own approach to working with it. Each thing I’ve built so far has been a small step in evolution on from what I did in the previous thing. I doubt I’ve arrived at a plateau of understanding just yet.

  • Python and macOS

    Introduction

    On Twitter, a few weeks back, @itsBexli asked me how I go about setting up Python for development on macOS. It’s a great question and one that seems to crop up in various places, and since I first got into using macOS and then subsequently got back into coding lots in Python it’s absolutely an issue I ran into.

    With my previous employer, while I was the only developer, I wasn’t the only person writing code and more than one other person had this issue so I eventually wrote up my approach to solving this problem. That document is on their internal GitLab, but I thought it worth me writing my personal macOS/Python “rules” up again, just in case they’re useful to anyone else.

    I am, of course, not the first person to tackle this, to document this, to write down a good approach to this. Before and after I settled on my approach I’d seen other people write about this. So… this post isn’t here to try and replace those, it’s simply to write down my own approach, so if anyone asks again I can point them here. I feel it’s important to stress: this isn’t the only way or thoughts on this issue, there are lots of others. Do go read them too and then settle on an approach that works for you.

    One other point to note, which may or may not make a difference (and may or may not affect how this changes with time): for the past few years I’ve been a heavy user of pipenv to manage my virtual environments. This is very likely to change from now on, but keep in mind that what follows was arrived at from the perspective of a pipenv user.

    So… with that admin aside…

    The Problem

    When I first got back into writing Python it was on macOS and, really early on, I ran into all the usual issues: virtual environments breaking because they were based on the system Python and it got updated, virtual environments based on the Homebrew-installed Python and it got updated, etc… Simply put, an occasional, annoying, non-show-stopping breaking of my development environment which would distract me when I’d sat down to just hack on some code, not do system admin!

    My Solution

    For me, what’s worked for me without a problem over the past few years, in short, is this:

    1. NEVER use the system version of Python. Just don’t.
    2. NEVER use the Homebrew’s own version of Python (I’m not even sure this is an issue any more; but it used to be).
    3. NEVER use a version of Python installed with Homebrew (or, more to the point, never install Python with Homebrew).
    4. Manage everything with pyenv; which I do install from Homebrew.

    The Detail

    As mentioned earlier, what I’m writing here assumes that virtual environments are being managed with pipenv (something I still do for personal projects, for now, but this may change soon). Your choices and mileage may vary, etc… This is what works well for me.

    The “one time” items

    These are the items that need initially installing into a new macOS machine:

    Homebrew

    Unless it comes from the Mac App Store, I try and install everything via Homebrew. It’s really handy for keeping track of what I’ve got installed, for recreating a development environment in general, and for keeping things up to date.

    pyenv

    With Homebrew installed the next step for me is to install pyenv. Doing so is as easy as:

    $ brew install pyenv
    

    Once installed, if it’s not done it for you, you may need to make some changes to your login profile. I’m a user of fish so I have these lines in my setup. Simply put: it asks pyenv to set up my environment so my calls to Python go via its setup.

    Plenty of help on how to set up pyenv can be found in its README.

    Once I’ve done this I tend to go on and install the Python versions I’m likely to need. For me this tends to be the most recent “active” stable versions (as of the time of writing, 3.7 through 3.10; although I need to now start throwing 3.11 into the mix too).

    I use this command:

    $ pyenv install --list
    

    to see the available versions. If I want to see what’s available for a specific version I’ll pipe through grep:

    $ pyenv install --list | fgrep "  3.9"
    

    This is handy if I want to check what the very latest release of a specific version of Python is.

    The “Global” Python

    When I’m done with the above I then tend to use pyenv to set my “global” Python. This is the version I want to get when I run python and I’m not inside a virtual environment. Doing that is as simple as:

    $ pyenv global 3.10.7
    

    Of course, you’d swap the version for whatever works for you.

    When Stuff Breaks

    It seems more rare these days, but on occasion I’ve had it such that some update somewhere still causes my environment to break. Now though I find that all it takes is a quick:

    $ pyenv rehash
    

    and everything is good again.

    Setting Up A Repo

    With all of the stuff above being mostly a one-off (or at least something I do once when I set up a new machine – which isn’t often), the real “work” here is when I set up a fresh repository when I start a new project. Really it’s no work at all. Again, as I’ve said before, I’ve tended to use pipenv for my own work, and still do for personal stuff (but do want to change that), mileage may vary here depending on tool.

    When I start a new project I think about which Python version I want to be working with, I ensure I have the latest version of it installed with pyenv, and then ask pipenv to create a new virtual environment with that:

    $ pipenv --python 3.10.7
    

    When you do this, you should see pipenv pulling the version of Python from the pyenv directories:

    $ pipenv --python 3.10.7
    Creating a virtualenv for this project...
    Pipfile: /Users/davep/temp/cool-project/Pipfile
    Using /Users/davep/.pyenv/versions/3.10.7/bin/python3 (3.10.7) to create virtualenv...
    ⠙ Creating virtual environment...created virtual environment CPython3.10.7.final.0-64 in 795ms
      creator CPython3Posix(dest=/Users/davep/temp/cool-project/.venv, clear=False, no_vcs_ignore=False, global=False)
      seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/davep/Library/Application Support/virtualenv)
        added seed packages: pip==22.2.2, setuptools==65.3.0, wheel==0.37.1
      activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
    ✔ Successfully created virtual environment!
    Virtualenv location: /Users/davep/temp/cool-project/.venv
    Creating a Pipfile for this project...
    

    The key thing here is seeing that pipenv is pulling Python from ~/.pyenv/versions/. If it isn’t there’s a good chance you have a Python earlier in your PATH than the pyenv one – something you generally don’t want. It will work, but it’s more likely to break at some point in the future. That’s the key thing I look for; if I see that I know things are going to be okay.

    Conclusion

    Since I adopted these personal rules and approaches (and really, calling them “rules” is kind of grand – there’s almost nothing to this) I’ve found I’ve had near-zero issues with the stability of my Python virtual environments (and what issues I have run into tend to be trivial and of my own doing).

    As I said at the start: there are, of course, other approaches to this, but this is mine and works well for me. Do feel free to comment with your own, I’m always happy to learn about new ways!

  • Build in public, even in private

    As mentioned yesterday, I’m about to start working at Textualize, and building Open-source software is important to the company. Will – the CEO – is all about building in public. If you follow him on Twitter you’ll notice that his Python coding adventure tweets actually outnumber is cooking tweets!

    As someone who has long been a supporter of and fan of Free Software and Open-source software, and has made some small contributions along the way, I’ve also always made a point of building my own tools in public. In most cases they’re things that are likely only helpful to me, but some are more generally useful. The point being though: it’s all there in case it’s helpfull to someone else.

    Which means that, as much as possible, when I’m writing code, I write it as if it’s going to be visible in public and someone else is going to be reading it. I try and make the code tidy. I try and comment it well. I try (but don’t always manage for personal projects) to fully document it. The important thing here being that someone coming to the code fresh should be able to follow what’s going on.

    Against that background, and having just gone through the process of handing off almost 5 years of work to someone else as a left an employer, I got to thinking: we should always “build in public”, even if it’s in private.

    When I started with my previous employer, and even to the day I left, I was the only software developer there. I worked with a team who wrote code, but being software developers wasn’t what they did. Bioinformaticians and machine learning scientists have other things to be doing. But, as I wrote my code, I wrote every line assuming they, or some other developer down the line, would be reading it. Pretty much every line was written for an audience I couldn’t see and didn’t fully know. This, as mentioned above, meant trying to keep the code clean, ensuring it was commented in helpful ways, ensuring the documentation was helpful, and so on.

    But it wasn’t just about the code. Any non-trivial system will have more to it than code. We had an internal instance of GitLab and I tracked all of my work on there. So, as I planned and worked on new features, or went on bug hunts, I’d document the process in the issue tracker. As much as possible I’d be really verbose about the process. Often I wouldn’t just open an issue, go work on it, and then mark it closed; as I worked through the issue I’d add comment after comment under it, documenting my thinking, problems, solutions, cite sources when looking something up, that sort of thing.

    The whole process was an act in having a conversation with current or future team members if they ever needed to look; with future me (really, that helped more than once – we all have those “that the hell was I thinking?” moments); with any developer(s) who took over from me in the future.

    I did all this as if I was broadcasting it in public on Twitter or on GitHub, etc. It was in private, of course, but I approached it as if it was in public.

    There were always three main reasons for this, I felt:

    1. Accountability. At any moment someone who I worked with could review what I was doing and why I was doing it; it was an invitation to anyone curious enough to talk with me about what I was building and how I was building it.
    2. Continuity of support for unplanned reasons. Life happens, sometimes you may, unplanned, never be available at work again. I never wanted to leave my employer in a position where picking up from such an event was a nightmare.
    3. Continuity of support for planned reasons. It was possible, and it became inevitable, that I’d move on to something else. If that was to happen I wanted to be sure that whoever picked up after me would be able to do so without too much effort.

    In the end item 3 seemed to really pay off. When it came time for me to hand over my work to someone else, as I left, the process was really smooth and trouble-free. I was able to point the developer at all the documentation and source code, at all the issues, and invite them to read through it all and then come back to me with questions. In terms of time actually spent talking about the main system I was handing over I’d say that 4 years of work was handed over with just a few hours of actual talking about it.

    It remains to be seen if it really paid off, of course. If they get really stuck they do have an open invitation to ping me a question or two; I care enough about what I designed and built that I want it to carry on being useful for some time to come. But… I like to think that all of that building in public, in private, will ensure that this is an invitation that never needs to be called on. I like to think that, if something isn’t clear, they’ll be able to check the code, the documentation and the issue history and get to where they need to go.

  • On to something new (redux)

    Just over five years ago I got a message from my then employer to say I was going to be made redundant after 21 years working for them. After the 3 month notice period the final day came. Meanwhile, I found something new that looked terrifying but interesting. In the end it was less terrifying and way more interesting than I imagined it would be. It was fun too.

    But… (there’s always a but isn’t there?)

    In the four and change years I’ve been there the company got bought out, and then the result of that got bought up. As I’ve mentioned before I’m generally not a “big company” kind of person; in all my years I’ve found that I’m happier working in a smaller place. After a couple of buyouts my employer had gone from being 10s of people in size to 100s of people in size (and technically 10s of 1,000s of people in size depending on how you look at it).

    This change in ownership and size meant the culture became… well, let’s just say not as friendly as you tend to enjoy when it’s a smaller group of folk. On top of that I was starting to notice that my efforts were making less of an impact as things got bigger, and I started to feel like my contributions weren’t really relevant any more. There were some problematic things happening too: undermining of efforts, removal of responsibilities without consultation or communication, that sort of thing. Plus worse. There’s little point in going into the detail, but it’s fair to say that work wasn’t as fun as it used to be.

    That felt like a good time to start to look around. If work makes you feel unhappy and you can look around… look around.

    Thing is, I wasn’t sure what to look for. I was in the comfortable position of, unlike last time, not needing to find something, so I could take my time at least. Over the course of the last year I’ve spoken to many different companies and organisations, some big (yes, I know, I said I don’t like big places – sometimes what’s on offer deserves a fair hearing), some small, but none of them quite said “this feels like me”. In some cases the whole thing didn’t have the right vibe, in others the industry either didn’t interest me, or felt uncomfortable given my personal values. In one particular case a place looked interesting until I checked the CTO’s socials and OMG NO NO NO AVOID AVOID (that was a fun one).

    Then I saw Will McGugan saying he was hiring to expand Textualize. This caught my interest right away for two good reasons.

    I can’t remember how long I’ve been following Will on Twitter; I likely stumbled on him as I got back into Python in 2018 and I also remember noting that he was a Python hacker just up the road from me. We’d vaguely chatted on Twitter, briefly, in that “Twitter acquaintance” way we all often do (I remember one brief exchange about fungus on The Meadows), and he’d seemed like a good sort. A small company run by a “good sort” kinda person felt like a damn good reason.

    The second reason was Textual itself. I’d been watching Will develop it, in open, with great interest. I had (and still have) a plan to write a brand new CHUI-based (okay fine TUI-based as the kids like to say these days!) Norton Guide reader, all in Python, and Textual looked like the perfect framework to do the UI in. The chance to be involved with it sounded awesome.

    Now, I said two reasons, but there’s also a third I guess: Will’s pitch for applying to Textualize felt so damn accessible! I’m on the older end of the age range of this industry; for much of my working life as a developer I’ve worked in isolation from other developers; while I first touched Python in the 90s, I’ve only been using it in anger since 2018 and still feel like I’ve got a lot to learn. Despite all these things, and more, saying “aye Dave this is beyond you” I felt comfortable dropping Will a line.

    Which resulted in a chat.

    Which resulted in some code tinkering and chatting.

    Which resulted in…

    Something new.

    So, yeah, as of 2022-10-10 I’m on yet another new adventure. Time for me to really work on my Python coding as I work with Will and the rest of the team as part of Textualize.

    Or, as I put it on Twitter a few days ago: I’m going to be a Python impostor syndrome speedrunner!

  • I must be getting old

    A little earlier this evening I got a new issue raised against boxquote.el. Apparently Emacs 29 (I’m running 28.1 as of the time of writing) is moaning about the likes of:

    (setf (point) some-location-or-other)
    

    and

    (setf (buffer-string) "")
    

    There’s a whole background to why I’ve tended to code like that, that stems from enjoying Common Lisp, my days reading (and sometimes posting to) comp.lang.lisp, and I think some of the stuff Erik Naggum wrote back in the day. I won’t get into it all now; I’m not sure I can even remember a lot of how I got there given how far back it was.

    But…

    Wanting to quickly get to the bottom of why the above was suddenly an issue, I dived into the NEWS file and found the following:

    ** Many seldom-used generalized variables have been made obsolete.
    Emacs has a number of rather obscure generalized variables defined,
    that, for instance, allowed you to say things like:
    
       (setf (point-min) 4)
    
    These never caught on and have been made obsolete.  The form above,
    for instance, is the same as saying
    
       (narrow-to-region 4 (point-max))
    
    The following generalized variables have been made obsolete:
    'buffer-file-name', 'buffer-local-value', 'buffer-modified-p',
    'buffer-name', 'buffer-string', 'buffer-substring', 'current-buffer',
    'current-column', 'current-global-map', 'current-input-mode',
    'current-local-map', 'current-window-configuration',
    'default-file-modes', 'documentation-property', 'frame-height',
    'frame-visible-p', 'global-key-binding', 'local-key-binding', 'mark',
    'mark-marker', 'marker-position', 'mouse-position', 'point',
    'point-marker', 'point-max', 'point-min', 'read-mouse-position',
    'screen-height', 'screen-width', 'selected-frame', 'selected-screen',
    'selected-window', 'standard-case-table', 'syntax-table',
    'visited-file-modtime', 'window-height', 'window-width', and
    'x-get-secondary-selection'.
    

    As suggested above… this is my thing, this is how I coded some Elisp stuff. Look through much of my Emacs Lisp code and you’ll find me setfing stuff all over the place.

    Apparently my style is “obscure”. Actually, I’m kinda okay with that if I’m honest.

    This is going to be a bit of a pain in the arse; I’m going to have to go through a whole bunch of code and make it “less obscure”, at some point.

    But…

    This isn’t the part that had me thinking I must be getting old. Oh no. The NEWS file had another little surprise in store:

    ** The quickurl.el library is now obsolete.
    Use 'abbrev', 'skeleton' or 'tempo' instead.
    

    That…. that’s me that is. Well, it’s one of the me things. If you run about-emacs, dive into Authors, and search for my name, in any copy of GNU Emacs from the last decade or two, you’ll find this:

    Dave Pearson: wrote 5x5.el quickurl.el
    

    quickurl.el was a package I wrote back in the late 1990s, back when I was a very heavy user of Usenet, and often found myself posting the same URLs in posts again and again; especially in comp.lang.clipper. As a fairly quick hack I wrote the code so that I could very quickly insert often-used URLs.

    Some time later, I got an email from the FSF (I actually think it was from RMS – but that’s an mbox I’ve long ago lost – or a backup of it might be in storage back in England, on a DVD), asking if I wanted to contribute it to Emacs proper. This seemed like an odd thing to add to Emacs but, sure, why the hell not?

    And so I had my second contribution to a body of code I used a lot (the first being 5x5.el – which itself was my first ever attempt at writing some non-trivial Elisp code).

    So… yeah… here we are. I’m now old enough to have written some Emacs Lisp code, had it required by the FSF for inclusion in Emacs, had it live in there for something like two decades, and then become obselete!

subscribe via RSS