Ever since GitHub introduced the profile README¹ I've had a massively low-effort one in place. I made the repo, quickly wrote the file, and then sort of forgot about it. Well, I didn't so much forget as just keep looking at it and thinking "I should do something better with that one day".

Thing is, while there are lots of fancy approaches out there, and lots of neat generator tools and the like... they just weren't for me.

Then yesterday, over my second morning coffee, after getting my blog environment up and going again, I had an idea. It could be cool to use Textual's screenshot facility to make something terminal-themed! I mean, while it's not all I am these days, so much of what I'm doing right now is aimed at the terminal.

So... what to do? Then I thought it could be cool to knock up some sort of login screen type thing; with a banner. One visit to an online large terminal text generator site later, I had some banner text. All that was left was to write a simple Textual application to create the "screen".

The main layout is simple enough:

def compose(self) -> ComposeResult:
    yield Label(NAME, classes="banner")
    yield Label(PRATTLE)
    yield Label("github.com/davep login: [reverse] [/]")

where NAME contains the banner and PRATTLE contains the "login message". With some Textual CSS sprinkled over it to give the exact layout and colour I wanted, all that was left was to make the snapshot. This was easy enough too.

While the whole thing isn't fully documented just yet, Textual does have a great tool for automatically running an application and interacting with it; that meant I could easily write a function to load up my app and save the screenshot:

async def make_banner() -> None:
    async with GitHubBannerApp().run_test() as pilot:
        pilot.app.save_screenshot("davep.svg")

Of course, that needs running async, but that's simple enough:

if __name__ == "__main__":
    asyncio.run(make_banner())

Throw in a Makefile so I don't forget what I'm supposed to run:

.PHONY: all
all:
    pipenv run python make_banner.py

and that's it! Job done!

From here onward I guess I could have some real fun with this. It would be simple enough I guess to modify the code so that it changes what's displayed over time; perhaps show a "last login" value that relates to recently activity or something; any number of things; and then run it in a cron job and update the repository.

For now though... I'll stick with keeping things nice and simple.

Introduction¶

Patience, Highlander. You have done well. But it'll take time. You are generations being born and dying. You are at one with all living things. Each man's thoughts and dreams are yours to know. You have power beyond imagination. Use it well, my friend. Don't lose your head.

Juan Sánchez Villalobos Ramírez, Chief metallurgist to King Charles V of Spain

As of the time of writing, I'm a couple or so days off having been with Textualize for 3 months. It's been fun, and educational, and every bit as engaging as I'd hoped, and more. One thing I hadn't quite prepared for though, but which I really love, is how so many other people are learning Textual along with me.

Even in those three months the library has changed and expanded quite a lot, and it continues to do so. Meanwhile, more people are turning up and using the framework; you can see this online in social media, blogs and of course in the ever-growing list of projects on GitHub which depend on Textual.

This inevitably means there's a lot of people getting to grips with a new tool, and one that is still a bit of a moving target. This in turn means lots of people are coming to us to get help.

As I've watched this happen I've noticed a few patterns emerging. Some of these good or neutral, some... let's just say not really beneficial to those seeking the help, or to those trying to provide the help. So I wanted to write a little bit about the different ways you can get help with Textual and your Textual-based projects, and to also try and encourage people to take the most helpful and positive approach to getting that help.

Now, before I go on, I want to make something very clear: I'm writing this as an individual. This is my own personal view, and my own advice from me to anyone who wishes to take it. It's not Textual (the project) or Textualize (the company) policy, rules or guidelines. This is just some ageing hacker's take on how best to go about asking for help, informed by years of asking for and also providing help in email, on Usenet, on forums, etc.

Or, put another way: if what you read in here seems sensible to you, I figure we'll likely have already hit it off over on GitHub or in the Discord server. ;-)

Where to go for help¶

At this point this is almost a bit of an FAQ itself, so I thought I'd address it here: where's the best place to ask for help about Textual, and what's the difference between GitHub Issues, Discussions and our Discord server?

I'd suggest thinking of them like this:

Discord¶

You have a question, or need help with something, and perhaps you could do with a reply as soon as possible. But, and this is the really important part, it doesn't matter if you don't get a response. If you're in this situation then the Discord server is possibly a good place to start. If you're lucky someone will be hanging about who can help out.

I can't speak for anyone else, but keep this in mind: when I look in on Discord I tend not to go scrolling back much to see if anything has been missed. If something catches my eye, I'll try and reply, but if it doesn't... well, it's mostly an instant chat thing so I don't dive too deeply back in time.

My own advice would be to treat Discord as an ephemeral resource. It happens in the moment but fades away pretty quickly. It's like knocking on a friend's door to see if they're in. If they're not in, you might leave them a note, which is sort of like going to...

GitHub¶

On the other hand, if you have a question or need some help or something where you want to stand a good chance of the Textual developers (amongst others) seeing it and responding, I'd recommend that GitHub is the place to go. Dropping something into the discussions there, or leaving an issue, ensures it'll get seen. It won't get lost.

As for which you should use -- a discussion or an issue -- I'd suggest this: if you need help with something, or you want to check your understanding of something, or you just want to be sure something is a problem before taking it further, a discussion might be the best thing. On the other hand, if you've got a clear bug or feature request on your hands, an issue makes a lot of sense.

Don't worry if you're not sure which camp your question or whatever falls into though; go with what you think is right. There's no harm done either way (I may move an issue to a discussion first before replying, if it's really just a request for help -- but that's mostly so everyone can benefit from finding it in the right place later on down the line).

The dos and don'ts of getting help¶

Now on to the fun part. This is where I get a bit preachy. Ish. Kinda. A little bit. Again, please remember, this isn't a set of rules, this isn't a set of official guidelines, this is just a bunch of "if you want my advice, and I know you didn't ask but you've read this far so you actually sort of did, don't say I didn't warn you!" waffle.

This isn't going to be an exhaustive collection, far from it. But I feel these are some important highlights.

Do...¶

When looking for help, in any of the locations mentioned above, I'd totally encourage:

Be clear and detailed¶

Too much detail is almost always way better than not enough. "My program didn't run", often even with some of the code supplied, is so much harder to help than "I ran this code I'm posting here, and I expected this particular outcome, and I expected it because I'd read this particular thing in the docs and had comprehended it to mean this, but instead the outcome was this exception here, and I'm a bit stuck -- can someone offer some pointers?"

The former approach means there often ends up having to be a back and forth which can last a long time, and which can sometimes be frustrating for the person asking. Manage frustration: be clear, tell us everything you can.

Say what resources you've used already¶

If you've read the portions of the documentation that relate to what you're trying to do, it's going to be really helpful if you say so. If you don't, it might be assumed you haven't and you may end up being pointed at them.

So, please, if you've checked the documentation, looked in the FAQ, done a search of past issues or discussions or perhaps even done a search on the Discord server... please say so.

Be polite¶

This one can go a long way when looking for help. Look, I get it, programming is bloody frustrating at times. We've all rage-quit some code at some point, I'm sure. It's likely going to be your moment of greatest frustration when you go looking for help. But if you turn up looking for help acting all grumpy and stuff it's not going to come over well. Folk are less likely to be motivated to lend a hand to someone who seems rather annoyed.

If you throw in a please and thank-you here and there that makes it all the better.

Fully consider the replies¶

You could find yourself getting a reply that you're sure won't help at all. That's fair. But be sure to fully consider it first. Perhaps you missed the obvious along the way and this is 100% the course correction you'd unknowingly come looking for in the first place. Sure, the person replying might have totally misunderstood what was being asked, or might be giving a wrong answer (it me! I've totally done that and will again!), but even then a reply along the lines of "I'm not sure that's what I'm looking for, because..." gets everyone to the solution faster than "lol nah".

Entertain what might seem like odd questions¶

Aye, I get it, being asked questions when you're looking for an answer can be a bit frustrating. But if you find yourself on the receiving end of a small series of questions about your question, keep this in mind: Textual is still rather new and still developing and it's possible that what you're trying to do isn't the correct way to do that thing. To the person looking to help you it may seem to them you have an XY problem.

Entertaining those questions might just get you to the real solution to your problem.

Allow for language differences¶

You don't need me to tell you that a project such as Textual has a global audience. With that rather obvious fact comes the other fact that we don't all share the same first language. So, please, as much as possible, try and allow for that. If someone is trying to help you out, and they make it clear they're struggling to follow you, keep this in mind.

Acknowledge the answer¶

I suppose this is a variation on "be polite" (really, a thanks can go a long way), but there's more to this than a friendly acknowledgement. If someone has gone to the trouble of offering some help, it's helpful to everyone who comes after you to acknowledge if it worked or not. That way a future help-seeker will know if the answer they're reading stands a chance of being the right one.

Accept that Textual is zero-point software (right now)¶

Of course the aim is to have every release of Textual be stable and useful, but things will break. So, please, do keep in mind things like:

• Textual likely doesn't have your feature of choice just yet.
• We might accidentally break something (perhaps pinning Textual and testing each release is a good plan here?).
• We might deliberately break something because we've decided to take a particular feature or way of doing things in a better direction.

Of course it can be a bit frustrating at times, but overall the aim is to have the best framework possible in the long run.

Don't...¶

Okay, now for a bit of old-hacker finger-wagging. Here's a few things I'd personally discourage:

Lack patience¶

Sure, it can be annoying. You're in your flow, you've got a neat idea for a thing you want to build, you're stuck on one particular thing and you really need help right now! Thing is, that's unlikely to happen. Badgering individuals, or a whole resource, to reply right now, or complaining that it's been $TIME_PERIOD since you asked and nobody has replied... that's just going to make people less likely to reply.

Unnecessarily tag individuals¶

This one often goes hand in hand with the "lack patience" thing: Be it asking on Discord, or in GitHub issues, discussions or even PRs, unnecessarily tagging individuals is a bit rude. Speaking for myself and only myself: I love helping folk with Textual. If I could help everyone all the time the moment they have a problem, I would. But it doesn't work like that. There's any number of reasons I might not be responding to a particular request, including but not limited to (here I'm talking personally because I don't want to speak for anyone else, but I'm sure I'm not alone here):

• I have a job. Sure, my job is (in part) Textual, but there's more to it than that particular issue. I might be doing other stuff.
• I have my own projects to work on too. I like coding for fun as well (or writing preaching old dude blog posts like this I guess, but you get the idea).
• I actually have other interests outside of work hours so I might actually be out doing a 10k in the local glen, or battling headcrabs in VR, or something.
• Housework. :-/

You get the idea though. So while I'm off having a well-rounded life, it's not good to get unnecessarily intrusive alerts to something that either a) doesn't actually directly involve me or b) could wait.

Seek personal support¶

Again, I'm going to speak totally for myself here, but I also feel the general case is polite for all: there's a lot of good support resources available already; sending DMs on Discord or Twitter or in the Fediverse, looking for direct personal support, isn't really the best way to get help. Using the public/collective resources is absolutely the best way to get that help. Why's it a bad idea to dive into DMs? Here's some reasons I think it's not a good idea:

• It's a variation on "unnecessarily tagging individuals".
• You're short-changing yourself when it comes to getting help. If you ask somewhere more public you're asking a much bigger audience, who collectively have more time, more knowledge and more experience than a single individual.
• Following on from that, any answers can be (politely) fact-checked or enhanced by that audience, resulting in a better chance of getting the best help possible.
• The next seeker-of-help gets to miss out on your question and the answer. If asked and answered in public, it's a record that can help someone else in the future.

Doubt your ability or skill level¶

I suppose this should really be phrased as a do rather than a don't, as here I want to encourage something positive. A few times I've helped people out who have been very apologetic about their questions being "noob" questions, or about how they're fairly new to Python, or programming in general. Really, please, don't feel the need to apologise and don't be ashamed of where you're at.

If you've asked something that's obviously answered in the documentation, that's not a problem; you'll likely get pointed at the docs and it's what happens next that's the key bit. If the attitude is "oh, cool, that's exactly what I needed to be reading, thanks!" that's a really positive thing. The only time it's a problem is when there's a real reluctance to use the available resources. We've all seen that person somewhere at some point, right? ;-)

Not knowing things is totally cool.

Conclusion¶

So, that's my waffle over. As I said at the start: this is my own personal thoughts on how to get help with Textual, both as someone whose job it is to work on Textual and help people with Textual, and also as a FOSS advocate and supporter who can normally be found helping Textual users when he's not "on the clock" too.

What I've written here isn't exhaustive. Neither is it novel. Plenty has been written on the general subject in the past, and I'm sure more will be written on the subject in the future. I do, however, feel that these are the most common things I notice. I'd say those dos and don'ts cover 90% of "can I get some help?" interactions; perhaps closer to 99%.

Finally, and I think this is the most important thing to remember, the next time you are battling some issue while working with Textual: don't lose your head!

That didn't go to plan¶

So... yeah... the dogfooding... When I wrote my previous post I had wanted to try and do a post towards the end of each week, highlighting what I'd done on the "dogfooding" front. Life kinda had other plans. Not in a terrible way, but it turns out that getting both flu and Covid jabs (AKA "jags" as they tend to say in my adopted home) on the same day doesn't really agree with me too well.

I have been working, but there's been some odd moments in the past week and a bit and, last week, once I got to the end, I was glad for it to end. So no blog post happened.

Anyway...

What have I been up to?¶

While mostly sat feeling sorry for myself on my sofa, I have been coding. Rather than list all the different things here in detail, I'll quickly mention them with links to where to find them and play with them if you want:

FivePyFive¶

While my Textual 5x5 puzzle is one of the examples in the Textual repo, I wanted to make it more widely available so people can download it with pip or pipx. See over on PyPI and see if you can solve it. ;-)

textual-qrcode¶

I wanted to put together a very small example of how someone may put together a third party widget library, and in doing so selected what I thought was going to be a mostly-useless example: a wrapper around a text-based QR code generator website. Weirdly I've had a couple of people express a need for QR codes in the terminal since publishing that!

PISpy¶

PISpy is a very simple terminal-based client for the PyPI API. Mostly it provides a hypertext interface to Python package details, letting you look up a package and then follow its dependency links. It's very simple at the moment, but I think more fun things can be done with this.

OIDIA¶

I'm a big fan of the use of streak-tracking in one form or another. Personally I use a streak-tracking app for keeping tabs of all sorts of good (and bad) habits, and as a heavy user of all things Apple I make a lot of use of the Fitness rings, etc. So I got to thinking it might be fun to do a really simple, no shaming, no counting, just recording, streak app for the Terminal. OIDIA is the result.

As of the time of writing I only finished the first version of this yesterday evening, so there are plenty of rough edges; but having got it to a point where it performed the basic tasks I wanted from it, that seemed like a good time to publish.

Expect to see this getting more updates and polish.

Wait, what about this Keymaster thing?¶

Ahh, yes, about that... So one of the handy things I'm finding about Textual is its key binding system. The more I build Textual apps, the more I appreciate the bindings, how they can be associated with specific widgets, the use of actions (which can be used from other places too), etc.

But... (there's always a "but" right -- I mean, there'd be no blog post to be had here otherwise).

The terminal doesn't have access to all the key combinations you may want to use, and also, because some keys can't necessarily be "typed", at least not easily (think about it: there's no F1 character, you have to type F1), many keys and key combinations need to be bound with specific names.

So there's two problems here: how do I discover what keys even turn up in my application, and when they do, what should I call them when I pass them to Binding?

That felt like a "well Dave just build an app for it!" problem. So I did:

If you're building apps with Textual and you want to discover what keys turn up from your terminal and are available to your application, you can:

pipx install textual-keys

and then just run textual-keys and start mashing the keyboard to find out.

There's a good chance that this app, or at least a version of it, will make it into Textual itself (very likely as one of the devtools). But for now it's just an easy install away.

I think there's a call to be made here too: have you built anything to help speed up how you work with Textual, or just make the development experience "just so"? If so, do let folk know, and come yell about it on the #show-and-tell channel in the Discord server.

Introduction¶

Cutler, armed with a schedule, was urging the team to "eat its own dog food". Part macho stunt and part common sense, the "dog food diet" was the cornerstone of Cutler’s philosophy.

G. Pascal Zachary — Show-Stopper!

I can't remember exactly when it was -- it was likely late in 1994 or some time in 1995 -- when I first came across the concept of, or rather the name for the concept of, "eating your own dog food". The idea and the name played a huge part in the book Show-Stopper! by G. Pascal Zachary. The idea wasn't new to me of course; I'd been writing code for over a decade by then and plenty of times I'd built things and then used those things to do things, but it was fascinating to a mostly-self-taught 20-something me to be reading this (excellent -- go read it if you care about the history of your craft) book and to see the idea written down and named.

While Textualize isn't (thankfully -- really, I do recommend reading the book) anything like working on the team building Windows NT, the idea of taking a little time out from working on Textual, and instead work with Textual, makes a lot of sense. It's far too easy to get focused on adding things and improving things and tweaking things while losing sight of the fact that people will want to build with your product.

So you can imagine how pleased I was when Will announced that he wanted all of us to spend a couple or so weeks building something with Textual. I had, of course, already written one small application with the library, and had plans for another (in part it's how I ended up working here), but I'd yet to really dive in and try and build something more involved.

Giving it some thought: I wasn't entirely sure what I wanted to build though. I do want to use Textual to build a brand new terminal-based Norton Guide reader (not my first, not by a long way) but I felt that was possibly a bit too niche, and actually could take a bit too long anyway. Maybe not, it remains to be seen¹.

Eventually I decided on this approach: try and do a quick prototype of some daft idea each day or each couple of days, do that for a week or so, and then finally try and settle down on something less trivial. This approach should work well in that it'll help introduce me to more of Textual, help try out a few different parts of the library, and also hopefully discover some real pain-points with working with it and highlight a list of issues we should address -- as seen from the perspective of a developer working with the library.

So, here I am, at the end of week one. What I want to try and do is briefly (yes yes, I know, this introduction is the antithesis of brief) talk about what I built and perhaps try and highlight some lessons learnt, highlight some patterns I think are useful, and generally do an end-of-week version of a TIL. TWIL?

Yeah. I guess this is a TWIL.

gridinfo¶

I started the week by digging out a quick hack I'd done a couple of weeks earlier, with a view to cleaning it up. It started out as a fun attempt to do something with Rich Pixels while also making a terminal-based take on slstats.el. I'm actually pleased with the result and how quickly it came together.

The point of the application itself is to show some general information about the current state of the Second Life grid (hello to any fellow residents of the original Metaverse!), and to also provide a simple region lookup screen that, using Rich Pixels, will display the object map (albeit in pretty low resolution -- but that's the fun of this!).

So the opening screen looks like this:

and a lookup of a region looks like this:

Here's a wee video of the whole thing in action:

Worth a highlight¶

Here's a couple of things from the code that I think are worth a highlight, as things to consider when building Textual apps:

Don't use the default screen¶

Use of the default Screen that's provided by the App is handy enough, but I feel any non-trivial application should really put as much code as possible in screens that relate to key "work". Here's the entirety of my application code:

class GridInfo( App[ None ] ):
    """TUI app for showing information about the Second Life grid."""

    CSS_PATH = "gridinfo.css"
    """The name of the CSS file for the app."""

    TITLE = "Grid Information"
    """str: The title of the application."""

    SCREENS = {
        "main": Main,
        "region": RegionInfo
    }
    """The collection of application screens."""

    def on_mount( self ) -> None:
        """Set up the application on startup."""
        self.push_screen( "main" )

You'll notice there's no work done in the app, other than to declare the screens, and to set the main screen running when the app is mounted.

Don't work hard on_mount¶

My initial version of the application had it loading up the data from the Second Life and GridSurvey APIs in Main.on_mount. This obviously wasn't a great idea as it made the startup appear slow. That's when I realised just how handy call_after_refresh is. This meant I could show some placeholder information and then fire off the requests (3 of them: one to get the main grid information, one to get the grid concurrency data, and one to get the grid size data), keeping the application looking active and updating the display when the replies came in.

Pain points¶

While building this app I think there was only really the one pain-point, and I suspect it's mostly more on me than on Textual itself: getting a good layout and playing whack-a-mole with CSS. I suspect this is going to be down to getting more and more familiar with CSS and the terminal (which is different from laying things out for the web), while also practising with various layout schemes -- which is where the revamped Placeholder class is going to be really useful.

unbored¶

The next application was initially going to be a very quick hack, but actually turned into a less-trivial build than I'd initially envisaged; not in a negative way though. The more I played with it the more I explored and I feel that this ended up being my first really good exploration of some useful (personal -- your kilometerage may vary) patterns and approaches when working with Textual.

The application itself is a terminal client for the Bored-API. I had initially intended to roll my own code for working with the API, but I noticed that someone had done a nice library for it and it seemed silly to not build on that. Not needing to faff with that, I could concentrate on the application itself.

At first I was just going to let the user click away at a button that showed a random activity, but this quickly morphed into a "why don't I make this into a sort of TODO list builder app, where you can add things to do when you are bored, and delete things you don't care for or have done" approach.

Here's a view of the main screen:

and here's a view of the filter pop-over:

Worth a highlight¶

Don't put all your BINDINGS in one place¶

This came about from me overloading the use of the escape key. I wanted it to work more or less like this:

• If you're inside an activity, move focus up to the activity type selection buttons.
• If the filter pop-over is visible, close that.
• Otherwise exit the application.

It was easy enough to do, and I had an action in the Main screen that escape was bound to (again, in the Main screen) that did all this logic with some if/elif work but it didn't feel elegant. Moreover, it meant that the Footer always displayed the same description for the key.

That's when I realised that it made way more sense to have a Binding for escape in every widget that was the actual context for escape's use. So I went from one top-level binding to...

...

class Activity( Widget ):
    """A widget that holds and displays a suggested activity."""

    BINDINGS = [
        ...
        Binding( "escape", "deselect", "Switch to Types" )
    ]

...

class Filters( Vertical ):
    """Filtering sidebar."""

    BINDINGS = [
        Binding( "escape", "close", "Close Filters" )
    ]

...

class Main( Screen ):
    """The main application screen."""

    BINDINGS = [
        Binding( "escape", "quit", "Close" )
    ]
    """The bindings for the main screen."""

This was so much cleaner and I got better Footer descriptions too. I'm going to be leaning hard on this approach from now on.

Messages are awesome¶

Until I wrote this application I hadn't really had a need to define or use my own Messages. During work on this I realised how handy they really are. In the code I have an Activity widget which takes care of the job of moving itself amongst its siblings if the user asks to move an activity up or down. When this happens I also want the Main screen to save the activities to the filesystem as things have changed.

Thing is: I don't want the screen to know what an Activity is capable of and I don't want an Activity to know what the screen is capable of; especially the latter as I really don't want a child of a screen to know what the screen can do (in this case "save stuff").

This is where messages come in. Using a message I could just set things up so that the Activity could shout out "HEY I JUST DID A THING THAT CHANGES ME" and not care who is listening and not care what they do with that information.

So, thanks to this bit of code in my Activity widget...

    class Moved( Message ):
        """A message to indicate that an activity has moved."""

    def action_move_up( self ) -> None:
        """Move this activity up one place in the list."""
        if self.parent is not None and not self.is_first:
            parent = cast( Widget, self.parent )
            parent.move_child(
                self, before=parent.children.index( self ) - 1
            )
            self.emit_no_wait( self.Moved( self ) )
            self.scroll_visible( top=True )

...the Main screen can do this:

    def on_activity_moved( self, _: Activity.Moved ) -> None:
        """React to an activity being moved."""
        self.save_activity_list()

Pain points¶

On top of the issues of getting to know terminal-based-CSS that I mentioned earlier:

• Textual currently lacks any sort of selection list or radio-set widget. This meant that I couldn't quite do the activity type picking how I would have wanted. Of course I could have rolled my own widgets for this, but I think I'd sooner wait until such things are in Textual itself.
• Similar to that, I could have used some validating Input widgets. They too are on the roadmap but I managed to cobble together fairly good working versions for my purposes. In doing so though I did further highlight that the reactive attribute facility needs a wee bit more attention as I ran into some (already-known) bugs. Thankfully in my case it was a very easy workaround.
• Scrolling in general seems a wee bit off when it comes to widgets that are more than one line tall. While there's nothing really obvious I can point my finger at, I'm finding that scrolling containers sometimes get confused about what should be in view. This becomes very obvious when forcing things to scroll from code. I feel this deserves a dedicated test application to explore this more.

Conclusion¶

The first week of "dogfooding" has been fun and I'm more convinced than ever that it's an excellent exercise for Textualize to engage in. I didn't quite manage my plan of "one silly trivial prototype per day", which means I've ended up with two (well technically one and a half I guess given that gridinfo already existed as a prototype) applications rather than four. I'm okay with that. I got a lot of utility out of this.

Now to look at the list of ideas I have going and think about what I'll kick next week off with...

As mentioned yesterday, I'm about to start working at Textualize, and building Open-source software is important to the company. Will -- the CEO -- is all about building in public. If you follow him on Twitter you'll notice that his Python coding adventure tweets actually outnumber is cooking tweets!

As someone who has long been a supporter of and fan of Free Software and Open-source software, and has made some small contributions along the way, I've also always made a point of building my own tools in public. In most cases they're things that are likely only helpful to me, but some are more generally useful. The point being though: it's all there in case it's helpful to someone else.

Which means that, as much as possible, when I'm writing code, I write it as if it's going to be visible in public and someone else is going to be reading it. I try and make the code tidy. I try and comment it well. I try (but don't always manage for personal projects) to fully document it. The important thing here being that someone coming to the code fresh should be able to follow what's going on.

Against that background, and having just gone through the process of handing off almost 5 years of work to someone else as a left an employer, I got to thinking: we should always "build in public", even if it's in private.

When I started with my previous employer, and even to the day I left, I was the only software developer there. I worked with a team who wrote code, but being software developers wasn't what they did. Bioinformaticians and machine learning scientists have other things to be doing. But, as I wrote my code, I wrote every line assuming they, or some other developer down the line, would be reading it. Pretty much every line was written for an audience I couldn't see and didn't fully know. This, as mentioned above, meant trying to keep the code clean, ensuring it was commented in helpful ways, ensuring the documentation was helpful, and so on.

But it wasn't just about the code. Any non-trivial system will have more to it than code. We had an internal instance of GitLab and I tracked all of my work on there. So, as I planned and worked on new features, or went on bug hunts, I'd document the process in the issue tracker. As much as possible I'd be really verbose about the process. Often I wouldn't just open an issue, go work on it, and then mark it closed; as I worked through the issue I'd add comment after comment under it, documenting my thinking, problems, solutions, cite sources when looking something up, that sort of thing.

The whole process was an act in having a conversation with current or future team members if they ever needed to look; with future me (really, that helped more than once -- we all have those "that the hell was I thinking?" moments); with any developer(s) who took over from me in the future.

I did all this as if I was broadcasting it in public on Twitter or on GitHub, etc. It was in private, of course, but I approached it as if it was in public.

There were always three main reasons for this, I felt:

1. Accountability. At any moment someone who I worked with could review what I was doing and why I was doing it; it was an invitation to anyone curious enough to talk with me about what I was building and how I was building it.
2. Continuity of support for unplanned reasons. Life happens, sometimes you may, unplanned, never be available at work again. I never wanted to leave my employer in a position where picking up from such an event was a nightmare.
3. Continuity of support for planned reasons. It was possible, and it became inevitable, that I'd move on to something else. If that was to happen I wanted to be sure that whoever picked up after me would be able to do so without too much effort.

In the end, item 3 seemed to really pay off. When it came time for me to hand over my work to someone else, as I left, the process was really smooth and trouble-free. I was able to point the developer at all the documentation and source code, at all the issues, and invite them to read through it all and then come back to me with questions. In terms of time actually spent talking about the main system I was handing over I'd say that 4 years of work was handed over with just a few hours of actual talking about it.

It remains to be seen if it really paid off, of course. If they get really stuck they do have an open invitation to ping me a question or two; I care enough about what I designed and built that I want it to carry on being useful for some time to come. But... I like to think that all of that building in public, in private, will ensure that this is an invitation that never needs to be called on. I like to think that, if something isn't clear, they'll be able to check the code, the documentation and the issue history and get to where they need to go.

Introduction

This post will cover the most important content of a 2bit file: the actual sequence data itself. In the first post I wrote about the format of the file's header, and in the second post I wrote about the content of the file's index.

At this point that's enough information to know what's in the file and where to find it. In other words we know the list of sequences that live in the file, and we know where each one is positioned within the file. So, assuming we have our index in memory (ideally some sort of key/value store of sequences names and their offsets in the file), given the name of a sequence we can know where to go in the file to load up the data.

So the next obvious question is, what will we find when we get there? Actual sequence data is stored like this:

Breaking the above down:

N blocks

As mentioned in passing in the first post: technically it's necessary to encode 5 different characters for the bases in the sequences. As well as the usual T, C, A and G, there also needs to be an N, which means the base is unknown. Now, of course, you can't pack 5 states into two bits, so the 2bit file format solves this by having an array of block positions and sizes where any data in the actual DNA itself should be ignored and an N used in its place.

Mask blocks

This is where my ignorance of bioinformatics shows, and where it's made very obvious that I'm a software developer who likes to muck about with data and data structures, but who doesn't always understand why they're used. I'm actually not sure what purpose mask blocks serve in a 2bit file, but they do affect the output. If a base falls within a mask block the value that is output should be a lower-case letter, rather then upper-case.

The DNA data

So this is the fun bit, where the real data is stored. This should be viewed as a sequence of bytes, each of which contains 4 bases (except for the last byte, of course, which might contain 1, 2 or 3 depending on the size of the sequence).

Each byte should be viewed as an array of 2 bit values, with the values mapping like this:

So, given a byte whose value is 27, you're looking at the sequence TCAG. This is because 27 in binary is 00011011, which breaks down as:

How you pull that data out of the byte will depend on the language and what it makes available for bit-twiddling; those that don't have some form of bit field will probably provide the ability to bit shift and do a bitwise and (it's also likely that doing bitwise operations is better than using bit fields anyway). In the version I wrote in Emacs Lisp, it's simply a case of shifting the two bits I am interested in over to the right of the byte and then performing a bitwise and to get just its value. So, given an array called 2bit-bases whose content is this:

(defconst 2bit-bases ["T" "C" "A" "G"]
  "Vector of the bases.

Note that the positions of each base in the vector map to the 2bit decoding
for them.")

I use this bit of code to pull out the individual bases:

(aref 2bit-bases (logand (ash byte (- shift)) #b11))

Given code to unpack an individual byte, extracting all of the bases in a sequence then becomes the act of having two loops, the outer loop being over each byte in the file, the inner loop being over the positions within each individual byte.

In pseudo-code, assuming that start and end hold the base locations we're interested in and dna_pos is the location in the file where the DNA starts, the main loop for unpacking the data looks something like this:

# The bases.
bases = [ "T", "C", "A", "G" ]

# Calculate the first and last byte to pull data from.
start_byte = dna_pos + floor( start / 4 )
end_byte   = dna_pos + floor( ( end - 1 ) / 4 )

# Work out the starting position.
position = ( start_byte - dna_pos ) * 4

# Load up the bytes that contain the DNA.
buffer = read_n_bytes_from( start_byte, ( end_byte - start_byte ) + 1 )

# Get all the N blocks that intersect this sub-sequence.
n_blocks = relevant_n_blocks( start, end )

# Get all the mask blocks that interest this sub-sequence.
mask_blocks = relevant_mask_blocks( start, end )

# Start with an empty sequence.
sequence = ""

# Loop over every byte in the buffer.
for byte in buffer

  # Stepping down each pair of bits in the byte.
  for shift from 6 downto 0 by 2

    # If we're interested in this location.
    if ( position >= start ) and ( position < end )

      # If this position is in an N block, just collect an N.
      if within( position, n_blocks )
        sequence = sequence + "N"
      else

        # Not a N, so we should decode the base.
        base = bases[ ( byte >> shift ) & 0b11 ]

        # If we're in a mask block, go lower case.
        if within( position, mask_blocks )
          sequence = sequence + lower( base )
        else
          sequence = sequence + base
        end

      end

    end

    # Move along.
    position = position + 1

  end

end

Note that some of the detail is left out in the above, especially the business of loading up the relevant blocks; how that would be done will depend on language and the approach to writing the code. The Emacs Lisp code I've written has what I think is a fairly straightforward approach to it. There's a similar approach in the Common Lisp code I've written.

And that's pretty much it. There are a few other details that differ depending on how this is approached, the language used, and other considerations; one body of 2bit reader code that I've written attempts to optimise how it does things as much as possible because it's capable of reading the data locally or via ranged HTTP GETs from a web server; the Common Lisp version I wrote still needs some work because I was having fun getting back into Common Lisp; the Emacs Lisp version needs to try and keep data as small as possible because it's working with buffers, not direct file access.

Having got to know the format of 2bit files a fair bit, I'm adding this to my list of "fun to do a version of" problems when getting to know a new language, or even dabbling in a language I know.

As mentioned in the first post, once you've read in the header data for a 2bit file, the next step is to read the index. This is an index into all the different sequences held in the file. Reading the index itself is fairly straightforward.

The index comes right after the header -- so it starts on the 17th byte of the file. Each entry in the index contains three items of information:

So, in some sort of pseudo-code, you'd read in the index as follows:

index = dict()
for seq = 1 to seq_count // seq_count comes from the header
  name_len = (int) read_bytes( 1 )
  name     = (string) read_bytes( name_len )
  offset   = (int) read_bytes( 4 )
  index[ name ] = offset
end

Note, as mentioned in the first post, the index will need to be byte-swapped if the file is in an endian form other than the machine you're running your code on. How you'd go about this will, of course, vary from language to language, but the main idea is always going to be the same.

There's a fairly striking downside to this approach though: reading data can often be an expensive (in terms of time) operation -- this is especially true if the data is coming in from a remote machine, perhaps even one that's being accessed over the Internet. As such, it's best if you can make as few "trips" to the file as possible.

With this in mind, the best thing to do is to read the whole index into memory in one go and then process it from there -- the idea being that that's just one trip to the data source. The problem here, however, is that there's nothing in the header or the index that tells you how large the index actually is. What you can do though is work on the worst case scenario (assuming memory will allow). The worst case is fairly easy to handle: it's going to be 1 byte for the name length, plus 255 bytes for the name (the longest possible name), plus 4 bytes for the offset; multiply all that by the number of sequences in the index and you have the worst-case buffer size.

When reading this data in you might also want to ensure you're not going to run off the end of the file (perhaps the names are all quite small and so are the sequences).

Recently I've been working on a package for Emacs that can read data from 2bit files, so here's the core code for reading in the index:

(defun 2bit--read-index (source)
  "Read the sequence index from SOURCE.

As a side effect `2bit-data-pos' of SOURCE will move."
  (cl-loop
   ;; The index will be a hash of sequence names, with the values being the
   ;; offsets within the file.
   with index = (make-hash-table :test #'equal)
   ;; We could read each name/value pair one by one, but because we're doing
   ;; this within Emacs, which means making a temp buffer for every read,
   ;; that could get pretty expensive pretty fast. So instead we'll read the
   ;; index data in in one go. However, there is no easy-to-calculate size
   ;; for the index. The best we can do is calculate the worst case size. So
   ;; let's do that. The worst case size is the maximum size of the name of
   ;; a sequence (255), plus the size of the byte that tells us the name
   ;; (1), plus the size of the word that is the offset in the file (4).
   with buffer = (2bit--read source (* (2bit-data-sequence-count source) (+ 255 1 4)))
   ;; For every sequence in the file...
   for n from 1 to (2bit-data-sequence-count source)
   ;; Calculate the position within the buffer for this loop around. Note
   ;; that the skip is the last position plus 1 for the size byte plus the
   ;; size plus the length of the offset word.
   for pos = 0 then (+ pos 1 size 4)
   ;; Get the length of the name of the sequence.
   for size = (aref buffer pos)
   ;; Pull out the name itself.
   for name = (substring buffer (1+ pos) (+ pos 1 size))
   ;; Pull out the offset.
   for offset = (2bit--word-from-bytes source (substring buffer (+ pos 1 size) (+ pos 1 size 4)))
   ;; Collect the offset into the hash.
   do (setf (gethash name index) offset)
   ;; Once we're all done.... return the index.
   finally return index))

This code does what I mention above: it grabs enough data into a buffer in one go that I'll have the whole index in memory to pull apart, and then I work with the in-memory copy. The index is added to a hashing dictionary. Note that, in this case, I don't actually do the test for running off the end of the file because at the heart of the file reading code is insert-file-contents-literally and it doesn't error if you request too much.

With that done you'll have a list of all the sequences in the file. The next part, which will come in the next post, is the properly tricky part: the decoding of the sequence data itself.

Introduction

I've written a bit before about the value of having simple but interesting "problems", that you know the solution to, as a way of exercising yourself in a new environment. Recently I've added another to the list I already have, and I used it as an excuse to get back into writing some Common Lisp; and then went on to use it as a reason to write yet another package for Emacs.

Having gone through the process of writing code to handle 2bit files twice in about a month, and in two very similar but slightly different languages, I thought it might be interesting for me to then use it to exercise my ability to write blog posts (something I always struggle with -- I find writing very hard on a number of levels) and especially posts that explain a particular problem and how I implemented code relating to that problem.

Also, because the initial version of this post rambled on a bit too much and I lost the ability to finish it, I'm starting again and will be breaking it up a number of posts spanning a number of days -- perhaps even weeks -- so that I don't feel too overwhelmed by the process of writing it. I will, of course, make sure every post links to the other posts.

Now, before I go on, I'll make the important point that everything here is written from the perspective of a software developer who happens to work as part of a bioinformatics team; I don't do bioinformatics, I don't claim to understand it, I just happen to sit with (well, used to sit with them -- hopefully we'll all make it back to the office one day!) and work with them, and develop software that supports their work. Anything you see in any of the posts that's wrong about that subject: that's just my ignorance being shown through the lens of a software developer (all corrections are welcome).

So, with all those disclaimers aside, I'm going to go on a slow wander through what a 2bit file is, how you'd go about reading it, and related issues. This isn't designed as a tutorial or anything like that, this is simply me taking what I've learnt and writing it down. Perhaps someone else will benefit one day, but the purpose is to simply enjoy cementing it in my own mind and to enjoy the process of putting it all in writing.

What is a "2bit file"?

So what's this new "problem" I've added to my list? It's code to read sequence data from 2bit format files. For anyone who doesn't know (bioinformatics people look away now; a software developer is going to explain one of your file formats), this is a file format that is intended to hold sequences in an efficient way. As I'm sure you know, DNA is made up of 4 bases, represented by the letters T, C, A and G. So, in the simplest case, we could just represent a genome using those four letters. Simple enough, right? Nice big text file with just those 4 letters in?

The thing is, something like the human genome is around 3 billion bases in length. That'd make for a petty big file to have to store and move around. So why not compress it down a bit? That's where the 2bit format comes in.

Given this problem I'm sure most developers would quickly notice that, given 4 different characters, you only need 2 bits to actually hold them all (two bits gets us 00, 01, 10 and 11, so four different states). This means with a little bit of coding you can store 4 bases in a single byte. Just like that you've pretty much squished the whole thing down to 1/4 of the original size. And that's more or less what the 2bit format does. If you take a look at the actual data for the human genome you'll see that hg38.2bit is roughly 1/4 of 3 billion bytes, ish, give or take.

There is a wrinkle, however. There are parts of a genome where you might not know what base is there. Generally an N is used for that. So, actually, we want to be able to store 5 different characters. But 5 isn't going to go into 2 bits... Damn! Well, it's okay, 2bit has a solution to that too, and I'll cover that later on.

How is a 2bit file formatted?

As you can see from the format information available online, a 2bit file is a binary file format that is split into 3 key parts:

• A fixed size header with some key information
• An index into the rest of the file
• A series of records that contain actual sequence information

In this first post I'll cover the details of the header. Subsequent posts will cover the index and the actual sequence data records.

The header¶

The header of a 2bit file is fixed in size and contains some key information. It can be broken down as follows:

The signature value is used to test if what you're looking at is a 2bit file, but also tells you some vital information about how to read the file -- see below for more on that. The version value is always 0 -- as such another useful test would be to error out if you get a valid signature but get a version other than 0. The sequence count is, as you'd guess, the number of sequences that are held within the file -- this is important when loading in the index of the file (more on that in the next post).

The signature, big and little endianness, and byte swapping¶

The header mentioned above comprises of 4 32-bit word values. The very first word is important to how you read the rest of the file. This is the signature for the 2bit file and it should always be 0x1A412743. And this is where it gets interesting and fun right away. The 2bit file format allows for the fact that the file can be built in either a little-endian or a big-endian machine, and the 32-bit word values can be binary-written to the file in the local architecture's byte order. The effect of this is that, from reading the very first value in the file, you need to decide if every other numeric value you read needs to be byte-swapped in some way. The early logic being (in no particular language) something like:

if signature == 0x1A412743 then
  must_swap = False
else if byte_swap( signature ) == 0x1A412743 then
  must_swap = True
else
  raise "This isn't a valid 2bit file"
end

Simply put, to read the rest of the file you will need a function that byte-swaps a 32bit numeric value, and a flag of some sort to mark that you need to do this every time you read such a value. Of course, depending on your language of choice, you could do it in a number of different ways. In a language like JavaScript or Scheme, where you can easily throw around functions, I'd probably just assign the appropriate 32bit-word-reading function to a global function name and call that regardless throughout the rest of the code. In other languages I'd probably just check the flag each time and call the swapping function if needed. In something like Python I'd likely just use the signature to decide on which format to pass to struct.unpack. For example, some variation on:

# Assuming that 'header' is the whole header of the file read as a binary buffer.

word_fmt = ""

for test_fmt in ( "<I", ">I" ):
    if struct.unpack( test_fmt, header[ 0:4 ] )[ 0 ] == 0x1A412743
        word_fmt = test_fmt
        break

if not word_fmt:
    raise Exception( "This isn't a 2bit file" )

Now, the Python approach sort of hides the important detail here. With it we'd simply use struct.unpack's ability to handle different byte orders and not worry about the detail. Which isn't fun, right? So how might code to byte-swap a 32bit value look?

Assuming you've got the value as an actual numeric integer, it can be as simple as using a bit of bitwise anding and shifting. Here's the basic code I wrote in Common Lisp, for example:

(defun swap-long (value)
  "Swap the endianness of a long integer VALUE."
  (logior
   (logand (ash value -24) #xff)
   (logand (ash value -8) #xff00)
   (logand (ash value 8) #xff0000)
   (logand (ash value 24) #xff000000)))

JavaScript might be something like:

function swapLong( value ) {
    return ( ( value >> 24 ) & 0xff       ) |
           ( ( value >>  8 ) & 0xff00     ) |
           ( ( value <<  8 ) & 0xff0000   ) |
           ( ( value << 24 ) & 0xff000000 )
}

and other variations on that theme in different languages.

Up next¶

In the next post I'll write about how the sequence index is stored and how to load it, including some considerations about how to make the loading as efficient as possible.

Earlier this week something in my development environment, relating to Homebrew, Python, pyenv and pipenv, got updated and broke a handful of repositories. Not in a way that I couldn't recover from, just in a way that was annoying, got in the way of my workflow, and needed attention. (note to self: how I set up for Python/Django development on a machine might be a good post in the future)

Once I was sure what the fix was (pretty much: nuke the virtual environment and recreate it with pipenv, being very explicit about the version of Python to use) the next step was to figure out how many repositories were affected; not all were and there wasn't an obvious pattern to it. What was obvious was that the problem came down to python in the .venv directory pointing to a binary that didn't exist any more.

So... tracking down problematic repositories would be simple enough, just look for every instance of .venv/bin/python and be sure it points to something rather than nothing; if it points to nothing I need to remake the virtual environment.

I quickly knocked up a script that was based around looking over the results of a find, and initially decided to use file to perform the test on python. It seemed to make sense, as I wrote the script I checked the man page for file(1) on macOS and sure enough, this exists:

-E On filesystem errors (file not found etc), instead of handling the error as regular output as POSIX mandates and keep going, issue an error message and exit.

Given that file dereferences links by default, that should get me an error for a broken link, right? Bit hacky I guess, but it was the first thing that came to mind for a quick bit of scripting and would do the trick. Only...

$ file -E does-not-exist
file: invalid option -- E
Usage: file [bcCdEhikLlNnprsvzZ0] [-e test] [-f namefile] [-F separator] [-m magicfiles] [-M magicfiles] file...
       file -C -m magicfiles
Try `file --help' for more information.

Wat?!? But it's right there! It says so in the manual! -E is documented right in the manual page! And yet it's not in the valid switch list as put out by the command, and it's an invalid option. The hell?

So I go back and look at the man page again and then I notice it isn't in the list of switches in the synopsis.

SYNOPSIS
file [-bcdDhiIkLnNprsvz] [--extension] [--mime-encoding] [--mime-type] [-f namefile] [-m magicfiles] [-P name=value] [-M magicfiles] file
file -C [-m magicfiles]
file [--help]

I then did the obvious tests. Did I have file aliased in some way? No. Was some other thing that works and acts like file in my path? No. Was I absolutely 100% using /usr/bin/file? Yes.

Long story short: it seems the man page for file, on macOS, fibs about what switches it supports; it says that -E is a valid option, but it's not there.

What's even odder is the man page says it documents v5.04 of the command, but --version reports v5.37. Meanwhile, if I check on a GNU/Linux box I have access to, it does support -E, reports it in the switches, documents it in the man page (in both the synopsis and in the main body of the page) and it is v5.25 (and so is its man page).

So that was something like 20 minutes lost to a very small problem, for which there was no real solution, but was time that had to be spent to get to the bottom of it.

In the end I went with what I probably should have gone with in the first place: stat -L.

for venv in $(find . -name .venv)
do
    if ! stat -L "$venv/bin/python" > /dev/null 2>&1
    then
        echo "$(dirname $venv)"
    fi
done

And now I have that script in my ~/bin directory, ready for the next time Homebrew and friends conspire to throw my day off for a while.

This is something I've being meaning to do for a couple or so years now, and unsurprisingly it's bubbled up again recently: the business of swapping the name of the master branch in git out for something better.

Because it's one of those jobs that's simultaneously simple and also laborious, I kept putting it off. Changing up the local configuration so that main (or whatever name you prefer) is used "out of the box" is simple enough; the laborious part is updating all of the repositories that live in the "forge of choice". In my case, over on GitHub, I have getting on for 200 repositories -- 142 of which are public (as of the time of writing). At work we use GitLab as our internal forge and I've got a non-trivial number of repositories on there too.

The obvious first step to tackling this is to knock up a little tool to help find the repos that still need swapping. That was simple enough:

#!/bin/bash

# Quick and dirty tool to find repositories that still make use of a
# "master" branch. Helps with tracking down the ones that need
# updating/improving.

for repo in $(find . -name .git)
do
    (
        cd "$(dirname $repo)"

        if git branch | grep master > /dev/null 2>&1
        then
            echo "$(dirname $repo)"
        fi
    )
done

### git-archaic ends here

It's not meant to be clever, just something I can run when I'm in a "default branch swapping" mood so find a repository or two to tackle. The idea being that it uses find to pull out any instance of .git in or below the current directory, changes to it (inside a sub-process to ensure the PWD gets put back after the cd that happens, before the next iteration of the loop), gets a list of the branches and, if master is one of them, prints the directory name.

Using this, I can now slowly work through my more active repositories and make the swap -- the idea being that if I currently have them cloned down to my machine, they're obviously some level of "active". At some point I imagine I could get more clever and use the APIs of the forges to look at all the repositories I own; that's another job for another day.

This gives me enough to be going on with. :-)