Python posts by Daniel Roy Greenfeld

TIL: Setting environment variables for pytest

2025-09-02T02:29:33.084710+00:00

When writing tests in pytest, often there's a need to set environment variables for your tests. Instead of modifying os.environ directly, which can lead to side effects and make tests harder to manage, here's how to do it with the pytest-env package.

First, install the package.

pip install pytest-env # classic but works great
uv add pytest-env # if you're one of us cool kids using uv
uv add pytest-env --group test # if you use a specific test group of dependencies

A lot of people use pytest.ini to configure pytest, but I prefer using pyproject.toml for a more modern approach. Here's how you can set environment variables in pyproject.toml:

[tool.pytest_env]
# Test value for the SUPER_SECRET_KEY environment variable
SUPER_SECRET_KEY = "ABC123"

Then run your tests just like you would normally do, and now the environment variable will be picked up by whatever test or code is looking for it. That's correct, you don't need to do any further configuration.

pytest .

Since a lot of API SDK libraries use environment variables for configuration, this is a great way to ensure your tests run in a controlled environment. Either with mock replies or against a test instance of the service.

Over Twenty Years of Blogging Tools

2025-09-24T00:45:00.884329+00:00

On my articles page, you can read near the top that I've been writing for the past 20 years (plus a little more). It's not all my online public writing, but it's a majority of it. The primary reason parts are missing is that over time, I've used a variety of tools to publish my thoughts. Some of those tools were hosted services, and material that was lost was because of services being shut down. Hence, why I prefer markdown files in a git repository over a database solution.

Here's the story of my writing tools over the years.

Geocities 1997-2010

In 1997, I first remember writing something for general publication on the web. Until then, my writings had been transmitted by printing paper, hand-copying onto disks, sending by email, or posting to usenet. I think it was in 1997 that I made my first essay-style writing on GeoCities site I put together. Geocities is gone, but periodically I've thought of digging through GeoCities archives to find my writing from that time between 1997 and 2004. Otherwise, those articles are gone.

Even though there's nothing from that time on this site, as Geocities overlaps with the first entry on this site, I've included it for the sake of posterity.

Livejournal 2004-2009

From 2004 to 2009, I used LiveJournal to put down my thoughts, and there I wrote on a regular basis about life, activities, fitness, coding, and general work stuff. When I revisit those posts, I see some fun things, some hard challenges, and between the lines a lot carefully hidden despair. The few articles that made it onto this site capture extended family memories.

Some of the articles from LiveJournal exist here.

Blogger 2007-2012

In 2007, blogs had taken off, and it was clear that professionally having my own blog was important. I decided to keep things simple and use Blogger to host my first professional blog. I opened a few other blogger sites for other purposes, but the main one was PyDanny.

This was my formative years of Python (and Django). Everything was bright and shiny and new. I met Audrey and fell in love with her.

A few years ago, I migrated that content here on this page.

Pelican 2012-2018

In 2012, I started writing content on a Pelican static site. I liked not having to set up a database for content. What I didn't like was having to work through the abstraction of Pelican to do anything.

Mountain 2018-2019

Frustrated with the lack of control of Pelican in 2018, I wrote my own Flask-powered blog engine, which I called Mountain. The challenge there was that I had some crazy ideas for a feature that caused too much complexity. The result was a very slow rendering of the static site. It was too much trouble to fix, so I stuck with it.

Uma was born in early 2019. I was too busy being a new father to do much writing, only getting in 5 articles for the whole of 2019.

Vuepress 2019-2021

By 2019, I was in the Jamstack world. I've since left it, but for a while my site was on Vuepress. The speed of compilation was nice compared to Mountain, and I could stick in special pages as needed. However, there were a few too many layers of abstraction, and that made extending it to work outside the default behaviors of Vuepress too difficult.

Next.js 2021-2024

In early 2021, for work, I needed to learn Next.js quickly. I used the official Next.js tutorial that existed at the time to learn the framework and React over a weekend. This was still the early days of Next.js, and I found it to be tons of fun. It reminded me of the best features of Flask combined with tons of frontend power.

Next.js has changed rapidly. I believe Next.js went in an uncomfortable direction in terms of complexity and decreasing quality of documentation for new or changed features. Bitrot happened frequently, and I felt like I was fighting version upgrades rather than working with the Framework.

FastHTML 2024-2025

I tried out FastHTML in the summer of 2024. Keeping the features spec in my head, tiny, I coded up the kernel of a cached markdown content site in 45 minutes. This was a vast improvement in terms of speed and complexity over the unnecessarily complex Mountain project of 2018. It was nice to return to Python, but it chafed to use a framework that wasn't PEP8 nor fully type-annotated.

Air 2025+

For a few years now, I've wanted to create my own web framework. This year I did so, launching AIR with my wife and coding partner Audrey. Air is a shallow layer over FastAPI, adding features to expedite authoring of dynamic HTML pages. One of the early projects I did with it was to convert this site to use Air. It was nice to return to a PEP8-formatted codebase that is fully type-annotated. I also like the smaller dependency tree.

The Future

Perhaps I'll grow restless again in a few years and try something new. Or perhaps I'll stick with Air for a long time. What I do know is that I enjoy writing and sharing my thoughts. So there will be posts going into the future.

TIL: Loading .env files with uv run

2025-09-28T22:44:18.296914+00:00

We don't need python-dotenv, use uv run with --env-file, and your env vars from .env get loaded.

For example, if we've got a FastAPI or Air project we can run it locally with env vars like:

uv run --env-file .env fastapi dev

You can specific different env files for different environments, like .env.dev, .env.prod, etc.

uv run --env-file .env.dev fastapi dev

All credit goes to Audrey Roy Greenfeld for pointing this out.

Using pyinstrument to profile Air apps

2025-10-05T08:00:13.982725+00:00

Air is built on FastAPI, so we could use pyinstrument's instructions modified. However, because profilers reveal a LOT of internal data, in our example we actively use an environment variable.

You will need both air and pyinstrument to get this working:

# preferred
uv add "air[standard]" pyinstrument
# old school
pip install "air[standard]" pyinstrument

And here's how to use pyinstrument to find bottlenecks:

import asyncio
from os import getenv
import air
from pyinstrument import Profiler

app = air.Air()

# Use an environment variable to control if we are profiling
# This is a value that should never be set in production
if getenv("PROFILING"):
    @app.middleware("http")
    async def profile_request(request: air.Request, call_next):
        profiling = request.query_params.get("profile", False)
        if profiling:
            profiler = Profiler()
            profiler.start()
            await call_next(request)
            profiler.stop()
            return air.responses.HTMLResponse(profiler.output_html())
        else:
            return await call_next(request)


@app.page
async def index(pause: float = 0):
    if pause:
        await asyncio.sleep(pause)

    title = f"Pausing for {pause} seconds"
    return air.layouts.mvpcss(
        air.Title(title),
        air.H1(title),
        # Provide three options for testing the profiler
        air.P('Using asyncio.sleep to simulate bottlenecks'),
        air.Ol(
            air.Li(
                air.A(
                    f"Pause for 0.1 seconds",
                    href="/?profile=1&pause=0.1",
                    target="_blank",
                )
            ),
            air.Li(
                air.A(
                    f"Pause for 0.3 seconds",
                    href="/?profile=1&pause=0.3",
                    target="_blank",
                )
            ),
            air.Li(
                air.A(
                    f"Pause for 1.0 seconds",
                    href="/?profile=1&pause=1.0",
                    target="_blank",
                )
            ),
        ),
    )

Running the test app:

Rather than set the environment variable, for this kind of thing I like to prefix the CLI command with a PROFILING=1 prefix to send the environment variable for just this run of the project. By doing so we trigger pyinstrument:

PROFILING=1 fastapi dev main.py

Once you have it running, check it out here: http://localhost:8000

Screenshots

Using Asyncpg with FastAPI and Air

2025-10-19T05:07:02.438338+00:00

Recently I've been on a few projects using PostgreSQL where SQLAlchemy and SQLModel felt like overkill. Instead of using those libraries I leaned on writing SQL queries and running those directly in asyncpg instead of using an ORM powered by asyncpg.

Here's how I got it to work

Defined a lifespan function for ASGIApp

Starlette ASGIApp frameworks like FastAPI (and by extension Air) can leverage lifespan functions, which are generators. I've commented the lifespan object for clarity.

from contextlib import asynccontextmanager
from os import environ
from typing import AsyncIterator

import asyncpg
from starlette.types import ASGIApp

DATABASE_URL = environ['DATABASE_URL']


@asynccontextmanager 
async def lifespan(app: ASGIApp) -> AsyncIterator[None]:
    """A lifespan for maintaining the connection to the PostgreSQL DB
        Without this, the connection will timeout and queries will fail.
    """
    # app.state is where the connection pool is created, which can
    # be accessed later inside of views. The is only run once during
    # app startup.
    app.state.pool = await asyncpg.create_pool(
        dsn=DATABASE_URL,
        min_size=1,
        max_size=10,
    )
    try:
        # This is where the app runs all the URL route functons.
        yield
    finally:
        # This is run once when the app is shut down.
        await app.state.pool.close()

Using the lifespan function

Just add the lifespan function to the app when it is instantiated.

Using the lifespan function for FastAPI projects

All you have to do is pass the lifespan callable to the FastAPI app instantiation.

from fastapi import FastAPI

# Adding the lifespan app
app = FastAPI(lifespan=lifespan) 

@app.get('/users')
async def users(): # every function must be async
    # Use the pool object to get the database connection object
    async with app.state.pool.acquire() as conn:
        results = await conn.fetch('SELECT * from users;')

    # FastAPI responses automatically convert dicts to JSON
    return {'count': len(results), 'users': results}

Using the lifespan function for Air projects

Air is powered by FastAPI (and Starlette), so uses this lifespan function the same way as FastAPI.

import air

# Adding the lifespan app
app = air.Air(lifespan=lifespan)


@app.get('/users')
async def users(): # every function must be async
    # Use the pool object to get the database connection object
    async with app.state.pool.acquire() as conn:
        users = await conn.fetch('SELECT * from users;')

    # Air tags are converted to HTML during the response stage
    # Jinja is also an option, but is outside the scope of this article
    return air.layouts.mvpcss(
        air.H1(f'Users: {len(users)}'),
        air.Ul(
            *[air.Li(u['email']) for u in users]
        )
    )

Incoming data

Changing data requires use of the conn.execute function. Of course these examples will show how to use pydantic to validate the incoming data before we allow it to touch our database.

Adding data with FastAPI via asyncpg

As part of the request process for REST API, FastAPI uses pydantic to validate incoming data. This results a delightfully small view for accepting data.

from fastapi import FastAPI
from pydantic import BaseModel, EmailStr

# Adding the lifespan app
app = FastAPI(lifespan=lifespan) 


class User(BaseModel):
    email: EmailStr


@app.post('/users')
async def users_add(user: User):
    # Get the conn object from the database connection pool
    async with app.state.pool.acquire() as conn:
        # Insert the record with an execute method
        await conn.execute(
            'INSERT INTO users (email, created_at) VALUES ($1, NOW())',
            user.email
        )

    return user

Adding data with Air via asyncpg

There's no consistent standard within HTML for how to construct a form, much less respond to a bad implementation. Therefore in order to handle incoming data Air needs a bit more code than FastAPI.

import air
from pydantic import BaseModel, EmailStr

# Adding the lifespan app
app = air.Air(lifespan=lifespan) 


class User(BaseModel):
    email: EmailStr


class UserForm(air.AirForm):
    model = User    


@app.post('/users')
async def users_add(request: air.Request):
    # AirForms make handling incoming forms easier
    form = await UserForm.from_request(request)

    # AirForms, once instantiated with data, have an `is_valid` property
    # which returns a boolean of whether or not the submitted data has
    # passed pydantic.
    if form.is_valid:
        # Get the conn object from the database connection pool
        async with app.state.pool.acquire() as conn:
            # Insert the record with an execute method
            await conn.execute(
                'INSERT INTO users (email, created_at) VALUES ($1, NOW())',
                form.data.email
            )
        return air.layouts.mvpcss(
            air.H1(f"User: {form.data.email}"),
        )        

    # Simplistic handling of bad signup. 
    return air.RedirectResponse('/signup')

AirForms supports reporting of bad data. I'll cover how to do that in follow-up article.

uv+just for testing multiple Python versions

2025-10-22T13:00:56.868382+00:00

For years I used tox and nox to test my Python projects against multiple Python versions on multiple operating systems. While both tools are powerful, they require configuration complex enough that I usually just copy/paste it from project to project. In comparison, uv+just isn't just a completely workable replacement, it's simple enough that I can either memorize the commands or if I forget can look up in moments.

An improvement on this post, which just covers the uv side.

What is Just?

Just is a rust-powered command runner inspired by Make but with a slightly simpler syntax. Where Make was originally designed to build executable code, just focuses more on being a command-runner. Also, Just benefits from the lessons learned by Make. The result is an easier-to-use tool that is also much easier to get running on Windows than Make. The way Just works is you define commands in a Justfile that can be easily executed from the command line. It can be installed from pypi as the rust-just package.

Assumptions about our project

Uses pytest for testing
Has a tests/ directory containing the test cases
Supposed to work with both Python 3.13 and 3.14

Installing dependencies using pyproject.toml

Make sure to include the necessary dependencies in your pyproject.toml file:

[dependency-groups]
dev = [
    "rust-just"
]

test = [
    "pytest",
]

Once in your pyproject.toml get your development dependencies ready with this command:

uv sync --group dev

Building the Justfile

Justfiles are simple to write. Create a file named Justfile in the root of your project with the following content:

# Run all the tests against multiple Python versions
test:
    uv run --python 3.13 --group test pytest tests/
    uv run --python 3.14 --group test pytest tests/

If you have more Python versions to test against, simply add more uv run lines with the desired version.

Running the tests

just test

And that's it! You now have a simple and effective way to test your Python projects against multiple Python versions using uv and just, without the overhead of more complex tools like tox or nox. This setup is easy to maintain and adapt as your project evolves.

Just bonus feature: command lookup

A nice feature about just is you can trivially look up the commands you've authored with just -l, and leading docstrings are included in the display:

$ just -l
Available recipes:
    test # Run all the tests against multiple Python versions

If you don't specify a command, just will run the first one it finds in the file. This means you can set up a default command to run when you just type just with no arguments. What I like to do is put a list action as the first command. By that I mean something like this:

# List all the commands in this file
list:
    just -l

# Run all the tests against multiple Python versions
test:
    uv run --python 3.13 --group test pytest tests/
    uv run --python 3.14 --group test pytest tests/

Visiting Tokyo, Japan from November 12 to 24

2025-11-11T14:45:22.984908+00:00

I'm excited to announce that me and Audrey will be visiting Japan from November 12 to November 24, 2025! This will be our first time in Japan, and we can't wait to explore Tokyo. Yes, we'll be in Tokyo for most of it, near the Shinjuku area, working from coffee shops, meeting some colleagues, and exploring the city during our free time. Our six year old daughter is with us, so our explorations will be family-friendly.

Unfortunately, we'll be between Python meetups in the Tokyo area. However, if you are in Toyo and write software in any shape or form, and would like to get together for coffee or a meal, please let me know!

If you do Brazilian Jiu-Jitsu in Tokyo, please let me know as well! I'd love to drop by a gym while I'm there.

TIL: Default code block languages for mkdocs

2025-11-22T12:08:34.618632+00:00

When using Mkdocs with Material, you can set default languages for code blocks in your mkdocs.yml configuration file. This is particularly useful for inline code examples that may not have explicit language tags.

markdown_extensions:
  - pymdownx.highlight:
      default_lang: python

You can see what this looks like in practice with Air's API reference for forms here: feldroy.github.io/air/api/forms/. With this configuration, any code block without a specified language defaults to Python syntax highlighting, making documentation clearer and more consistent.

Adding Type Hints to my Blog

2025-12-11T08:25:02.790731+00:00

I've decided to add static type checking to my blog engine project. The tool I chose is pyrefly, a fast, Rust-based library for checking types in Python.

Installing Pyrefly with UV

My project uses uv for package management. To install pyrefly as a development-only dependency, I ran the following command:

uv add pyrefly --dev

pyrefly is a Rust-based Python tool, so its package includes pre-compiled binaries. This makes the package larger (around 10MB) than a pure Python equivalent. This can be an issue with a slower connection. However, uv caches the downloaded package, making subsequent installations of the same version much faster.

Running the First Type Check

With pyrefly installed, I ran the first check across the entire project.

uv run pyrefly check .

The initial scan found 31 errors. To make the task more manageable, I narrowed the scope to just the main application file.

uv run pyrefly check main.py

This reduced the list to 11 errors, giving me a focused starting point.

Debugging a Type Error

I decided to tackle one of the reported errors. pyrefly pointed out an issue with the get_post function. Here's the pyrefly output

ERROR Type `None` is not iterable [not-iterable]
   --> main.py:258:9
    |
258 |         content, metadata = get_post(slug)
    |         ^^^^^^^^^^^^^^^^^
    |

The function's type hint declared that it returns a tuple or None.

# The incorrect type hint
def get_post(...) -> tuple | None:
    # ... function implementation

However, after reviewing the code, I saw that the function never actually returns None. If a post is not found, it raises a ContentNotFound exception. The type hint was wrong.

def get_post(slug: str) -> tuple | None:
    posts = list_posts(content=True)
    post = next((x for x in posts if x["slug"] == slug), None)
    if post is None:
        raise ContentNotFound
    return (post["content"], post)

Verifying the Fix

I corrected the type hint by removing the incorrect | None part.

# The corrected type hint
def get_post(...) -> tuple:
    # ... function implementation

After saving the change, I re-ran the check on main.py.

uv run pyrefly check main.py

The error count dropped from 11 to 10. The fix was successful. You can see the commit where the work was done on the repo.

Conclusion

Adding a type checker like pyrefly immediately exposed incorrect type hints in the codebase. The process of installing the tool, running a check, and fixing the first error was straightforward. This small change improved the code's correctness and demonstrated the value of static analysis for maintaining a healthy project.

As for the rest of the errors, rather than attack them in one big effort as this is a stable side project what I like to do is make it a daily chore to do a single correction per day. This is slower (and could be done quickly with an LLM assist) but through practice I get better with the tool. Mastery is found through repetition.

Writing tools to download everything

2026-01-16T11:22:35.582258+00:00

Over the years, Audrey and I have accumulated photos across a variety of services. Flickr, SmugMug, and others all have chunks of our memories sitting on their servers. Some of these services we haven't touched in years, others we pay for but rarely use. It was time to bring everything home.

Why Bother?

Two reasons pushed me to finally tackle this.

First, money. Subscriptions add up. Paying for storage on services we barely use felt wasteful. As a backup even more so because there are services that are cheaper and easier to use for that purpose, like Backblaze.

Second, simplicity. Having photos scattered across multiple services means hunting through different interfaces when looking for a specific memory. Consolidating everything into one place makes our photo library actually usable.

Using Claude to Write a Downloader

I decided to start with SmugMug since that had the largest collection. I could have written this script myself. I've done plenty of API work over the years. But I'm busy, and this felt like a perfect use case for AI assistance.

My approach was straightforward:

Wrote a specification for a Smugmug downloader. I linked to the docs for the service then told it to make a CLI for downloading things off that service. For the CLI I insist on typer but otherwise I didn't specify dependencies.
Told Claude to generate code based on the spec. I provided the specification and let Claude produce a working Python script.
Tested by running the scripts against real data. I started with small batches to verify the downloads worked correctly. Claude got everything right when iy came to downloads on the first go, which was impressive.
Adjust for volume. We had over 5,000 files on Smugmug. Downloading everything at once took longer than I expected. I asked Claude to track files so if the script was interrupted it could resume where it left off. Claude kept messing this up, and after the 5th or 6th attempt I gave up trying to use Claude to write this part.

I Wrote Some Code

I wrote a super simple image ID cache using a plaintext file for storage. It was simple, effective, and worked on the first go. Sometimes it's easier to just write the code yourself than try to get an AI to do it for you.

The SmugMug Downloader

The project is here at SmugMug downloader. It authenticates, enumerates all albums, and downloads every photo while preserving the album structure. Nothing fancy, just practical.

I'll be working on the Flickr downloader soon, following the same pattern. There's a few other services on the list too; I'm scanning our bank statements to see what else we have accounts on that we've let linger for too long.

Was It Worth It?

Absolutely. What would have taken me a day of focused coding took an hour of iterating with Claude. Our photos are off Smugmug and we're canceling a subscription we no longer need. I think this is what they mean by "vibe engineering".

Summary

These are files which in some cases we thought we lost. Or had forgotten. So the emotional and financial investment in a vibe engineered effort was low. If this were something that was touching our finances or wedding/baby photos I would have been much more cautious. But for now, this is a fun experiment in using AI to handle the mundane parts of coding so I can focus on more critical tasks.

We moved to Manila!

2026-02-03T06:41:22.863302+00:00

Last year we relocated to Metro Manila, Philippines for the foreseeable future. Audrey's mother is from here, and we wanted our daughter Uma to have the opportunity to spend time with her extended family and experience another line of her heritage.

Where are you living?

In Makati, a city that contains one of the major business districts in Metro Manila. Specifically we're in Salcedo village, a neighboorhood in the CBD, made of towering residential and business buildings with numerous shops, markets, and a few parks. This area allows for a walkable life, which is important to us coming from London.

What about the USA?

The USA is our homeland and we're US citizens. We still have family and friends there. We're hoping to visit the US at least once a year.

What about the UK?

We loved living in London, and have many good friends there. I really enjoyed working for Kraken Tech, but my time came to an end there so our visas were no longer valid. We hope to visit the UK (and the rest of Europe) as tourists, but without the family connection it's harder to justify than trips to the homeland.

What about your daughter?

Uma loves Manila and is in second grade at an international school here in walking distance of our residence. We had looked into getting her into a local public school with a notable science program, but the paperwork required too much lead time. We do like the small class sizes at her current school, and how they accomodate the different learning speeds of students. She will probably stay there for a while.

For extra curricular activities she's enjoying Brazilian Jiu-Jitsu, climbing, yoga, and swimming.

If I'm in Manila can I meet up with you?

Sure! Some options:

We're long-time members of the Python Philippines community, so you can often find us at their events
If you train in BJJ, I'm usually at Open Mat Makati quite a bit. Just let me know ahead of time so I can plan around it
If you want to meet up for coffee, hit me up on social media. Manila is awesome for coffee shops!

TIL: Using PygmentsRenderer with mistletoe as a partial

2026-02-22T10:09:16.103693+00:00

For the past 18 months or so on this site, I've been using marked.js for the web and python-markdown for the atom feed. I decided not long ago to switch to using mistletoe so there's one consistent source of truth for markdown rendering.

I also thought that instead of defining a reusable function that both the web and atom feed could use, I would just use Python's functools.partial to create a new function that has the renderer argument set to PygmentsRenderer. Normally I prefer fully defined functions over using partial, but in this case, since it is for my personal site it felt like a good use of partial.

Add the dependencies

uv add mistletoe pygments

Create the partial function

from functools import partial

import mistletoe
from mistletoe.contrib.pygments_renderer import PygmentsRenderer

markdown = partial(mistletoe.markdown, renderer=PygmentsRenderer)

Use the new function

html = markdown(markdown_string)

See it in action

Here's the commits where I implemented this change, at least on the web side: Using pygments for highlighting of code. Before I change the atom feed generation to use this, I'll make sure that it renders nicely on planetpython and other feed aggregators. I'll post that in a seperate TIL when I do figure out how to do that.

Top Terminal Tools

2026-03-05T07:30:50.970143+00:00

When I sit down to code, these are the tools I use at this time whenever I touch code. In alphabetical order:

Atuin

Atuin is a replacement for the default shell history. It saves your history to a local, encrypted SQLite database. Then it allows for blazing fast searches. You can sync your history across devices, and is has a lot of other features. I can't imagine using a terminal without Atuin.

For OSX users, I recommend installing following these instructions.

bat

A Rust-based cat replacement. It has syntax highlighting, line numbers, and a lot of other features that make it a great tool for quickly looking at files in the terminal.

Ghostty

Ghostty is a fast, feature-rich, and cross-platform terminal emulator that I believe works everywhere. Yes, TMUX (and competitors) are more configurable and have more features, but Ghostty just works out of the box. Ghostty eschews the arcane key combinations of its predecessors in favor of intuitive keybindings. A ghostty terminal can be split horizontally and vertically and copy/paste works as expected. It also doesn't appear to interfere with any other shell tool, something that annoyed me about TMUX.

Usually I have three vertical panels:

On the far left a panel for running general shell commands like git, uv, and whatever. Atuin is a must for me, giving me shell history on steriods so I don't have to remember commands anymore
In the middle I run a CLI-based text editor, these days usually pad
On the righthand side a panel for running LLM CLI tools, usually AMP, Codex, and sometimes Claude

GitHub CLI

Even though I use a fraction of the available commands, this is just a really useful tool for working on GitHub-hosted projects.

Pad

Pad is an easy-to-use terminal editor I created for those of us who aren't into vim. It's powered by Will McGugan's Textual. I did what I can to make pad work with typical VS Code keybindings.

Right now I'm not accepting general contributions to Pad. I am still recovering from a head injury and that unfortunately makes reviewing PRs challenging. As I get better I plan to put more features into the project and perhaps even accept contributions. For now it does what I need it to do and is a handy addition to my tool chain.

Various LLM CLI tools

I have no loyalty to any particular LLM provider. I use whatever tool gets the job done and for which I have free or affordable credits. For now I use:

Amp
ChatGPT Codex
Claude Code

For a while I was using gemini-cli, but it stopped working for me weeks ago. Rather than try to debug it I just used the competition.

To return a value or not return a value

2026-03-13T02:41:19.009972+00:00

I believe any function that changes a variable should return a variable. For example, I argue that Python's random.shuffle() is flawed. This is how random.shuffle() unfortunately works:

import random

my_list = [1, 2, 3, 4, 5]
print(f"Original list: {my_list}")

# Change happens in place
# my_list is forever changed
random.shuffle(my_list) 
print(f"Shuffled list: {my_list}")

In my opinion, random.shuffle() should work like this:

import random

my_list = [1, 2, 3, 4, 5]

# Function returns a new, shuffled list
new_list = random.shuffle(my_list)
print(f"Original list: {my_list}")
print(f"Shuffled list: {new_list}")

Of course, Python won't fix this mistake to fit my preference. There's too many places in the universe expecting random.shuffle to change a list in place. Yet it still bugs me every time I see the function. Stuff like this is why I created my listo package, it allowed me to get past my own sense of annoyance. The listo library is barely used, even by myself, serving mostly as a fun exercise that allowed me to scratch an itch about objects changing in place.

Counterargument

Some of you might say, "It's not practical to return giant dict or list objects when you are changing a single value". You are correct. However, does it make sense for random.shuffle and other offenders to muck around with the entirety of a variable's contents? Why shouldn't a function that disrupts the entirety of a variable just return a new variable?

Closing statement

My preference is that when it is reasonable, that the scope is not outrageous, to create functions that return values.

Also, to the people who implemented the original random.shuffle function, you are awesome. I'm just taking advantage of having 20/20 hindsight.

Word counter that ignores Markdown

2026-05-04T11:55:23.590783+00:00

I've been doing a lot of writing recently, and tracking my word count. I write in markdown. I could just render the text using a markdown library and then do a count on the generated output, but then I wouldn't have the fun of writing out a bunch of regular expressions. Yes, I know the cautionary meme by that says:

"Some people, when confronted with a problem, think ‘I know, I’ll use regular expressions.’ Now they have two problems." -- Jamie Zawinski

I don't care.

I love working in regular expressions. It was the one thing I got out of my brief foray in Perl at the very start of my software development career. I carried it into my Java and ColdFusion days and periodically use it in Python. Yes, Python has lots of useful string tools, but playing with regular expressions until they are just right remains a fun puzzle for me.

So here you go, a Python-powered word counter powered by my desire to noodle with regular expressions:

"""
word_count.py — Count words in a Markdown file or a directory of markdown files.

Dependencies:
    typer
    rich

Usage:
    python word_count.py README.md
    python word_count.py README.md --no-strip-markdown
    python word_count.py README.md --verbose
    python word_count.py book/
"""

import re
from pathlib import Path

import typer
from rich.console import Console
from rich.panel import Panel
from rich.table import Table
from rich import box

app = typer.Typer(
    name="word-count",
    help="Count words in Markdown files.",
    add_completion=False,
)
console = Console()


MARKDOWN_PATTERNS = [
    r"```[\s\S]*?```",  # fenced code blocks
    r"`[^`]+`",  # inline code
    r"!\[.*?\]\(.*?\)",  # images
    r"\[.*?\]\(.*?\)",  # links => keep link text
    r"^#{1,6}\s+",  # ATX headings
    r"^\s*[-*+]\s+",  # unordered list markers
    r"^\s*\d+\.\s+",  # ordered list markers
    r"[*_]{1,2}([^*_]+)[*_]{1,2}",  # bold / italic => keep inner text
    r"~~([^~]+)~~",  # strikethrough => keep inner text
    r"^>+\s*",  # blockquote markers
    r"^\s*\|.*\|\s*$",  # table rows (kept as-is, words counted)
    r"^[-*_]{3,}\s*$",  # horizontal rules
    r"<!--[\s\S]*?-->",  # HTML comments
    r"<[^>]+>",  # remaining HTML tags
]

_STRIP_RE = re.compile("|".join(MARKDOWN_PATTERNS), re.MULTILINE)


def strip_markdown(text: str) -> str:
    """Remove Markdown syntax, keeping readable prose."""
    # Replace links/images with their label text
    text = re.sub(r"!\[.*?\]\(.*?\)", "", text)
    text = re.sub(r"\[(.*?)\]\(.*?\)", r"\1", text)
    # Remove fenced code blocks entirely
    text = re.sub(r"```[\s\S]*?```", "", text)
    # Remove inline code
    text = re.sub(r"`[^`]+`", "", text)
    # Unwrap bold / italic
    text = re.sub(r"[*_]{1,2}([^*_\n]+)[*_]{1,2}", r"\1", text)
    text = re.sub(r"~~([^~]+)~~", r"\1", text)
    # Remove HTML comments and tags
    text = re.sub(r"<!--[\s\S]*?-->", "", text)
    text = re.sub(r"<[^>]+>", "", text)
    # Strip leading syntax characters
    text = re.sub(r"^#{1,6}\s+", "", text, flags=re.MULTILINE)
    text = re.sub(r"^\s*[-*+]\s+", "", text, flags=re.MULTILINE)
    text = re.sub(r"^\s*\d+\.\s+", "", text, flags=re.MULTILINE)
    text = re.sub(r"^>+\s*", "", text, flags=re.MULTILINE)
    text = re.sub(r"^[-*_]{3,}\s*$", "", text, flags=re.MULTILINE)
    return text


def count_stats(text: str) -> dict:
    words = text.split()
    lines = text.splitlines()
    chars_no_space = len(re.sub(r"\s", "", text))
    sentences = len(re.findall(r"[.!?]+", text))
    return {
        "words": len(words),
        "lines": len(lines),
        "chars": len(text),
        "chars_no_space": chars_no_space,
        "sentences": max(sentences, 1),
        "avg_word_len": (
            round(sum(len(x) for x in words) / len(words), 1) if words else 0.0
        ),
        "reading_time_min": max(1, round(len(words) / 200)),  # ~200 wpm
    }


def _count_single_file(
    file: Path,
    strip: bool,
    verbose: bool,
    plain: bool,
) -> dict:
    """Count words for a single file, print output, and return stats."""
    raw = file.read_text(encoding="utf-8")
    text = strip_markdown(raw) if strip else raw
    stats = count_stats(text)

    if plain:
        typer.echo(f"{file.name}\t{stats['words']}")
        return stats

    if not verbose:
        console.print(
            Panel(
                f"[bold cyan]{stats['words']:,}[/bold cyan] words  ·  "
                f"[dim]{stats['reading_time_min']} min read[/dim]",
                title=f"[bold]{file.name}[/bold]",
                border_style="cyan",
            )
        )
        return stats

    # Verbose: full table
    table = Table(box=box.ROUNDED, show_header=True, header_style="bold magenta")
    table.add_column("Metric", style="bold")
    table.add_column("Value", justify="right")

    rows = [
        ("Words", f"{stats['words']:,}"),
        ("Lines", f"{stats['lines']:,}"),
        ("Characters (with spaces)", f"{stats['chars']:,}"),
        ("Characters (no spaces)", f"{stats['chars_no_space']:,}"),
        ("Sentences (approx.)", f"{stats['sentences']:,}"),
        ("Average word length", f"{stats['avg_word_len']} chars"),
        ("Estimated reading time", f"{stats['reading_time_min']} min"),
        ("Markdown stripped", "yes" if strip else "no"),
    ]
    for label, value in rows:
        table.add_row(label, value)

    console.print()
    console.print(f"  [bold]{file}[/bold]", style="dim")
    console.print(table)
    console.print()
    return stats


@app.command()
def count(
    path: Path = typer.Argument(
        ...,
        help="Path to a Markdown file or a directory with digit-prefixed .md files.",
        exists=True,
        file_okay=True,
        dir_okay=True,
        readable=True,
    ),
    strip: bool = typer.Option(
        True,
        "--strip-markdown/--no-strip-markdown",
        help="Strip Markdown syntax before counting (default: True).",
    ),
    verbose: bool = typer.Option(
        False,
        "--verbose",
        "-v",
        help="Show a full breakdown table.",
    ),
    plain: bool = typer.Option(
        False,
        "--plain",
        help="Print a bare number (word count only) — useful for scripting.",
    ),
):
    """Count words in a Markdown FILE or all digit-prefixed .md files in a directory."""
    if path.is_file():
        _count_single_file(path, strip, verbose, plain)
        return

    # Directory mode: find .md files starting with a digit
    files = sorted(x for x in path.glob("[0-9]*.md") if f.is_file())
    if not files:
        console.print(f"[red]No digit-prefixed .md files found in {path}[/red]")
        raise typer.Exit(code=1)

    total_words = 0
    for f in files:
        stats = _count_single_file(f, strip, verbose, plain)
        total_words += stats["words"]

    if plain:
        typer.echo(f"TOTAL\t{total_words}")
    else:
        console.print(
            Panel(
                f"[bold green]{total_words:,}[/bold green] words across "
                f"[bold]{len(files)}[/bold] files  ·  "
                f"[dim]{max(1, round(total_words / 200))} min read[/dim]",
                title=f"[bold]{path}[/bold] — Total",
                border_style="green",
            )
        )


if __name__ == "__main__":
    app()