Why Not All of Them

When AI tools make text processing ubiquitous and almost free, the previous constraints fall away.

I had a quick thought this morning and I thought I’d use this blog to write about it some more.

It came to me when I was considering a new feature for a simple CRUD app that I’ve written for the day job. (Of course, when I say I’ve written, I mean it was wholly brought into life by GPT and Claude).

The idea was this: ‘Instead of filtering emails for processing data, why not take all of them, and let AI do the grunt work that I’ve been trying to avoid?’

In the day job, I’ve largely eschewed using ’tender management tools’ as in my experience they turn out to be overpriced boondoggles that I use only infrequently, and so I can’t justify the often extortionate cost.

The most common feature is that they will present a range of opportunities, gathered from public sources, filtered using complex terminology, to create a list of opportunities tailored to my business and areas of expertise. That’s great and all, but the initial fees for doing so were largely based on having humans in the loop - and each potential opportunity was manually reviewed and updated to compensate for the poor quality of the initial request.

That’s not a dig at potential clients, by the way - most are used to categorising their tenders in their own way, and fitting them into a standardised framework for publication can be unnecessarily complex.

As I’d abandoned the infrequently used, expensive platforms, I needed to deploy an alternate solution; and that was as simple as can be. Sign up to all the tender notifications, apply blunt filtering at the search page, and then receive as many as 1-200 opportunities per day via email for manual processing.

My initial thinking was to divert these emails to an AI tool, and gradually train it to filter out only the opps I want, given that the wording and go/no go criteria can be fiddly to manually implement. I could ask AI to find all the ones that match my criteria, and send me just one summary email every morning. This was initially going to be just taking the ‘pre-filtered’ emails and checking those - but then the thought hit me: ‘why not all of them?’

This led me to thinking in more depth about the processing of text and data in the light of AI. I’m used to applying procedures and filters at an operational level for almost all the data I deal with. That’s simply a condition of ADHD traits and the obvious opportunities to get distracted. When dealing with a veritable firehose of data, the chances to get distracted are many and frequent.

But the point of working out and applying pre-filters is largely gone when you can apply an incredibly fast and relentless ‘intelligence’ to the problem. Pre-filtering is just not needed. In fact, in this case, the opportunity to find new tenders that may not fit into the original selection criteria is much greater. AI can just find every needle in every haystack, with minimal effort (consuming water, power, and generating CO2 aside, of course).

And - and - the ability to tweak the english language features to explain how search criteria should be applied makes the work substantially easier, faster, and fits my mental model of ‘modelling’ the results that I want rather than just expecting miracles from a single dumb command.

So now I disappear off into LLM land, to implement a feature that will save me half an hour every day and further reduce the chance of distractions.

And the next time I think about filtering data, I’ll consider if an AI can do the job, far quicker, and consume EVERYTHING.

Why not all of them, indeed?