I was recently talking to a manufacturing company for automating one of their factory floor processes with AI. It’s a large family business of over 50 years in a traditional industry, things largely done with human labor and less digitization. They currently have 50+ humans managing that particular process. While it is grossly inefficient, I noticed that per unit cost of deploying a human is ~50p.

AI transformation is costly - both in money, time and focus. You need to have a reasonable scale to pursue it.

Something didn’t sit well with me here. Even if we’re able to get the cost down to 10p, the cost of change management, new hardware, training etc wouldn’t justify the investment at their current scale. A lot of founders are so fascinated with AI that they forget to evaluate whether it makes sense for their business.

I went back to them with my concerns and that’s when I met the founder. He’s a 3rd generation owner of the company. 15 minutes into the conversation, he mentioned something you rarely hear in selling software - “I don’t care about cost.” (Read: Cost is not my primary concern).

He says,

“I do this not to reduce cost but to create a brand. With a clear digital footprint, I can offer something that none of my other competitors in the market can offer - A real time view into the production. My customers can now see what stage their order is in, quality of the order without follow ups or back and forth. I also reduce returns because there is no ambiguity around what is produced. This will help me establish myself as the preferred vendor in my market. I’m not saving cost. I’m building my brand.”

I’m not saving cost. I’m building my brand.

This just clicked for me. A cost based value add is largely the default narrative in the age of AI. We often forget that AI is not just a chaos but an opportunity that helps companies redefine their business in ways that will make them resilient and strong.

It’s not just about “automating an operational process” but creating unique strategic advantages for your business.

Share Playing on the front foot

Source:

Mike Judge

Where's the Shovelware? Why AI Coding Claims Don't Add Up

I’m furious. I’m really angry. I’m angry in a knocking down sandcastles and punching Daniel LaRusso in the face and talking smack about him to his girl kind of way…

Read more

9 months ago · 582 likes · 116 comments · Mike Judge

Nope. There’s no improvement.

I agree with the data presented in these articles. AI is not making devs productive. In fact my personal opinion is that it makes it worse. This is the same for most people in appliedai.club who have been working on their side projects.

While actual research studies will come out soon, here’s my observations.

1. AI does improve productivity in specific fields

Take customer support. While, there are proofs that human support agents cannot be replaced yet, we see that there’s already a big productivity increase in support using AI.

Intercom’s Fin AI’s share of ticket completion.

Source: https://www.anthropic.com/customers/intercom

So, why does that not translate to software development? That’s because, addressing a ticket is a lot more standardized than building software. Software still needs a complete human + AI involvement. I’m sure if we take the AI + Human flow in CS, the productivity will be bad.

2. Context Switching

If you’re someone who’s coded in the past, you know that you need to get into some kind of a flow state to really start being productive. And these AI tools do not make you do that. The biggest problem with one of these tools is that you let them run and go back after 2 minutes to see if they have done the job. It’s like resetting your context every two minutes. I typically have a youtube video running in the background which I check when the AI is working. Completely unproductive.
PS: Let me know if you have a better approach to this.

3. Know Parkinson’s Law

“Work expands to fill the available time”

Read more at https://en.wikipedia.org/wiki/Parkinson's_law

This is a big problem at companies. Just because you give someone an AI tool does not mean they will take up more work. This usually does not happen. Engineers still estimate work based on pre-AI learnings. Even if AI makes people more productive, they spend that in other ways like browsing, taking to people or leaving early.

I see it in every organization that I work.

At ZORP, my previous startup, the way we solved this is by not hiring new developers when the demand increased. We were able to handle 3x the requests with the same bandwidth as people got better and better at using software. In some ways, we lucked out.

One way to get this is to let go of people and do the same thing with a smaller org. But this almost never works because letting go of people has much more devastating consequences.

I recently switched from my startup’s publication to my personal one. If you’re new hear, I write about technology, history and contemporary economics especially focused on incentives.

Subscribe now

Now, off to today’s topic.

A quick check on OpenRouter shows 486 models and close to 55 model providers.

credits: https://openrouter.ai/models?fmt=table

This does not show a lot of companies are involved in training and fine-tuning their own custom models for internal use. But a lot of industry impression is that we will have a very powerful universal model which will render all these individual expert models obsolete. Is that true? Let’s discuss.

Current State of Models

If you’re someone who is working with LLM models frequently, you might be using a few for different tasks. I use

Open AI - for all my daily tasks, brainstorming, project management

Claude - For coding

Gemini - For image processing

Deepseek, Gemini and Qwen - In my projects.

My objective here is to get the best performing model for each task. Clearly, Claude is better at coding than any other model today. Similarly, I would like to use smaller or cheaper models where I can so that I can spend less money overall.

Using custom models

Let’s take the example of just a marketing department in an ecommerce company. Here are some of the activities the company does, in no order of importance.

• Schedule Ad campaigns across all social media

• Review ad performance

• Create videos for YouTube

• Create short forms for tiktok, youtube, instagram and more

• Write SEO content. Now, also support LLM’s SEO.

• Email campaigns to past customers.

• Affiliate marketing

• Influencers coordination

• and so much more.

Assume that you want to staff each of these departments with models that improve their productivity. Would you be using the same generic model for all of those?

In the current state of things, No.

That is because, we know that the model that creates videos may not be the best at working on SEO content. And writing emails might need a completely different model compared to scheduling Ad campaigns. You get the gist.

And all-encompassing model

Now, in the (possibly) near future, someone might create a model that is so powerful that it can learn a new process so well that you need not fine-tune it at all. In that case, it is much easier to use this one single model than investing in that effort to switch between the models. When that happens, does it make sense for us to even think about using the other models?

Now, we’ve arrived at our primary question. Let’s park this for a bit. We will set some more context and come back to this.

In a future where a single model can effectively learn and perform any process better than fine-tuned models, does it make sense to invest the effort to work with multiple models?

How LLMs learn

Let’s understand some basics around how LLMs learn a new process. Here are some ways.

Inherent knowledge

Eg., Translating a text from German to English, creating a 140 char tweet from a long article.

A lot of models can do this out of the box because their training data has this information. They’re able to utilize this capability and perform these tasks efficiently.

Few shot examples

This is a case where you provide a few examples on how to perform the job well.

Eg., Say you want to update a lead in Hubspot. You can provide inputs to the LLM about how to use the Hubspot tool or MCP.

Another example is when you want the LLM to create marketing copy. You can instruct the LLM with a few examples about your style.

The LLM learns from this example and performs the job.

RAG and other forms of context engineering

Few shot examples, RAG, etc are different ways of providing context to the LLM. In cases where the process is not straightforward, we provide context to the LLM that will help the LLM to understand the process better.

Example: Say, you need the LLM to send an approval email for a vendor invoice. Now, LLM can use a memory or search to identify who it needs to send the approval to dynamically.

Another example is when the LLM provides coding suggestions for working with a specific language. Now, it can look at the documentation and provide the right suggestions or directly update the code.

Fine-tuning/reinforcement learning

These are mechanisms that help improve the inherent knowledge of the LLM. Instead of training the LLMs from scratch, we use a decent set of examples to add to the inherent knowledge.

**Important Note: Only in the case of fine-tuning that the model’s characteristics get changed. So it, by definition, becomes a new model.

…Back to model choices

Now that we know how a model acquires knowledge to perform a task, let’s go back to our main question.

At what point does the benefit of using a generalized model better than a fine-tuned model for that specific use case.

There are 3 criteria which will impact this decision. We will walk through each of them with an example of a TTS model.

**Note: TTS is a Text-To-Speech model that converts text input to an audio output.

Model Capability

Is the generalized model capable of learning and doing the task at the same level as the fine-tuned model? In tasks where this is a yes, you will choose the generalized model.

Eg., say you want to make a sales call. The SOTA model ElevenLabs provides multiple options that help you choose the right voice, tone etc to make the call and speak to the customer. In that case, fine-tuning a custom model at your end to do the same task might not even reach the capability of ElevenLabs.

But say you want to make a collections call or a contract negotiation. The tonality of this call is entirely different from Sales and ElevenLabs usually fails in these cases. In such cases, if you have past human conversations, the custom fine-tuned model will perform much better than the SOTA models.

Model Cost

Is the generalized model as cost efficient as a fine-tuned model.

Cost of ElevenLabs TTS Model: On their scale plan, it’s $0.11 per minute.

A custom trained model hosted on your infra: ~$0.02 per minute. (Much less in a lot of cases)

When you make a lot of calls and you can get your fine-tuned model closer to the SOTA model, it makes sense to fine-tune your own model.

Data Security

All large model providers provide enterprise plans with complete data security. Even with this, a lot of companies are reluctant to expose their internal data to the outside world.

If you’re one of those, then still choose internal fine-tuned and open source models.

So how will the future look like?

Here’s my thoughts on this.

While the large model providers will cover more and more use cases in the public world and get better at them, there will always be complex internal workflows that the models won’t be able to learn just with context. You’ll always need a way to fine-tuned these models for those specific jobs. Eg., would you simply trust open AI to make your stock investments? Ever? I don’t think I would. But a specific model created to understand markets, maybe.

But the big factor is going to be cost. The model costs are already very low. If they could get this to a stage where the cost is incredibly low, the effort involved in fine-tuning, serving the models and managing their infrastructure is simply not worth it.

A good analogy to this is what happened with cloud where cost of running your servers is simply not worth it for most companies. Expect to see the same with models.

I currently belong to the group that believes that a single universal model that does everything well is simply not going to happen. We will see plenty of use case specific models that will help you solve specific problems really well at a much lower cost.

What do you think?

“When you’re a carpenter making a beautiful chest of drawers, you’re not going to use a piece of plywood on the back, even though it faces the wall and nobody will ever see it. You’ll know it’s there, so you’re going to use a beautiful piece of wood on the back. For you to sleep well at night, the aesthetic, the quality, has to be carried all the way through.” - Steve Jobs

There are times when I have obsessively worked on things that adds 0 value to the business. It’s because I want it to be perfect. More often than not, it has not added any additional value to me or to the business but it has made me happy.

To be frank, this doesn’t happen often. I only have this kind of motivation rarely.

I love when you can see the love put into work. I came across this landing page for “LoveFrom” recently and just made me smile.

https://www.lovefrom.com/

This is so tasteful, you can feel the love that’s put into it.

Quality

There’s this attribute called Quality. It’s hard to define quality when you ask someone what it is. But it’s one of those things that you know when you see it. (It’s like PMF :)).

The concept is so amazing that an entire book was written around it.

I strongly believe that things that are done with love often charge a premium. My mental model on quality vs value look like this.

I collect the list of things that are done with taste and quality that I often look at them to keep reminding what pure love of art looks like. It’s hard to replicate it with any AI.

Let's say you recently had a COVID test done (I sincerely hope you did not have to). And the test came positive. Let's say the accuracy of the test is 99%, how sure are you that you indeed have COVID infection? Even with the 99% accuracy, there is a smaller chance that you have the virus.

In this post, let's have a look at why that would be so.

Let's take a step back and look at probabilities. Starting with a simple example. When you toss a fair coin, the probability of getting heads is 50% and tails is 50%. It does not mean that you will get 50% head and 50% tail. The outcome will finally be either heads or tails it is just that there is a possibility of getting the outcome is 50%. This is an important distinction.

Another way to look at it is, if I toss the same coin a large number of times, I would get heads 50% of the time and tails 50% of the time.

Another example would be to think of a bet you make on a cricket match. You predict that there is a 30% chance of India winning the match and 70% that India loses the match. You bet 50K on India winning the match. Mathematics tells us the expected payoff of this bet is (30%*50K) + (70%*0) = 15K.

But there is no outcome where you earn 15K. You earn either 50K or 0. The probability of 30% win means that if you were to repeat the bet over independent events a large number of times, you would win 30% of the time and the expected average payout is 15K.

I find the following definition of probability more relevant - probability measures the degree of rational belief to which proposition is entitled in the light of the given evidence

Bayesian Theorem is a way of updating your belief (in turn your probability) with new information. This tells you that Bayesian Theorem deals with base information (Prior Information) and New Information (Posterior Information).

Enough definition, let's jump into our problem statement - What is the real chance that you have the virus?

Let's take a sample test with the following characteristics:

• On a patient with covid, the test is accurate up to 99% of the time. Meaning, when 100 covid positive patients take the test, 99% of them have a positive result and 1 person has a negative result. We can write this as True positive (TP) = 99%, False Negative (FN) = 1%

• On a patient without covid, it is accurate up to 95% of the time. Meaning, when 100 covid negative patients take the test, 95 of them have a negative result and 5 people have a positive result. We can write this as True Negative (TN) = 95% and False Positive (FP) = 5%

Let's take a sample set of 1000 people taking the test, and you are one of them. Of this 1000, let's say 10% of the population (100 people) have covid. The results of the test could be expected as the following.

Of the 100 patients with covid, 99 test positive and 1 tests negative.

TP = 99

FN = 1

Of the rest 900 patients without covid, 855 test negative and 45 test positive

TN = 855

FP = 45

So the total positive test results are TP + FP = 99 + 45 = 144. And you are one of them. So, the chances of you having the virus = You being one of the true positive patients among all the patients who tested positive = 99/144 = 68.75%

So even with a test that is accurate up to 99%, the real chances of the test being accurate is only 68.75%! Breathe easy :)

Generalising this,

Prob that a person tested positive has covid = TP/(TP+FP)

If you want to understand the reason behind this, the change is influenced by 2 variables.

• What % of the total population being tested is actually infected (In our case, 10%) (What does our base population look like?). The behavior of the base population has a significant impact on the additional behavior of the test.

• What % of patients without covid have covid positive results (FP). (In our case, 5%).

The above logic is a working of Bayes Theorem. In simple terms, Bayes Theorem takes the behavior of underlying population (Prior Data), adds the new information (FP and TP) and provides the framework to understand the change in probability.

Zooming out, Bayes Theorem provides a mental framework to understand how, rather than looking at new information in isolation, we can improve our thinking by adding new information to the prior knowledge we possess. It is a useful mental model in decision making.

There are fantastic materials on Bayes theorem that are available in the web, I am sharing some useful ones here for additional reading/viewing.

Book: How not to be Wrong by Jordan Ellenberg