Yield Improvement AI — Join Keys, Defect Codes and Payback

The first question anyone asks about yield improvement AI is what accuracy the model will reach. In practice, projects stall well before the model. The defect records and the process records cannot be joined on a common key, and on that single point most of the investment ends without reaching a result. This article works through the two design items that decide the outcome — the join key and the defect code — and follows the money to a five-layer cost breakdown and a payback period. It is written at a level of detail you can take straight to your own data and check.

What yield improvement AI actually does — and how it differs from visual inspection AI

The technologies lumped together as “AI for factories” are in fact three separate things. Ask for a quote while they are still mixed together and you end up buying something that does not move the metric you wanted to move.

The first is the detection type. It judges scratches, chips and foreign matter from images captured by a camera. This is what people normally mean by visual inspection AI. The second is the prediction type. It reads process data such as temperature, pressure, current and torque, and anticipates the defects that are about to appear. The third is the root cause type. It matches past defects against process data and narrows down which conditions are driving the defects. This is the field known as factor analysis or root cause analysis.

What deserves the name yield improvement AI is the second and the third. The first, detection, is not a technology that raises yield.

The cell to study in that table is “Metric it moves” on row 1. What visual inspection AI improves is the proportion of defects that slip past inspection and reach the customer — the escape rate — and the labour hours spent on visual checks. The defects themselves keep occurring at exactly the same rate. In fact, the better the inspection accuracy, the more defects get caught inside the plant. As long as you define yield as good units divided by units started, installing visual inspection AI will not raise it. The detailed decision criteria are set out separately in how to judge an AI visual inspection project, but once you separate the goal into preventing escapes versus suppressing occurrence, it becomes clear that the two are not even being compared on the same basis.

Types 2 and 3 are siblings standing on the same data foundation. Only once type 3 has established that defects appear under a given condition does the prediction from type 2 become information the floor can act on. Skip type 3 and build only type 2, and you get alerts with no indication of what to fix. The floor watches those alerts for the first few weeks and then stops looking.

There is a reason the three get confused. Detection has many reference cases and is easy to demonstrate. Put a defective part in front of the camera and the judgement appears on the spot. Types 2 and 3 cannot be shown that way. They produce results only after process data has been joined and matched against past defects, so there is nothing to demonstrate live in a sales meeting. The result is a familiar drift — a project that began as a conversation about yield gradually turns into a visual inspection proposal.

There is one more thing to settle internally before any of this. The definition of the number your own company calls yield. Is it final good units against units started, or the product of pass rates at each operation? Are units that became good after rework counted in the numerator? If the definition differs between departments while an improvement target is being set, the numbers will not reconcile when you measure the effect. In a plant that counts reworked units as good, the benefit shows up as a reduction in rework hours rather than as a movement in yield, so it is more accurate to put the measurement on the labour side from the outset.

When a quotation arrives, the first thing to check is not the amount but the subject of the sentence. If the subject of the proposal is “images”, it is an investment in the escape rate. If it is “process data”, it may be an investment in yield. If the proposal mixes both, ask for the cost breakdown to be split between type 1 and types 2 and 3. That small extra step almost entirely removes the later conversation that begins with “this is not the effect we expected”.

Why yield improvement AI stalls — the gap between 70% collecting and 40% benefiting

The 2026 White Paper on Manufacturing Industries (cabinet decision, 29 May 2026) puts a number directly on this problem. Roughly 70% of businesses are capturing data from their manufacturing processes. Only about 40% are using that data and getting a benefit from it.

The gap is roughly 30 points. That 30 points is not made up of companies that fail to collect data. It is made up of companies that collect it but cannot join it. The sensors are installed. The inspection records exist. The daily production reports are archived. And the analysis still does not come together.

Break the reason down on the shop floor and it narrows to three fault lines.

Of the three, the one that wastes the most money is the second. The first can be fixed by rebuilding the defect code table, and improvement starts from the next period. The third is technically solved by standing up an NTP server and synchronising every system. But the second is about changing how the data itself is held, which means altering the output specification of inspection machines, PLCs and MES systems that are already running. Discovering after the implementation project has started that the data will not join means going back and redoing the design.

Clock synchronisation is treated lightly relative to its impact. On an operation with a cycle time of tens of seconds, if the inspection machine and the PLC are a few minutes apart, the process conditions attached to a defective unit will actually belong to the unit next to it, or the one after that. You end up with individual unit IDs neatly in place and the wrong conditions stuck to them. If the offset is constant it can be corrected afterwards, but where an operator periodically sets the clock by hand, the drift is not constant and cannot be corrected. Whether a synchronisation mechanism exists has to be confirmed system by system.

Here is the check you can run tomorrow. Ask quality assurance for twelve months of defect records and production engineering for the same twelve months of process data, both as CSV. Open the two files side by side and look for columns that appear in both. If the only thing they share is a date, your plant is on the wrong side of that 30-point gap. This check takes half a day and does not require calling an AI vendor.

Designing the join key — the three layers of lot, unit and timestamp

A join key has three layers. How far you go determines the range of questions you can answer.

Most plants hold only the lot layer. And with only the lot layer, the cause cannot be identified. The reason is not a matter of attitude — it is the structure of the data.

Take a lot of 2,000 units. The model line in this article runs at a current yield of 96.4%, so the defect rate is 3.6% and there are 72 defects per lot. Recording only at lot level means all 2,000 units carry exactly the same process condition values. The 72 defects and the 1,928 good units are all tied, in the data, to the same single row reading “temperature 185°C, pressure 4.2 MPa, machine A”.

Factor analysis is the work of finding a difference in the distribution of explanatory variables between good and defective units. If good and defective units hold identical values, neither statistics nor AI can detect a difference. No model you bring in will change that result. Analytical resolution cannot exceed the resolution of the record.

Add the unit layer and the situation changes. You can now see whether those 72 defects skew towards machine A, cluster in a particular time band, or concentrate in a specific cavity position. At that point, a good deal is visible to the human eye before any AI is involved. It is genuinely common for the cause to become clear within the first few weeks of establishing unit-level traceability, before a model has been built at all.

The timestamp layer goes one step deeper. Even within the same machine and the same lot, the temperature profile at the first run of the morning differs from the profile in the early afternoon. This variation is flattened into an average at the lot layer and survives at the timestamp layer. Once you can see that defects concentrate in the first 30 minutes after start-up, the countermeasure lands as a concrete work instruction — revise the preheat time.

The thing most easily overlooked here is the record on the good side. In many plants the practice is to record details only when a defect occurs, and to pass good units through with nothing more than a count. But factor analysis compares the conditions of defective units against those of good ones. With no basis for comparison, finding a condition common to the defective units tells you nothing about whether that same condition appears just as often on good ones. The observation “the temperature was high when the defect occurred” becomes meaningless the moment you learn the temperature was equally high on good units. If you are going to assign unit IDs, assign them to good units too. Limiting them to defective units lowers the cost, but it also stops the analysis from working.

Which of the three layers you need is decided by how your defects appear

You do not need to build all three layers at once. The criterion for judgement is the way your defects appear.

If defects appear grouped by lot, the cause is likely to sit on the material lot or the changeover, and the lot layer alone will point you in a useful direction. If defects are scattered within a lot, the cause sits on the unit, the machine or the moment in time, and the lot layer will never reach it. Start by confirming which of the two patterns your plant has using the last three months of defect records.

The second criterion is whether the process runs in parallel. Several identical machines on the same operation, or a single mould with multiple cavities. With that configuration, unless you identify the machine and the cavity, half the cause is invisible from the start. A plant with parallel operations has no realistic option of stopping at the lot layer.

How to number units and how to choose a reading method are organised by scenario in choosing barcodes for traceability. The point to carry across here is that choosing a marking method that cannot be read at downstream operations breaks the join partway through, however carefully you numbered the units.

What to confirm. Whether machine, cavity and timestamp exist as columns in your own defect records. If they exist but are always blank, they may as well not exist.

Designing defect codes — analysis stops once “Other” passes 30%

Even with join keys in place, analysis will not move forward if the defect codes are coarse.

Open your defect code table and sort the last twelve months of counts in descending order. If “Other”, “Appearance defect” and “Dimensional defect” sit at the top, and “Other” alone is over 30%, factor analysis on that data will not produce an answer. The reason is simple — “Other” is the collection of defects nobody could classify. In other words, everything you most want to know is sitting inside it. The model will merely learn to predict a label called “Other”, which leads to no countermeasure whatsoever.

Defect codes are easiest to design when broken into three axes.

Multiply the three axes together and the number of codes appears to explode. But what grows is the number of rows in a table, not the input burden on the floor. The inspector selects from three drop-downs, which is faster than free text. If anything, the present practice — selecting “Other” and writing a note in Thai in the remarks field — imposes a far greater burden downstream.

And Thai plants carry one additional fault line that plants in Japan do not have. Three languages and three systems refer to the same part by three different IDs.

All three refer to the same part, yet as strings they do not match. Trying to join data in this state creates a name-matching exercise. Worse, handwritten Thai is not searchable as it stands. Plenty of plants have people reconciling this by hand every month, and those hours rarely appear anywhere in the data preparation line of a quotation.

The countermeasure comes down to fixing a single code system and making all three systems carry that code. The display language can stay as three languages. Make it Thai on the screen, code in storage and the inspector’s input burden does not increase while the downstream join becomes possible. The overall design of how quality records are held is covered in how to choose a quality data management system, but in terms of sequence, fixing the code system before selecting a software package causes far less rework.

It is worth writing down a realistic way to rebuild a code system as well. Do not go back and re-code history. Re-coding the past consumes an unpredictable number of hours, and because the people who witnessed those phenomena are no longer around, it usually just recreates “Other” under a different name. Apply new codes from the switchover date and leave everything before it on the old codes. Count the analysis period from the switchover onwards. In the early stage of the transition, simply giving inspectors a printed sheet showing the old codes alongside the new ones cuts the variation in selection considerably.

Decide, too, who is allowed to add options to the code list. If the floor can add freely, the list balloons within six months and fills with codes that mean the same thing. If quality assurance alone controls it and additions take a long time, people escape to “Other” every time a new phenomenon appears. Fixing the route for requesting an addition and setting a monthly review slot avoids both failure modes.

What to confirm. Sort the last twelve months of defect counts by code in descending order and see what share “Other” and “Appearance defect” account for between them. Then check whether their content is written as free text in the remarks field.

The cost of yield improvement AI — five layers and a payback period

Cost is easier to judge when split by layer rather than by feature. The breakdown below assumes a Japanese-owned plant in Thailand and a single line. The currency is THB.

Annual running cost is estimated at 780,000. Maintenance of the data platform, retraining of the model and updates to master data make up the bulk of it.

What stands out in that table is that layer 2, joining, is the largest at 1,800,000. Intuitively, the cost of an AI project ought to centre on model development in layer 3. In reality layer 3 comes to 900,000, half of layer 2. The centre of gravity of yield improvement AI cost is ID design, not the model. And this is exactly the layer that gets collapsed into a single vague line in a quotation — “initial data preparation”, “master data setup” — which makes it the easiest target when someone wants the price cut.

What happens if you cut layer 2? Build only layers 3 and 4 first and a combined 1,500,000 sits there without producing a result. The analytics platform runs. The screens display. But because the table that feeds factor analysis cannot be built, all that comes out is a tally of results by operation. Tallies can be done in Excel, so that cannot be called a return on the investment. Starting with layer 1 alone is unlikely to be wasted effort, in the sense that data accumulates in the meantime. It is specifically the sequence of skipping layer 2 and moving to layers 3 and 4 that should be avoided. For a sense of the collection layer cost, the cost of factory IoT has a breakdown by configuration.

The benefit side is set out at the same granularity.

The yield improvement is set at +1.2 points. That may look modest, but the causes found by factor analysis usually concentrate in one or two conditions, and the effect of eliminating them lands in about that range. Conversely, if a proposal promises to halve your defects, it is worth asking which condition at which operation that claim rests on.

If the budget will not stretch, it helps to know which layer can wait. The 600,000 for layer 4, floor UI and report integration, is often fine in a simplified form the first time round. Feeding factor analysis results back to the floor can work as a monthly report for the time being. Layer 5, requirements definition and PM, looks like the easiest cut, but cutting it leaves nobody owning the coordination across three systems, and layer 2 then fails to progress. Layer 1, collection, can be reduced by narrowing scope. Rather than covering every operation from the start, limit it to the one operation with the most defects and the wiring and measurement points shrink accordingly. What you may cut is scope, not layers.

How to read the 29.7-month payback depends on your capital investment criteria. A plant with a three-year threshold will approve it. A two-year threshold will not. Before you stop the discussion there, look at the numerator rather than the denominator. If the target line has an annual production value larger than 240,000,000, the same cost pays back faster. Yield improvement AI does not scale its investment closely with production volume. Rolling the mechanism built for one line out to a second line adds only part of layer 1, while layers 2, 3 and 4 carry across almost entirely. That makes it rational to choose the line with the largest production value as the first one.

What to confirm. The annual production value of the target line and the current defect loss in money. Both can be pulled from accounting and quality assurance records in half an hour. If that loss is not comfortably larger than the 780,000 annual running cost, that line is not a suitable target.

What to check first in a Thai plant — what one point of yield is worth

Plants operating in Thailand have decision inputs that plants in Japan do not.

On 5 January 2026 the Joint Standing Committee on Commerce, Industry and Banking (JSCCIB) noted that capacity utilisation was running below 60% at many manufacturers, and that the export outlook was difficult. Reporting from the same period put the 2026 GDP growth outlook at 1.6–2.0%. When utilisation falls, fixed costs stay the same while the denominator shrinks, so the manufacturing cost per unit rises.

There are differences by sector, though. A Xinhua article of 8 June 2026 reported that electrical and electronic goods, air conditioning, food and beverage and rubber products are supported by demand, while older industries are squeezed by high electricity costs, shortages of raw materials and competition from abroad. Which side you sit on changes the moves available to you.

The question that bites in this environment is what internal source of profit is left when price increases will not go through. Selling prices are set by the market. Raw material costs depend on negotiation with suppliers. Labour costs cannot be reduced. Electricity tariffs are outside your control. What remains is one thing only — how far you can cut the share of material and labour you put in that is thrown away instead of becoming product.

On the model line, that discarded portion is 8,640,000 a year. Move yield by +1.2 points and 2,880,000 stays in the business. That is profit created neither by price negotiation nor by a recovery in utilisation, but entirely inside your own process. In a period of low utilisation there is even an argument that it is easier to execute. If the line is not running flat out, there is room to run verification trials with conditions deliberately varied.

On the practical side, Thailand adds one more item to confirm. Whether there is anyone in-house who can handle the data. Extracting process data and maintaining master data on a continuing basis requires a designated owner among the local staff. If you build the plan on the assumption that a Japanese manager will run it alongside another role, operations will stop at the next rotation. Making it operable in Thai and handed over in Thai is the condition for the system still running three years from now. How far other companies keep this in-house is set out, including operating structures, in AI use cases on the shop floor.

What to confirm. Your annual production value multiplied by your current defect rate. Whether that amount is proportionate to the budget of the activity you internally call yield improvement.

How to proceed — 90 days to confirm one thing only, whether the data joins

There is no need to sign a contract for a full build at the outset. What the first 90 days should establish is not model accuracy but whether the data can be joined at all.

Of the three phases, the one that exposes the most failures is the second. And the discovery is a correct one. Learning within 90 days that the data will not join is a successful verification. At that stage you can pinpoint whether what you need to add is unit ID numbering, clock synchronisation or a rebuilt defect code table, and spend money only there. Learn the same thing after signing for a full build and layers 3 and 4, already ordered, are left hanging.

There is also a reason for narrowing the final 30 days to a single theme. The broader the theme, the harder factor analysis results are to interpret. Ask about “the causes of defects in general” and what comes out are obvious variables like part number and operation. Narrow it to “cracks occurring at operation Y on part number X” and the variables that come out are the conditions themselves. Producing a result the floor can judge is the purpose of the 90 days.

Through this period, what you should require from a vendor is not an explanation of the model but a list of the fields that could not be joined. If they can produce that concretely, the work beyond it can proceed on a realistic footing.

The team should be settled in advance too. What the 90-day verification needs is three people — someone in quality assurance who can extract the defect records, someone in production engineering who can extract the process data, and someone in manufacturing who can judge whether it does occur under that condition. Without the third, nobody can assess the validity of the analysis and all you are left with is a report. Conversely, if those three can sit in the same meeting, the verification depends only lightly on the vendor’s capability. Almost all of the work can proceed without stopping the line, so a period of low utilisation is if anything a good moment to start.

What to confirm. If the proposal says nothing more than “data preparation”, ask for that line to be broken down to the level of a field list for all three systems. A proposal that cannot be broken down will stall on day 90.

Frequently asked questions

What is yield improvement AI?

It refers to mechanisms that match process data against defect records to narrow down the causes of defects, or to anticipate defects before they occur. Its purpose differs from visual inspection AI, which finds defects in images. Visual inspection AI moves the escape rate and inspection labour, while yield improvement AI moves the rate at which defects occur. The condition for it to work at all is that the defect records and the process records can be joined on a common key.

How much does yield improvement AI cost?

For a Japanese-owned plant in Thailand and a single line, the initial cost is 5,200,000 THB and the annual running cost is 780,000 THB. The breakdown is collection 1,200,000, joining 1,800,000, analytics platform and model 900,000, floor UI and report integration 600,000, and requirements definition and PM 700,000. The largest layer is joining, not the model, and cutting it leaves the 1,500,000 spent on the analytics platform and the floor UI without a result.

Should we install visual inspection AI or yield improvement AI first?

It depends on the metric giving you trouble. If customer complaints or escapes are the problem, visual inspection AI comes first. If the occurrence of defects inside the plant is the problem, yield improvement AI comes first. If both are issues, installing visual inspection AI first produces a benefit sooner. That said, if you keep the visual inspection AI judgements linked to unit IDs, those records become the input for yield improvement AI as they stand. In terms of sequence, completing unit ID numbering at the point you install visual inspection AI wastes the least effort.

How much defect data do we need, over what period?

Record granularity matters before period or volume. If records exist only at lot level, no number of years will yield a cause. Conversely, if you have unit and timestamp data, narrowing the scope to one operation and one part number and holding enough of a period to see seasonal variation will point you in the right direction. The first thing to confirm is not the volume accumulated but whether defective and good units carry different process condition values.

Is there any benefit on an operation where the defect rate is already low?

Judge it in money. On the model line the defect loss is 8,640,000 a year and a +1.2 point improvement is worth 2,880,000. Where production value is large, money remains even at a low defect rate. Conversely, on a line with small production value, the 780,000 annual running cost may not be recovered even at a high defect rate. Compare that line’s defect loss in money against the running cost, rather than looking at the defect rate as a percentage.

Should process data analysis be done in-house or outsourced?

Splitting it by layer is the realistic answer. ID design, master data preparation and clock synchronisation require your own process knowledge, so even when outsourced your own team leads. Building the analytics platform and implementing the model suit outsourcing. Retraining and master data updates in the operating phase have to stay with local staff or they will not continue. Whether generative AI can substitute for the entry point of factor analysis is a question that needs separate verification, so see can generative AI be used for data analysis.

Summary

What separates success from failure in yield improvement AI is not model accuracy but whether the defect records and the process records can be joined on a common key. Most of the gap shown in the 2026 White Paper on Manufacturing Industries — roughly 70% capturing data against roughly 40% getting a benefit — arises not among companies that fail to capture, but among companies that capture and cannot join.

The order of judgement comes down to three things. First, separate visual inspection AI from yield improvement AI. The former moves the escape rate, not yield. Second, treat the join key as three layers of lot, unit and timestamp, and decide which layers you need based on how your defects appear. With lot-level records, defective and good units carry the same condition values, so no cause will emerge. Third, reduce the “Other” category in your defect codes. Analytical resolution cannot exceed the design resolution of the defect codes.

Cost is 5,200,000 THB initial and 780,000 THB a year to run. The largest layer is joining at 1,800,000, and cutting it strands the 1,500,000 spent on the analytics platform and the floor UI. On the benefit side, moving yield from 96.4% to 97.6% on a line with an annual production value of 240,000,000 THB gives a reduction of 2,880,000, a net of 2,100,000 after running costs, and a simple payback of 29.7 months. In a period where utilisation is below 60% and price increases will not go through, that 2,880,000 is one of the few sources of profit you can create entirely inside your own process.

The first thing to do is neither a contract nor a selection exercise. It is to put the defect record CSV and the process data CSV side by side and look for a common column.

Whether a join will work on your own data can be judged before any system is selected. TOMAS TECH works with Japanese-owned plants in Thailand, reviewing the actual output of existing inspection machines, PLCs and production management systems, and setting out which layers need attention. We are happy to talk at the exploratory stage, before any decision to implement, looking at your data together. You can contact us here.

References

• 2026 White Paper on Manufacturing Industries (Monodzukuri White Paper), Ministry of Economy, Trade and Industry, Ministry of Health, Labour and Welfare, and Ministry of Education, Culture, Sports, Science and Technology, cabinet decision 29 May 2026 https://www.meti.go.jp/report/whitepaper/mono/2026/pdf/gaiyo.pdf
• Project Design Online (Jigyo Koso Online), “2026 White Paper on Manufacturing Industries approved by the Cabinet”, June 2026 https://www.projectdesign.jp/articles/news/874807b9-713f-420d-805c-5cb66072cf47
• The Nation Thailand, “FTI warns of ‘perfect storm’ as 2026 GDP growth seen at 1.6–2.0%”, 5 January 2026 https://www.nationthailand.com/business/economy/40060791
• Xinhua, “Thai industrial sector sees mixed outlook amid tech growth”, 8 June 2026 https://english.news.cn/asiapacific/20260608/faa838fea8ad46ef87ac64736d8dc6a9/c.html
• Databricks, “Manufacturing Root Cause Analysis with Causal AI” https://www.databricks.com/blog/manufacturing-root-cause-analysis-causal-ai
• Arch Systems, “From Machine Data Extraction to Manufacturing Root Cause Analysis” https://archsys.io/hub/articles/from-machine-data-extraction-to-manufacturing-root-cause-analysis/