Local LLM Deployment | GPU Sizing, Model Selection, Cost and RFP for Thai Factories

Drawing notes, formulations, quality records, customer specifications, cost data. Local LLM deployment, meaning running open-weight models on your own servers or in a closed environment inside Thailand, is the option that comes up when manufacturers in Thailand ask us how to use generative AI with data they would rather not send to an external cloud, or cannot send under their contracts with customers. Some departments have already started using SaaS generative AI. The remaining questions are: should we set up a local LLM for confidential data, and if so, which model, on which GPU, at what cost, and who will operate it? This article answers those questions in the order the work actually happens.

This article makes one argument. Whether a local LLM succeeds depends not on GPU or model performance, but on whether you decided the following three things first. (1) Target tasks and data classification: which data cannot leave the company, and why. Tasks whose data can leave the company stay on SaaS or APIs. (2) The yardstick for evaluation: an evaluation set and pass criteria built from your own documents and your own combinations of languages. (3) The owner of operations: who handles model updates, vulnerability response, logs, and rollback when something fails. The GPU and the model are chosen only after these three are settled, by working backward to the “minimum configuration required.” Neither “everything local” nor “everything on SaaS” is the goal.

The decision itself, choosing between API, self-hosting and a hybrid, is covered in LLM Deployment Comes Down to How You Own the Foundation — API, Self-Hosted, or Hybrid. This article focuses on design and procurement for companies that have chosen self-hosting (a local LLM), or made it a leading candidate, and are now actually building it. The model plant’s headcount, time savings, hourly rate, costs and payback periods in this article are all placeholder figures set for this article. They are not industry averages, survey values, or vendor prices.

Why manufacturers are considering local LLM deployment

Use of open-weight models is spreading

In an Enterprise Technology Research (ETR) survey reported by the industry outlet Techstrong.ai on September 22, 2026 (200 respondents, conducted in September 2026; the article does not state the respondents’ countries or regions), open-weight models accounted for 34% of enterprise AI token usage, up from 23% a year earlier, and are expected to reach 41% in 12 months. 42% of respondents were using them in production (31% in the July survey), and 43% were at the pilot stage. As deployment platforms, 46% used self-managed cloud and 42% used on-premises. On the other hand, 62% cited “security and compliance reviews not yet completed” as a barrier to production, and 50% cited the cost of self-hosting.

An article by Constellation Research (September 26, 2026) discussing the same survey reports that 69% of production users cited cost reduction as a reason, and 60% had moved some workloads away from proprietary models. However, the same article states that 93% are also increasing their use of proprietary models. Rather than a migration to open-weight models, it is more natural to read the reality as “parallel use,” choosing between them depending on the use case.

What Japanese companies worry about most is information leakage

In a corporate survey across four countries presented in the 2026 White Paper on Information and Communications in Japan from the Ministry of Internal Affairs and Communications (MIC), the share of companies citing “security risks such as leakage of internal information” as a risk of concern in using generative AI was 46.4% in Japan, 36.9% in the United States, 31.7% in Germany and 42.7% in China. In Japan this was the most common item, followed by “problems with the accuracy of output” at 35.3% and “the possibility of infringing copyrights and other rights” at 31.3%. In the United States and China, accuracy of output (44.3% in the United States, 52.1% in China) was the most common, and in Germany it was copyright and other rights (44.7%). The same white paper states that, as a risk countermeasure, “developing company-wide policies or guidelines” was the most common in Japan at 41.1%, while the other three countries tended to show higher shares than Japan for specialist review at the introduction stage and for regular evaluation and verification after introduction.

This sense of “we are worried about leakage” is likely one reason Japanese-affiliated manufacturers consider local LLMs. However, as we will see later, placing the model locally does not make the risk of leakage disappear. The tendency shown in the white paper, with Japan’s share of post-introduction evaluation and verification efforts lower than the other three countries, carries over directly as a challenge for local LLMs as well.

Thai-language models and domestic compute infrastructure are increasing

In Thailand, regarding the national LLM “ThaiLLM”, jointly developed by the Ministry of Digital Economy and Society (MDES), the Big Data Institute (BDI), the Ministry of Higher Education, Science, Research and Innovation (MHESI), NECTEC and others, an NSTDA article dated April 1, 2026 explains that there are two models, 8B (speed-oriented) and 30B (high-performance), trained on more than 100 billion tokens of Thai-language data, and that a Playground, an API and model downloads are offered free of charge. SCB 10X’s Typhoon series has also released Thai-language models, OCR, translation and more (according to the company’s website).

On the infrastructure side, the Thailand Board of Investment (BOI) announced on July 8, 2026 that it had approved investments totaling USD 1.99 billion (THB 66.3 billion), including a project in which the Thai subsidiary of Japan’s Data Section will build high-performance GPU server infrastructure in Bangkok and Pathum Thani for USD 235.2 million (THB 7.8 billion). According to a December 2024 report, SIAM.AI CLOUD, which presents itself as a Thai-owned NVIDIA Cloud Partner, launched an H100 GPU cluster with the stated aim of maintaining data sovereignty (its operating scale as of 2026 was not confirmed in the research for this article). Beyond placing servers in-house, renting GPUs in a closed cloud inside Thailand is becoming a realistic option.

What is a local LLM? Open-weight, on-premises and closed cloud in Thailand

The term “local LLM” means different things to different people. Before starting an internal discussion, separating the following three keeps everyone on the same page.

In this article, both on-premises and domestic closed cloud are treated as “local LLM.” What they have in common is that your company decides which model, in which version, runs against which data, and prompts and answers are not passed to an external model provider. The difference lies in whether you buy or rent the GPUs, and who repairs the hardware when it fails.

A comparison of three architectures, SaaS controls, an API wrapper, and closed or self-hosted, by what they can protect and their five-year total cost, is in Secure Generative AI Environment | 3 Architectures Costed Over 5 Years. We will not repeat that comparison here, and move on to what goes into a self-hosted build.

First decide which tasks to keep local, and classify the data

The first thing to decide when considering a local LLM is neither the GPU nor the model, but which tasks to keep local.

The figure above shows the six-step order for deciding on a local LLM: (1) target tasks and data classification, (2) selecting model candidates, (3) estimating GPU and memory, (4) inference software and quantization, (5) evaluation and acceptance, (6) operations. Until an upper step is decided, the steps below cannot be estimated. For example, if the goal stays at “let every employee ask anything,” you cannot pin down how many people will feed in how long documents at the same time, and you cannot decide how many GPUs you need.

Example data classification

The table below shows, for each type of data, whether it can leave the company and an example of where to place it. Classifications change with internal information security rules and contracts with customers, so this is only an illustration of the approach.

Fill in the second column from the right for each department. The key is to narrow down the tasks you keep local. If even general writing is pulled onto the local LLM, the number of users grows, the GPUs run short, and performance may also fall behind SaaS. Starting with departments that handle confidential data daily, such as design, quality assurance, production control and purchasing, is the realistic approach.

Write “why it cannot leave” in one line

When deciding the classification, write the reason it “cannot leave” in one line. If you can write it like “Our contract with customer A prohibits storage on third-party servers” or “Formulations are classified as confidential by the head office,” you can reuse it as is when writing the RFP, designing the acceptance tests, and explaining to auditors. Data for which you cannot write a reason may in fact be fine on SaaS. Requirements and acceptance tests for preventing information leakage from generative AI itself are covered in detail in Generative AI Data Leak Prevention: RFP and Acceptance Tests for Thai Factories.

Choosing open-weight models: license, language, size and context length

Once the target tasks are decided, select model candidates. The list below shows, as “examples of candidates,” models for which official model cards or developer pages could be confirmed in the research for this article. It is not a recommendation. All performance figures and specifications are as stated by the developers (vendor claims).

A “—” in the table means the item was not stated within the scope confirmed in the research for this article.

Read licenses by their conditions, not their names

Even among “open” models, the conditions differ. Some models are listed under Apache 2.0 or MIT, while Llama 4 uses its own community license, under which products or services with more than 700 million monthly active users are said to require a request to Meta. Typhoon2.1-Gemma3-12B requires agreement to the OpenTyphoon terms of use in addition to the Gemma License. For Gemma 4, the official overview page states that commercial use is possible and that the Gemma 4 terms of use apply to the model, but check the details of the conditions on the terms of use page itself. For ThaiLLM, the license name could not be confirmed in the research for this article. Assume that once candidates are chosen, you will read the original text of the terms of use as of that time together with your legal department.

Assess languages both by the “supported list” and by “measurement on your own documents”

In plants where Japanese, Thai and Vietnamese are mixed, language handling is what decides the selection. The Llama 4 model card lists 12 supported languages including Thai and Vietnamese, while Japanese is not in the official list. As a Japanese-specialized option, the Swallow project, including Institute of Science Tokyo, released GPT-OSS Swallow 20B / 120B (v0.1 on February 20, 2026, and the MXFP4 version on May 28, 2026), and states that on JamC-QA, a benchmark testing Japan-related knowledge, the 20B improved by 13.0 points and the 120B by 11.4 points over the base gpt-oss (developer’s announcement). For Thai, there are ThaiLLM and Typhoon2.1-Gemma3-12B mentioned above.

However, supported-language lists and benchmarks do not guarantee that a model will work on your own drawing notes or quality records. You need to measure with your own combinations of languages, such as summarizing a work standard written in Thai into Japanese, or conveying the key points of a Japanese customer specification to the shop floor in Thai. How to build business evaluations for Thai is covered in Thai-Language Generative AI: RFP and Acceptance Testing Guide.

Size and context length

Larger models can be expected to be more accurate, but GPU memory and cost increase. Models such as gpt-oss, Qwen3-30B-A3B and Llama 4, which list “total parameters” and “active (activated) parameters” separately, use an architecture called MoE (mixture of experts). Even though only some of the parameters are used in each computation, all of the weights must be loaded into memory, so GPU memory is estimated using total parameters.

Context length is the length of context that can be read in at once. As the table shows, it ranges from 128K to 10M tokens, but the more of a long context you actually use, the more memory the KV cache requires. This is covered in the next section.

GPU sizing for LLM: how to calculate the memory you need

Weight memory is “number of parameters × bytes”

Estimating GPU memory starts with the weights. NVIDIA’s technical blog (November 17, 2023) gives the basic formula for weight memory as “number of parameters × bytes per parameter,” and as an example states that loading Llama 2 7B in 16-bit (FP16 / BF16) takes about 14GB. Bytes per parameter are 2 bytes for 16-bit, 1 byte for 8-bit and 0.5 bytes for 4-bit.

Using this formula, the approximate weight memory by model size and precision is as follows (approximation with 1GB = 1 billion bytes; weights only).

This table does not include the KV cache, the inference software’s runtime memory, or quantization metadata. The GPU memory you actually need will be larger. How much larger depends on the number of concurrent connections, the length of the context and the inference software.

Let us also look at the values published by developers. For Gemma 4, Google shows in an official table the approximate memory required just to load the weights (vendor claim).

Google notes that this table shows only the memory required to load the static model weights, and does not include the additional memory needed for software or the context window (KV cache). It further states that memory consumption grows dynamically with the total number of tokens in the prompt and the generated response. Values from the simple formula and values in a developer’s table may not match depending on the model, so once candidates are chosen, check against the developer’s table or the model card.

Other model cards also give guidance. The model card for Mistral-Small-3.2-24B-Instruct-2506 states that running it on GPU in bf16 or fp16 requires about 55GB of GPU RAM. The gpt-oss model card states that by applying MXFP4 quantization to the MoE weights, 120b fits on a single 80GB GPU (for example NVIDIA H100 or AMD MI300X), and 20b runs within 16GB of memory. Llama 4 Scout is said to fit on a single H100 GPU with on-the-fly int4 quantization. All of these are developer claims, and conditions such as the number of concurrent connections and context length need to be checked separately.

The KV cache grows with concurrent connections and context length

After the weights, the next major factor is the KV cache. This is the area that holds the context of conversations and documents in GPU memory, and the NVIDIA technical blog mentioned above gives the total with the following formula.

KV cache = batch size × sequence length × 2 × number of layers × hidden size × bytes of FP16

In the same blog’s example, running Llama 2 7B with batch size 1, sequence length 4096, 32 layers, hidden size 4096 and FP16 gives a KV cache of about 2GB. As the formula shows, it grows in proportion to batch size (the number of requests processed at the same time) and sequence length (the length of the context). In uses where quality records or drawing descriptions are fed in at length through RAG and several people use the system at the same time, it may not stay at a fraction of the weights.

There is one caveat. This formula uses hidden size, but for models that adopt GQA (grouped-query attention), the KV cache is calculated with number of KV heads × head dimension. For example, Qwen3-30B-A3B-Instruct-2507 mentioned above has 4 KV heads, so applying this formula as is gives an overestimate. Check the exact value with the configuration of the model you adopt and actual measurements in the inference software.

There are also settings on the inference software side that reduce the KV cache. According to Ollama’s FAQ, the KV cache quantization setting (OLLAMA_KV_CACHE_TYPE) defaults to f16, and setting it to q8_0 uses about half the memory of f16, and q4_0 about one quarter (developer’s statement). As with weight quantization, the trade-off is that you need to check the impact on accuracy in exchange for the memory savings.

Official specifications of GPU products

The table below lists memory capacity and other figures from the official GPU product pages. All are as stated by the manufacturer. Prices are not listed on the official pages, so this article does not cover them.

The order for choosing is: (1) narrow down the combination of candidate model and precision using the weight table, (2) allow headroom for the KV cache based on concurrent connections and context length, (3) decide whether that fits on one card or is split across several. Power consumption and cooling, free rack space and power supply capacity are constraints that are easy to overlook in a plant’s server room. Confirm the conditions of the installation site with the facilities department before choosing the GPU.

Inference software and quantization: what changes depending on your choice

Between the model and the GPU sits the inference software (inference engine, serving software). Your choice changes how well concurrent connections are handled, which quantization formats are supported and which hardware it runs on. All of the following are as stated by the developers or maintainers.

A “—” in the table means the item was not stated within the scope confirmed in the research for this article.

Treat small-group trials and departmental production separately

In the early stage of a PoC, trying things quickly with Ollama or llama.cpp is reasonable. However, Ollama binds only to localhost by default, and exposing it on the network requires explicitly setting OLLAMA_HOST. The default number of parallel requests per model is also 1. If a department is to share it as an internal server, you need to separately design the concurrency settings and authentication and access control. The default context window is also stated as 4096 tokens in the FAQ at the time of retrieval, so for uses that feed in long documents, the setting needs to be checked (this default is a setting that has changed between versions).

For production where several people in a department use the system at the same time, serving software with continuous batching and KV cache management, such as vLLM or SGLang, becomes a candidate. Just as the model card for Qwen3-30B-A3B-Instruct-2507 lists vLLM 0.8.5 or later and SGLang 0.4.6.post1 or later as recommended serving options, some models specify recommended inference software and versions. Choose the model and the inference software as a combination.

Quantization reduces memory, but verify accuracy yourself

Quantization is a technique that shrinks weights from 16-bit to 8-bit or 4-bit to reduce memory. As the weight table showed, going to 4-bit brings it to one quarter of 16-bit. However, how much accuracy drops depends on the model, the quantization method and the task. For uses where a small degradation can lead to business errors, such as reading numbers, transcribing part numbers or specification values, or Thai proper nouns, include comparing the 16-bit version and the quantized version on the same evaluation set in the acceptance tests.

The supported quantization formats also vary by GPU generation. In vLLM’s quantization compatibility table, AWQ is supported on NVIDIA GPUs from Turing onward (Volta is not covered) among others, and FP8 (W8A8) via llm-compressor is supported on the Ada and Hopper generations and AMD GPUs, with a note that the compatibility table may change. Check the combination of GPU and quantization format against the inference software’s latest compatibility table before placing an order.

Overall architecture of a self-hosted LLM server: authentication, RAG, logs and monitoring

A local LLM cannot be used for work just by loading a model onto a GPU server. When built as an internal server, the overall architecture is roughly divided into the following layers.

• User entry point: an internal chat screen, or integration into existing document management systems or portals. Link it to employee accounts through single sign-on.
• Authentication and access control: match who can search which documents to the permissions of the existing documents. One of the big pitfalls of local LLMs is that people without permission can read quality records or cost data they should not see, via RAG.
• RAG (internal document search): the layer that searches drawing notes, customer specifications, quality records and internal rules and passes them to the LLM. Document version control, mixed languages, and the handling of tables and figures determine accuracy. Building and evaluating RAG as a whole is covered in Enterprise RAG Implementation: Architecture, Cost and Evaluation Guide, and measures against answering from outdated document versions in Factory RAG Document Version Control: Implementation and Acceptance.
• Inference server: the inference software and GPUs from the previous section. If you use multiple models for different purposes, switching between them is also handled here.
• Logs and monitoring: a record of who asked what and when, and what was returned on the basis of which documents. Monitoring of GPU utilization, response time and errors. Because the logs themselves contain confidential information, also decide where logs are stored and who can view them.
• Boundaries with existing systems: when letting the LLM reference MES or ERP data, decide whether it is read-only or writes are allowed, and up to which fields are passed. For a design that connects plant MES and ERP to AI agents safely, see MCP Server Development: Safe MES/ERP Links for Factory AI Agents.

When drawing the architecture diagram, write in “which layer sits in which country, on whose equipment.” Even if only the inference server is placed in-house, if the RAG search platform or the logs sit on an overseas SaaS, the data leaves the company.

Security: risks that do not disappear even when local

Placing the model locally has the advantage that prompts and answers are not passed to an external model provider. However, many of the risks specific to LLM applications remain regardless of where the model sits. Below we organize the items that deserve particular attention for local LLMs, drawn from the “OWASP Top 10 for LLM Applications 2025” published in March 2025 by OWASP (a nonprofit standards community) and “NIST AI 600-1,” the risk profile for generative AI published by NIST in July 2024.

Check where model files come from

An issue specific to local LLMs is the safety of model files. Hugging Face’s documentation explains that pickle, PyTorch’s default weight format, carries the danger that arbitrary code can be executed when loading. As countermeasures it lists obtaining files from trusted organizations, checking signed commits (although a signature guarantees origin, not safety), and alternative formats such as safetensors. safetensors is described as a format that, unlike pickle, stores tensors safely. The Hugging Face Hub scans uploaded files, but states explicitly that this is not 100% foolproof and that checking whether a file is safe is the user’s responsibility.

As an internal operating rule, the basic practice is to record which file, from which repository and in which version, was brought in and who approved it. Follow the same procedure every time the model is updated.

Do not explain it as “safe because it is local”

When explaining to management or customers, avoid stating flatly that “information leakage will not happen because it is local.” That data is not passed to an external model provider is a separate matter from internal permission overreach or misuse of output not happening. NIST AI 600-1 also lists, among risks unique to or amplified by generative AI, confabulation, data privacy, information security, and the opaque integration of third-party components, among others. In Japan, version 1.2 of the “AI Guidelines for Business” was published in March 2026, along with checklists and worksheets. These can be referred to as a framework for internal risk assessment.

Cautions when proceeding in Thailand and ASEAN

Cross-border transfer under the PDPA

Under Thailand’s Personal Data Protection Act (PDPA), two notifications from the Personal Data Protection Committee (PDPC) on cross-border transfer, based on Sections 28 and 29, were promulgated on December 25, 2023 and took effect on March 24, 2024. According to a law firm’s commentary, Section 28 requires that the destination country or international organization have an adequate level of protection, and Section 29 provides for transfers based on appropriate safeguards such as intra-group binding corporate rules (BCR) or standard contractual clauses. Sending prompts containing personal data to an overseas SaaS LLM or API may fall under such a cross-border transfer. Check with your legal department or outside experts whether it does. A local LLM is one option when you want to keep tasks involving personal data within the country.

Status of AI legislation

In Thailand, the Electronic Transactions Development Agency (ETDA) published a draft AI law on July 9, 2026 and invited public comment until August 14, 2026. According to a law firm’s commentary, the draft takes a risk-based framework in which sectoral regulators, rather than a single regulator, set the requirements. As of October 10, 2026 it is at the draft stage, and there is also a view that it could take 6 to 24 months from in-principle cabinet approval to enactment and entry into force.

In Vietnam, the AI law was adopted in December 2025 and took effect on March 1, 2026. According to the Vietnamese government portal, it has 35 articles in total. It is explained as adopting a three-tier risk classification of high, medium and low, with transitional provisions for existing AI systems (18 months for healthcare, education and finance, 12 months for others).

How regulatory developments in Thailand and Vietnam affect the self-hosting decision is touched on in LLM Deployment Comes Down to How You Own the Foundation. Here, on the premise that the regulatory situation differs by country and is changing, we limit ourselves to recommending an architecture that can keep records of logs and evaluations.

Handling Thai and Vietnamese

At Thai sites, Japanese, Thai and English are often mixed in a single document. For companies with Vietnamese sites, Vietnamese is added as well. As seen in the model selection section, language support differs by model. Compare Thai-specialized models (ThaiLLM, Typhoon), a Japanese-specialized model (GPT-OSS Swallow) and multilingual models on your own combinations of languages. The problem of Vietnamese accuracy breaking down in preprocessing is covered in Vietnamese Generative AI Accuracy — What Breaks Is Preprocessing, Not the Model.

Model calculation of costs and benefits (placeholder figures)

From here on is a model calculation to show the “structure” of the estimate. All of the figures below are placeholder figures set for this article. They are not industry averages, figures from research firms, or vendor prices. Replace them with your own figures.

Assumptions for the model plant

Assume one site of a Japanese-affiliated parts manufacturer in Thailand. 120 people in design, quality assurance, production control and purchasing use a local LLM for internal document search (RAG) and for drafting and summarizing documents, covering drawing notes, customer specifications, quality records and internal rules. The currency is the Thai baht (THB).

Main case (h = 3 hours/month)

As a general formula, this becomes:

Annual net benefit = 576,000 × h − 780,000

576,000 is the increase in the annual monetary value for each additional hour of monthly time saved per person (120 × 400 × 12).

How results move when time saved changes

Let us change only h, the monthly time saved per person.

The h at which annual net benefit becomes 0 is, from 576,000 × h = 780,000, h ≈ 1.35 hours/month. Below this, the monetary value of time saved does not even cover the annual operating cost. In addition, to recover the initial cost within the 5-year hardware service life, an annual net benefit of at least 480,000 THB (2,400,000 ÷ 5) is required, and from 576,000 × h ≥ 1,260,000 the condition is h ≈ 2.19 hours/month or more. At h = 2 hours/month, the annual net benefit is positive, but payback takes about 6.5 years, exceeding the assumed service life of 5 years.

In other words, under these placeholder assumptions, around 2 hours per person per month is the dividing line, and below about 1.35 hours per month the annual costs are not covered. Most of the cost is fixed: GPU servers, maintenance and operations staff. If there are few users, or usage time is short, the investment does not pay back.

Look at cash savings and time savings separately

The monetary value of time saved does not become a cash saving unless the freed-up time is redirected to other work. What goes out as cash is the initial cost I and the annual operating cost L. Whether the freed-up time can be put toward customer response, quality improvement or the next development project changes the meaning of the investment significantly.

What this calculation does not include

The following items are not given a monetary value because they vary greatly with conditions.

• Savings on the portion of SaaS or API usage fees moved to the local LLM (because it depends on whether there are existing contracts)
• Benefits for customer audits and contracts from not sending confidential data outside the company
• Avoidance of information leakage incidents
• GPU replacement costs (from year 6 onward)

Evaluate these separately. The approach of managing generative AI cost per unit of outcome is covered in AI FinOps: Managing Generative AI Cost by Verified Outcome.

What the calculation tells us

The time saved h must not be set by guesswork. In the PoC, measure the working time for the target tasks before and after introduction, and replace h with that value. On that basis, what can be said is that, rather than starting “for all employees,” the order is to start with “departments that handle confidential data daily,” measure the time saved in the PoC, and then decide the production GPU configuration.

For departments with few users, there is also the option of first renting GPUs monthly in a closed cloud inside Thailand to run the PoC, and deciding on a hardware purchase after measurement. Cloud GPU prices vary by provider and contract terms, so confirm them through quotations.

What to include in an RFP for a local LLM

When writing an RFP (request for proposal) for procuring GPU servers or outsourcing the build and operation, include the following items. An RFP that states only “the model and the GPU model number” does not let you compare proposals.

Items 4 and 6 in particular are what distinguish the quality of proposals. Proposals that cannot give a breakdown of the memory basis, or that state the acceptance criteria only as “high scores on general benchmarks,” risk falling short once operations begin.

What to check in the PoC: five cases to test on purpose

In a PoC, it is not enough to confirm that the system works on questions that go well. Check whether wrong answers can be caught, whether permissions hold and whether response times hold up under adversarial conditions.

KPIs to measure

• Working time for the target tasks (measured before and after introduction; the value that replaces h in the model calculation)
• Pass rate on the evaluation set (by language combination, and separately for 16-bit and quantized versions)
• Response time under concurrent connections
• Number of cases where information leaked on questions that exceed permissions (must be 0)
• Effort spent on re-evaluation

Case (5), swapping the model, is often overlooked but important. New versions of open-weight models or different models may be released, and it is quite conceivable that you will want to change the model after operations begin. Without a mechanism to rerun the evaluation set each time, you are left with two choices: keep using the old model, or switch without evaluation. Confirm the re-evaluation procedure and the time it takes at the PoC stage. Criteria for deciding to move from PoC to production, and the design of acceptance and operations handover, are covered in AI PoC Exit Criteria for Production, Acceptance and Handover.

Frequently asked questions

What is a local LLM?

It means running an open-weight model (a model whose weights are published and can be downloaded and run) in an environment your company manages, such as your own servers or a closed cloud inside Thailand. What distinguishes it from SaaS generative AI is that your company decides which model, in which version, runs against which data, and prompts and answers are not passed to an external model provider.

What GPU do you need for a local LLM?

It depends on the model’s size and precision, the number of concurrent connections and the length of the context. Weight memory can be approximated as “number of parameters × bytes per parameter,” and as the table in the article shows, for a 32B model, for example, the guide for weights alone is 64GB at 16-bit and 16GB at 4-bit. The KV cache and the inference software’s runtime memory are added to this. Rather than choosing from GPU product names, decide the target tasks and the number of concurrent connections first, and work backward to the memory you need.

How much does an on-premises LLM cost?

GPU prices are not listed on the official pages and vary greatly with configuration and quotations. In terms of cost structure, obtain quotations split into initial costs (hardware, build, evaluation set creation, PoC) and annual costs (power and installation, maintenance and external support, internal operations staff). In the model calculation in this article (placeholder figures), because most of the cost is fixed, the result was that below about 1.35 hours of monthly time saved per person the annual costs are not covered, and payback within 5 years requires about 2.19 hours or more.

Can a local LLM be used in Japanese and Thai?

It depends on the model. For example, the Llama 4 model card lists Thai and Vietnamese as supported languages, but Japanese is not in the official list. There are also models such as GPT-OSS Swallow, specialized for Japanese, and ThaiLLM and Typhoon2.1-Gemma3-12B for Thai. However, supported-language lists and benchmarks do not guarantee accuracy on your own documents. Measure with an evaluation set built from your own combinations of languages.

Does a local LLM prevent information leakage?

It has an advantage in that prompts and answers are not passed to an external model provider, but the risk of leakage does not go away. Risks remain even locally, such as employees without permission being able to read other departments’ confidential documents via RAG, bringing in model files of unknown origin, and logs themselves containing confidential information. Checking document permissions, managing the sources of model files, and designing log storage and viewing permissions are necessary.

How should you choose open-weight models?

Narrow down candidates on four points: license and terms-of-use conditions, support for the languages your company uses, model size (whether it fits in GPU memory) and context length, and finally compare them on an evaluation set built from your own documents. Read licenses by their conditions rather than by the word “open,” and check the original text of the terms of use with your legal department before adoption.

Summary

If local LLM deployment starts by comparing GPU model numbers and model benchmarks, it can end with an expensive server that nobody uses. First, decide which tasks to keep local based on data classification, and leave tasks whose data can leave the company on SaaS or APIs. Next, build an evaluation set and pass criteria from your own documents and your own combinations of languages. Then decide who is responsible for model updates, managing the sources of model files, logs, and rollback on failure. The GPU and the model come after these three are settled, working backward to the minimum configuration required by adding up the weights, the KV cache and headroom for concurrent connections.

In the model calculation (placeholder figures), because most of the cost is fixed, below about 1.35 hours of monthly time saved per person the annual costs were not covered, and payback within 5 years required about 2.19 hours or more. That is exactly why you should start with departments that handle confidential data daily, measure the time saved in the PoC, and then decide the production GPU configuration. While users are few, there is also the path of renting GPUs in a domestic closed cloud to test.

At TOMAS TECH, we help plants in Thailand organize internal documents, use generative AI and connect it with existing systems, tailored to their operations. You can consult us even before the model or GPU has been decided, for example at the stage of narrowing down target tasks or organizing data classification. Please contact us through this form.

References

• Hugging Face: openai/gpt-oss-120b model card
• Ollama Library: gpt-oss
• Hugging Face: Qwen/Qwen3-30B-A3B-Instruct-2507 model card
• Hugging Face: meta-llama/Llama-4-Scout-17B-16E-Instruct model card
• Google AI for Developers: Gemma models overview
• Hugging Face: deepseek-ai/DeepSeek-V3.2 model card
• Hugging Face: mistralai/Mistral-Small-3.2-24B-Instruct-2506 model card
• Swallow LLM project: GPT-OSS Swallow
• Hugging Face: scb10x/typhoon2.1-gemma3-12b model card
• vLLM documentation
• vLLM documentation: Quantization
• GitHub: ggml-org/llama.cpp
• Ollama Docs: FAQ
• SGLang Documentation
• NVIDIA: L4 Tensor Core GPU
• NVIDIA: L40S GPU
• NVIDIA: H200 Tensor Core GPU
• NVIDIA: RTX PRO 6000 Blackwell Server Edition
• NVIDIA: DGX Spark
• NVIDIA Technical Blog: Mastering LLM Techniques: Inference Optimization (November 17, 2023)
• OWASP GenAI Security Project: OWASP Top 10 for LLM Applications 2025
• NIST: AI 600-1 Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
• Hugging Face Hub docs: Pickle Scanning
• Hugging Face docs: Safetensors
• MIC (Japan): 2026 White Paper on Information and Communications, Figure 1-2-1-8 (CSV)
• MIC (Japan): 2026 White Paper on Information and Communications, Part 1, Chapter 2
• Techstrong.ai: Open-Weight AI Models Expected to Capture 41% of Enterprise Token Usage (September 22, 2026)
• Constellation Research: Evidence Shows Enterprises Use of Open Weight Models is Mainstream (September 26, 2026)
• Tilleke & Gibbins: Thailand Unveils Regulations for Cross-Border Personal Data Transfer (January 4, 2024)
• Rajah & Tann Asia: Thailand’s Proposed AI Law: Moving Towards a Risk-Based Regulatory Framework (July 28, 2026)
• Baker McKenzie: Vietnam: Artificial Intelligence Law – Foundation and Outlook (February 24, 2026)
• Vietnam Government News: First-ever law on artificial intelligence approved
• NSTDA: ThaiLLM (April 1, 2026)
• BOI: Thailand Approves $1.99 Billion in New Investment, Led by AI and Advanced Electronics (July 8, 2026)
• Kaohoon International: SIAM.AI CLOUD Launches Thailand’s AI Infrastructure (December 4, 2024)
• SCB 10X: OpenTyphoon
• MIC (Japan): AI Guidelines for Business