Self-Hosted LLM or Cloud API: Choosing a Secure AI Model for Business Data

After trying AI assistants to summarize documents, answer customer questions, or read invoices, many companies arrive at the same question: is business data safe when it is sent to an AI service in the cloud? Some IT teams then propose running a large language model (LLM) themselves on company servers. Others see that as expensive and troublesome, and prefer to use the API of a major model provider.

Both can be right, depending on the type of data, usage volume, team capability, and the rules that apply in your industry. This article explains the difference between a self-hosted LLM and a cloud API, how to calculate the cost honestly, the risks that are often overlooked, and the mixed pattern that most mid-sized companies now use.

Summary

• A cloud API gives access to the most capable models with a small upfront cost, but data leaves the company's infrastructure
• A self-hosted LLM keeps data inside, but the company carries the GPU servers, security, model updates, and specialist staff
• Enterprise plans from major providers generally commit not to use data for training, which is enough for many commercial needs
• Self-hosting usually only saves money at high, steady usage volumes, or when rules forbid data from leaving
• A mixed pattern, routing requests by data sensitivity, is often the most sensible middle ground

What is the difference between self-hosted and a cloud API?

With a cloud API, your application sends text or documents to the model provider's servers and receives an answer. You pay per token, the word fragments that are processed. There are no servers to maintain, and the provider updates the model. Examples are model services from the major AI providers, used directly or through cloud platforms such as AWS, Google Cloud, or Azure.

With self-hosting, the company downloads an open-weight model, meaning a model whose weights may be run independently, and runs it on servers with GPUs, either in its own data center or in a private cloud. Data never leaves the environment you control. In exchange, all operational concerns become your responsibility: capacity, security, monitoring, and model updates.

Between the two is a third option: models run by a cloud provider in a region and private network belonging to the customer, such as a private endpoint in a specific region. This option is often enough to meet data location requirements without having to manage GPUs yourself.

On privacy: what actually happens to the data?

The main worry is usually that data will be used to train the model or read by others. For business and enterprise plans, major providers generally state in the contract that customer data is not used for training, set data retention periods, and provide data processing agreements. Still, the data passes through their systems. For most commercial uses, such as summarizing emails, writing product descriptions, or answering questions from public documents, this level of protection is adequate.

The situation is different for highly sensitive or tightly regulated data: medical records, banking customer data, legal documents in active litigation, or data that regulations require to stay in a particular jurisdiction. In Indonesia, personal data protection requires data controllers to understand where data is sent and on what basis. For data like this, self-hosting or a private endpoint in the right region is often a requirement, not a choice.

Calculating the cost honestly

Cost comparisons that circulate often only compare GPU rental prices with API bills. The cost of self-hosting is much broader. To be fair, count all of the following components over at least two to three years.

• GPU servers, bought or rented, including spare capacity so the service does not stop when one server has a problem
• Electricity, cooling, and data center space if run on your own premises
• Specialist staff to set up, monitor, and secure the model service
• Time to test and update the model when better versions are released
• Idle capacity outside peak hours, because GPUs are paid for even when unused
• The cost of integration, logging, and access control, some of which already comes with an API

API costs, on the other hand, rise with usage. At small to medium volumes, such as hundreds or thousands of documents per day, an API is almost always cheaper. Self-hosting starts to make sense when token volume is very large and steady throughout the day, so GPUs are actually kept busy. The break-even point differs in every case, because it depends on model size, GPU prices, and usage patterns. To keep bills under control on either side, the same principles as controlling cloud costs apply: measure usage per feature, set limits, and review regularly.

Model size affects everything

Open-weight models come in many sizes. Small models can run on a single GPU and are good enough for narrow tasks such as classifying emails, extracting data from forms, or answering questions from internal documents. Large models need several GPUs and approach the capability of commercial models, but cost far more. A common mistake is to go straight for the largest model, when the task could be handled by a small model that is well tuned or well instructed.

Quality and capability

Leading commercial models still excel at tasks that demand long reasoning, high-quality writing, understanding of mixed languages, and complex tool use. Open-weight models keep improving, and for many routine business tasks the gap is now small. The key is to measure, not guess. Prepare a set of real examples from your work, such as a hundred customer emails or fifty contract documents, then compare the output of several models against the correct answers.

Language matters too. For Indonesian customers, test how well a model understands everyday language, abbreviations, and a mix of Indonesian and English. For regional markets, also test Mandarin or the other languages you serve. A model that is excellent in English is not necessarily as good in other languages.

Risks that are often overlooked

Self-hosting is not automatically safer. A model server exposed to the internet without authentication, logs that store whole conversations unencrypted, or a model that is never updated can all become new holes. Risks specific to AI applications, such as prompt injection and data leakage, remain wherever the model runs.

• Uncontrolled access: any internal application can call the model without limits and without a trace
• Logs containing sensitive data kept longer than necessary
• Dependence on a single specialist who understands the model server configuration
• Open-weight model licenses that restrict certain commercial uses
• Dependence on a single API provider with no fallback plan if prices or terms change

That is why this technical decision should follow an agreed AI usage policy and AI governance: what data may be sent where, who may use it, and how output is checked.

The mixed pattern: route by sensitivity

Many companies end up choosing neither exclusively, and instead place an intermediary layer, a gateway, between applications and models. This layer inspects every request and routes it to the appropriate model.

1. 01Classify data into several levels, for example public, internal, confidential, and highly confidential
2. 02Decide which models may receive each level: a cloud API for public and internal, a private endpoint or self-hosted model for confidential
3. 03Mask personal data such as names, phone numbers, and ID numbers before sending to external models where possible
4. 04Log every request with enough metadata for audit, without keeping sensitive content longer than needed
5. 05Set cost limits and quotas per application, plus a fallback model if one service is down

With this pattern, the website chatbot uses the most capable cloud model, while an assistant that reads employee contracts or payroll data runs on a model you manage yourself. The same architecture also makes it easy to switch models later without rewriting applications. If your AI needs to pull data from the ERP or CRM, standards such as the Model Context Protocol help govern that access in a controlled way.

Steps to decide for your company

1. 01List the planned AI use cases and the types of data each will process
2. 02Identify which cases are bound by regulations or customer contracts that restrict data location
3. 03Estimate monthly usage volume and its hourly pattern
4. 04Test two or three models, commercial and open-weight, with real sample data
5. 05Calculate the three-year total cost of each option, including specialist staff
6. 06Start with the simplest option that meets the requirements, then reassess every six months

For most companies just starting out, the practical step is to use an API on a business plan for non-sensitive data, while making sure of data readiness and governance. Self-hosting is considered once there is a use case with proven value, large volume, or data that genuinely must not leave.

Frequently asked questions

Can a self-hosted LLM run without a GPU? Very small models can run on a CPU, but slowly, and only suit low volume or trials. For a service used by many people, a GPU is practically required.

Is data sent through an API guaranteed to be safe? Nothing is guaranteed. Business plans give far better contractual assurances than consumer plans, but you still need to limit the data sent, encrypt connections, and review provider terms regularly.

Can an open-weight model be retrained with company data? Yes, through fine-tuning. For most needs, however, connecting the model to company documents through a RAG approach is cheaper, easier to update, and easier to audit.

How long does it take to set up a self-hosted LLM? A trial with one small model can be done in days. A reliable, secure, monitored production service usually takes several weeks to a few months, depending on security and integration needs.

Conclusion

Choosing between a self-hosted LLM and a cloud API is not about which is more modern, but about data, volume, team capability, and the rules that apply. Cloud APIs win on convenience and model capability; self-hosting wins on control over data at high volume. For many companies the answer is both, with an intermediary layer that routes each request according to its data sensitivity. Start with a clear use case, measure results with your own data, and let the numbers decide.