Sizing On-Premises AI: User Numbers Are Only the Starting Point

  • Post category:Sustainability
  • Reading time:7 mins read

Sizing on-premises AI requires more than an employee count and a model name. What matters is which tasks run concurrently, the length of inputs and the quality the business team needs. This guide shows how companies, public institutions and critical infrastructure operators can turn these needs into testable requirements for a local AI solution.

In this article

Start with the use case

Internal knowledge search, long-document summaries and an assistant for technical workflows have different requirements. Before choosing hardware, describe two or three concrete tasks. For each, identify typical inputs, the expected output and criteria the business team can use to assess it.

For knowledge search, the aim might be to give employees a clear answer to a technical question with a verifiable reference in approved documents. For document analysis, complete and accurate extraction of all required information may be decisive. Fluent text alone is not sufficient evidence of quality in either case.

Also record which information the application must not process. A bounded pilot is easier to evaluate when data sources, user roles and permitted actions are known beforehand. Adding tasks then becomes a deliberate decision, rather than an unintended consequence of expanding permissions.

Assess request volumes and concurrency separately

Active users, daily request counts and typical input and output lengths help estimate volume. Technical sizing also depends on when requests arrive. Daily demand may be evenly distributed or concentrated in short periods. Describe at least normal operation and a credible peak scenario.

A model example illustrates the difference: 50 active users making 20 requests each generate 1,000 requests a day. Assuming 2,000 input and 500 output tokens per request gives two million input tokens and 500,000 output tokens. Spread evenly across eight hours, output averages approximately 17.36 tokens per second. This is only the calculated daily average for the whole system.

Several long requests arriving together can create much higher short-term demand. The average alone cannot establish the required GPU count or an acceptable waiting time. The Edge AI requirements calculator helps make assumptions transparent. Reliable hardware selection then requires measurements using the proposed model and workload.

Performance metrics that matter in a pilot

Useful metrics for interactive applications include time to first token, time to complete the answer and throughput under concurrent requests. They answer different questions. A system may process many tokens overall while individual users still wait a long time. Definitions and measurement methods must match across comparisons. Source: NVIDIA, LLM benchmarking metrics, updated 20 July 2026.

For acceptance testing, agree concrete requirements with the business team. An interactive assistant may need a different response time from overnight processing. Make slow individual cases visible too: a daily average does not reveal how frequently users experience unusually long waits.

Document the model, memory format, software version, input and output lengths and concurrent load for every test. Also note whether the model was already loaded and whether cached results were used. Comparisons are most useful when another person can repeat them under the same conditions.

Model weights are only part of memory demand

A preliminary lower bound for model-weight memory can be estimated from parameter count and bits per parameter. A hypothetical 32-billion-parameter model at four bits per parameter needs 16 billion bytes, or approximately 14.9 GiB, for weights alone. This excludes metadata and working memory needed for execution.

This expressly does not mean a 16 GB graphics card is sufficient for that model and your use case. Context management, runtime and other components require additional resources. Format and implementation also matter. Input and output lengths affect inference and the memory needed for the KV cache. Source: NVIDIA, LLM Inference Benchmarking: Fundamental Concepts.

We recommend assessing memory requirements and task quality together. Smaller weight representations can be technically attractive, but the decisive question is whether the required tasks are performed reliably. A larger model should not be preferred solely for its parameter count either. Results on your representative tasks are what count.

Connect company knowledge while preserving permissions

For a knowledge assistant, select the document collection deliberately. First check that documents are current, clearly identified and approved for the intended users. Ambiguous versions and contradictory contents are data issues. Adding an AI layer does not automatically resolve them.

When search is followed by a generated answer, commonly called retrieval-augmented generation or RAG, the pilot should also examine retrieved sources. Is the right passage used? Is it sufficient for the question? Can an authorised person open it? Does the answer include claims that the source does not support?

Include negative access tests explicitly: a user without permission must not receive restricted information through search results or generated answers. Test indirect follow-up questions too. Task quality and access protection belong in the same acceptance criteria.

Local operation needs controlled data flows

An on-site server is an important architectural component. To assess data control, also map all data flows: which services are called, where logs and backups reside, how updates work, what access administrators and maintenance providers have, and whether external research functions or other interfaces are used.

Germany’s BSI highlights the importance of access permissions in AI applications: connected resources extend a system’s capabilities and require deliberate consideration. Source: BSI, Using AI applications securely.

For organisations with high protection requirements, start with the permissions the use case needs and assess every extension deliberately. A blanket claim that “local means completely secure” is too imprecise for investment decisions. Clear architecture, defined roles and tested operating procedures provide a stronger basis.

Authorise autonomous workflows in stages

If AI is to perform actions, distinguish proposal, review and execution. An assistant can first prepare a work order. Only after quality and permissions have been assessed should you decide which steps may run automatically. Provide suitable human approvals for consequential changes.

Prepare for errors too: what happens with incomplete input or contradictory sources? Can an action be stopped? Can you trace which request led to a change? A convincing pilot demonstrates controlled handling of these situations as well as the successful standard case.

Assessment against KRITIS, NIS2 or other organisational requirements must be separate. This article does not infer blanket compliance from local installation. Protection needs, responsibilities and required evidence are defined specifically in the project.

Prepare the pilot and operations together

For a testable pilot, we recommend this sequence:

  • Define tasks: Assemble representative questions and documents and describe expected results.
  • Assess quality: Check accuracy, completeness, source support and handling of missing information.
  • Measure load: Test normal operation, concurrency and relevant peaks with documented settings.
  • Check access: Test permitted and restricted information sources and intended actions.
  • Exercise operations: Prepare recovery, backups, restoration and a controlled update.

An acceptance report should record known limitations as well as results. If only a particular document type is processed reliably, make that boundary visible. The next expansion can then target it. This creates a verifiable path from the first use case to a production system.

Plan Edge AI and data centre infrastructure together

The selected AI platform must be operable at the intended site. Check power supply, cooling, space, networking and maintenance access together. Define reserves against a concrete expansion plan. A technical pilot at a desk does not replace this assessment for continuous operation.

High Knowledge connects sovereign AI planning with the infrastructure it needs. Learn more about applications, data control and implementation under Sovereign AI and Edge AI. If the project needs additional data centre capacity, a feasibility study provides a basis for comparing suitable options.

Editorial review: 8 September 2026. Numerical examples are model assumptions, not hardware or performance guarantees. Linked primary sources were checked on that date.