A company is not a playground for AI – why AI usage patterns from the consumer market are dangerous for businesses
SAVANT-AI is a research project in the inLABs programme. The capabilities described are research goals, not an available product. Project status: inLABs
Everyone is enthusiastic about generative models, but at the same time, they are increasingly fed up with their own implementations. On the one hand, algorithms write texts, analyze data, design campaigns, and “help” HR; on the other, customers see the side effects: hallucinations, automation without accountability, leakage of know-how to other people’s models, and growing dependence on the cloud. This text is about what happens when we leave AI to its own devices — and why serious organizations need sovereign, cognitive infrastructure, not just another “friendly chatbot.”
Privately, each of us has complete freedom to test artificial intelligence. However, when it comes to business, critical systems, law, defense, or human health, the rules dictated by global tech giants must give way to legal regulations and internal organizational procedures.
Just as, in traditional document circulation, the word processor’s manufacturer does not decide how contracts are stored, control over knowledge in AI models must remain in the hands of officials and entrepreneurs. Information sovereignty requires the user, not the tool provider, to set the rules of the game according to their own interests.
Implementing such a control framework is not only a matter of choice, but above all a duty of managers. In the era of digital transformation, ensuring data security, legal compliance, and protection of information resources is the foundation of responsible leadership and is essential for maintaining the continuity of institutions and companies.
In the SAVANT-AI project we research how to restore this control with tools. Its goal is to support businesses and officials in managing knowledge securely.
AI that makes things up better than humans
In theory, language models were supposed to be smarter than humans, but in practice, they turned out to be primarily shamelessly confident. When they don’t know something, they don’t remain silent, but add the missing pieces to make it sound “like the truth.”
- An algorithm that fabricates documents, research, or rulings does so consistently, logically, and with complete conviction. It does not say “I don’t know” or signal any doubts.
- People look at the polished form and elegant tone of the expert and instinctively think, “If it sounds professional, it must be true.”
At the level of a private conversation, this results in an embarrassing presentation at best. At the level of a bank, energy company, or government institution, it can mean real financial losses, wrong management decisions, and even violations of the law.
From the SAVANT-AI project’s perspective, an LLM is not “artificial wisdom,” but a very capable intern. An intern can prepare a sketch, a proposal, or a draft analysis — but does not make decisions on their own. They need a structure above them: an orchestrator, agents, a fact engine, policies, and audits that bring every answer down to earth, to specific sources and logs.
Bad prompt, bad decision — the person between the keyboard and the chair
Most AI dramas do not start with the model, but with the prompts.
- “Do something for me,” “write a text,” “get the report” — these are typical commands that a chatbot is left alone with. No goal, no context, no audience, no limits.
- Then the organization is surprised to receive five pages of generalities, marketing fluff, or recommendations that do not fit its processes, data, or risks.
This is the equivalent of walking into a restaurant and ordering “something nice.” You can count on luck and the charisma of the waiter, but not on repeatability.
The SAVANT-AI project approaches this from the other side. Instead of relying on the genius of an “LLM charmer”, the system is meant to:
- force clarification of intent (who is asking, in what role, in what process),
- rely on domain-related query templates (e.g., production incident, RTO analysis, supplier evaluation),
- connect specific sources to the query: ERP, MES, PLM, documents, logs.
The user does not have to be a prompt poet. All they need to know is what business decision is involved. Cognitive orchestration and governance are meant to take care of the rest.
When AI acts as HR
There is an even darker side to AI in business: algorithms that count people like rows in Excel.
- A system that calculates productivity, costs, and KPIs and, on this basis, indicates “positions for optimization” operates without emotion. It knows nothing about credit, illness, or family situations; it only sees red numbers.
- We very quickly arrive at a scenario where the manager is only an interface for communicating decisions, and the entire “restructuring logic” is created in a model trained on historical data.
The problem is not that Excel does the math for people. The problem is that no one can explain why the system singled out this particular person, this department, or this branch. There is no audit, no source trail, no understandable decision rule.
The SAVANT-AI project aims to counter this trend. Instead of “AI as a guillotine,” it is meant to be “AI as an evidentiary tool”:
- each recommendation is to be based on specific agents, data, and policies,
- the audit is to show which indicators had the greatest impact and which scenarios were considered,
- the system is built from the ground up with regulators, compliance, and supervisory boards in mind, not just “cost per FTE.”
AI in SAVANT-AI is meant to show the consequences of different scenarios and leave the decision and responsibility to people.
Fast-food AI
In the consumer space, an entire segment of “AI for relationships” has emerged: chatbot partners, virtual girlfriends and boyfriends, and apps that “always listen, never judge, and always reply that you are important.”
Technologically, it’s fascinating; socially, it’s disturbing.
- The user pours their fears, complexes, fantasies, traumas, and needs into the model.
- The model patiently records this, building a dense psychological profile: what calms them down, what excites them, and what sells best in a given mood.
This is not a “romance with AI.” It is a living, real-time behavioral profile that can fuel advertising, risk scoring, and political manipulation.
This exact pattern is being transferred to large companies. If employees talk to a cloud-based chatbot all day long:
- they reveal details of projects, conflicts within teams, strategies, and competitors’ mistakes,
- they give the provider a very precise picture of how that particular organization thinks, what hurts it, and where its weaknesses lie.
The SAVANT-AI project goes in the opposite direction:
- the entire cognitive layer is to run locally, in isolated infrastructure, within the client’s jurisdiction.
- logs, profiles, and session contexts are to belong to the organisation and are covered by its security policies — they do not become fuel for training other people’s models.
- in critical scenarios the system will be switchable to deep sovereignty mode: no access to external interfaces, no metadata leaks.
When algorithms know you better
You don’t even need a talkative chatbot for AI to know more about a person than their loved ones. All you need is simple behavioral analytics:
- login times, breaks, clicking patterns, navigation paths,
- moods read from music, video, and scrolling speed,
- stress response patterns, notifications, conflicts.
Such data already builds behavioral profiles of users, often in a silent and non-transparent way.
In the SAVANT-AI project we assume that:
- the same knowledge about user behavior is used on the client side — to improve ergonomics, security, and anomaly and fraud detection,
- profiling rules, alarm thresholds, and anonymization methods are described in policies and can be audited — it is not a “black box of marketing.”
AI may know better than family how people work in systems. The question is: who controls this knowledge — the organization or an external provider?
Deepfake and the law of evidence in the AI era
Until recently, video and voice recordings were considered “hard evidence.” Today, a phone, a few samples from social media, and a simple app are enough for:
- a “voice” to ask an accountant for an urgent transfer,
- a “face” to announce a management decision that was never made,
- an “image” to support a political manifesto that the addressee has never seen.
At a time when everything can be generated, form alone is no longer sufficient as evidence. In the EU, this approach is reflected in new legislation: the AI Act requires high-risk systems to keep detailed logs and enable decisions to be explained, while regulations such as DORA and NIS2 reinforce the obligation of auditability and decision tracking in the financial sector and critical infrastructure.
What matters is traceability: where the data came from, who processed it, and in what chain of decisions a given document or message was created.
The SAVANT-AI project aims to create an evidence infrastructure for the era of deepfakes and growing regulatory requirements:
- every significant system response is to have a source path — documents, transaction records, sensor logs,
- every step of cognitive processing (which agent, which model, which policies) is to be logged,
- in the event of a dispute, incident, or investigation, it will be possible to reconstruct how a recommendation was made, who approved it, and whether it was manipulated.
This will not eliminate fraud in the world, but it is meant to give the organisation its own internal standard of “operational truth” that cannot be replaced by a single clever video.
New illiteracy
AI has already turned the tables in education.
- Homework is no longer a measure of knowledge, because in three minutes you can generate an essay, a report, a presentation, and a summary of a book.
- Schools designed for the era of notebooks and encyclopedias are now face-to-face with a free “math homework buddy.”
This forces us to ask uncomfortable questions: what are we actually teaching — memory or thinking? Will the competence of the future be cramming, or the ability to ask questions, verify, and connect answers with practice?
Exactly the same thing is happening in companies.
- If employees only learn to “copy and paste from a chatbot to slides,” we will raise digital illiterates who understand neither data nor the decision-making process.
- Reports on AI transformation in Poland show that many companies are implementing AI “blindly” — without strategy, governance, or audit, focusing on impressive rather than critical applications.
If we give them a local, sovereign “organisational brain” — like the one we research in the SAVANT-AI project — and teach them how to work with agents, data, and auditing, we will raise the level of business understanding instead of lowering it. Illiteracy in the 21st century is not a lack of access to AI, but a lack of the ability to have a meaningful conversation with it and understand what it really does with data.
NO to command poets — YES to algorithm dancers
On the wave of enthusiasm surrounding generative AI, a new profession has emerged: “prompt engineer,” “AI whisperer,” “LLM charmer.” In the short term, this makes sense: someone in the company needs to know how to talk to the model so that it doesn’t produce nonsense.
Today, the ability to ask meaningful questions of AI is becoming as obvious as using a search engine or spreadsheet. What will really set organizations apart is not the magic of prompts, but the quality of their cognitive infrastructure: data, policies, agent orchestration, sovereignty.
The SAVANT-AI project deliberately shifts the centre of gravity:
- from the “charmers” who squeeze one more prompt out of a cloud chatbot,
- to architects, operators, and domain agent owners who design how the entire AI ecosystem works within an organization.
This is not the role of a “command poet.” It is the role of an “algorithm dancer” — someone who understands business, data, and security all at once.
Six steps before you hire another chatbot
Before an organization invests in its next “intelligent assistant,” it’s worth going through six simple but uncomfortable steps.
- Map the sources — where decisions are really made and what data drives them.
- Define roles — who can ask the system for what, in what process, and with what responsibility.
- Audit logging — does the system record how it arrived at each recommendation, and can you show this to the regulator?
- Protect data — does AI operate in infrastructure controlled by the organization or in the cloud of a provider who builds its own advantage on it?
- Test accountability — can every decision made by the system be explained to the management board and supervisory board, pointing to specific data, policies, and individuals?
- Train people — does the team understand what the system does, or do they just believe in its authority and treat it as an infallible oracle?
Organizations that take these steps will build a sovereign, controlled system. Those that skip them will get another half a million zlotys spent on a chatbot and the same problem, just packaged more nicely.
Why the SAVANT-AI project in all this?
If you look at all of the above phenomena together, the picture is brutal:
- models are prone to confabulation,
- users tend to be lazy and believe in authority,
- cloud providers tend to collect data,
- and organizations tend to shift responsibility onto “the system.”
The SAVANT-AI project aims to reverse this arrangement. The system is meant to:
- move AI from an anonymous cloud to a sovereign, local ECS running on the customer’s infrastructure,
- surround models with agents, policies, auditing, and deep access control,
- prompt the organisation to define the organization to define what is a fact, who has access to what, and what the chain of accountability looks like.
This is not just another chatbot. It is the foundation of mature technological sovereignty in a world where AI can be a brilliant assistant, a shameless liar, and a soulless accountant all at once. It is up to the organization to decide which of these faces to let in.