An algorithm judges you, and the authority says: it is not AI
The situation. The owner of a small haulage company applies to lease a tractor unit. Thirty seconds later the application is rejected. He asks the salesperson why. The answer: “The system calculated it. It is not artificial intelligence, just scoring”. The customer does not know what decided the outcome, whether anyone looked at his documents at all, or whether he can appeal.
What the law says
- What an AI system is. The AI Act treats as an AI system a program that infers from data, on its own, outputs such as predictions, assessments, recommendations or decisions. The European Commission’s guidelines of February 2025 exclude simple programs that only execute rules set by people. A model that has learned from data whom to regard as risky is already an AI system, even if the company calls it a “rules engine”.
- Assessing people is high risk. Creditworthiness assessment, selecting job candidates and granting public benefits are high-risk areas listed in Annex III to the AI Act. A system that profiles people in these areas is always a high-risk system (Article 6(3)). The full obligations for such systems, including the right to an explanation of a decision (Article 86), will apply from 2 December 2027. You can check whether your system is high-risk with our AI Act classifier.
- The GDPR applies already. A decision based solely on automated processing that significantly affects a person requires the right to human intervention, to express one’s point of view and to contest the decision (Article 22 GDPR). In the SCHUFA case (2023) the Court of Justice of the EU held that automated scoring alone can be such a decision, and in Dun & Bradstreet Austria (2025) that a person has the right to an intelligible explanation of how their assessment was made. In Poland banks also have this obligation under the Banking Law (Article 70a).
- The state has the same problem. In May 2026 the Polish NGO Watchdog Polska published a report on AI in 5,480 public institutions. 9% of them declare that they use AI. According to the report, the Social Insurance Institution (ZUS) uses a predictive model to select sick notes for inspection, but does not regard it as an AI system within the meaning of the AI Act. The boundary of the definition is disputed, which is exactly why it is worth clarifying in writing.
- Complaints are already possible. Since 28 October anyone can file a complaint about an AI system with Poland’s AI regulator, the Commission for the Development and Safety of Artificial Intelligence. Read more about the act on our page The Polish AI Systems Act.
What to do
- List the systems that assess people. Customer scoring, CV filters, selection for inspection, risk pricing. For each, note whether it runs on fixed rules or on a model trained on data. This is the first document an inspector will ask for.
- Leave the last word to a person. Where an assessment decides on a loan, a job or a benefit, the system suggests and an employee who understands what it is based on makes and signs the decision. Customers must know how to ask for a review.
- Prepare an explanation customers can understand. One page: what data the system takes into account, which of it has the greatest effect on the result and what the customer can do to get a different result. Such an explanation protects the company better than saying “it is not AI”.
Our SAIE ecosystem helps with this. In the CDF methodology we describe the risk classification and the rules of human oversight for each use case separately, before the system goes live. PROXY:AI keeps a register of AI use, so you can see who used the models, when and for what. Assistants in TWIN:DESK suggest and give their sources, while the decision and the signature stay with a person. The company has a ready description of the system in case of an inspection.
Frequently asked questions
Is every scoring system an AI system?
No. A program that only adds up points according to rules entered by people is usually not one. A model trained on historical data that decides for itself what indicates risk usually is. When in doubt, it is better to describe the system as if it were an AI system.
Must a company show its algorithm to the customer?
It does not have to disclose its code or trade secrets. It must, however, explain in an intelligible way which data affected the assessment and how. That is what the Court of Justice of the EU ruled in 2025.
What if an employee approves the decision?
That is good, provided they really check it. Clicking “approve” on hundreds of applications a day is not human oversight. We wrote about how to change that in our post on the Human Competence Gate.
When do AI Act fines apply?
For high-risk systems from 2 December 2027: up to EUR 15 million or 3% of turnover, the lower of the two for small and medium-sized companies. GDPR fines apply already today: up to EUR 20 million or 4% of turnover.
Does this also apply to a small company?
Yes, if it uses an off-the-shelf tool that assesses customers or job candidates. The obligations apply not only to the tool’s developer but also to the company that uses it.
The opportunity
A system that can explain its assessment responds faster and more fairly than a person with fifteen applications on the desk. 41% of Poles trust AI, compared with 46% worldwide (KPMG). Trust grows when people know why a decision was made. A company that shows this will win a customer, not a complaint.
