An algorithm judges you, and the authority says: it is not AI

Author: Andrzej Albera, Responsible for strategy and the company's direction

The situation. The owner of a small haulage company applies to lease a tractor unit. Thirty seconds later the application is rejected. He asks the salesperson why. The answer: “The system calculated it. It is not artificial intelligence, just scoring”. The customer does not know what decided the outcome, whether anyone looked at his documents at all, or whether he can appeal.

Black boxYou can see what decidedScoring with no explanationApplication: tractor leasePLN 420,000REJECTEDWhy?The system calculated it. Not AI.Assessment with an explanationApplication: tractor leasePLN 420,000Repayment historyCompany turnoverDebt levelDecision: analystRequest a reviewThe customer is left guessing. ComplaintThe customer knows what to improveGDPR Article 22 · the right to an explanation and to a human decision

What the law says

What to do

  1. List the systems that assess people. Customer scoring, CV filters, selection for inspection, risk pricing. For each, note whether it runs on fixed rules or on a model trained on data. This is the first document an inspector will ask for.
  2. Leave the last word to a person. Where an assessment decides on a loan, a job or a benefit, the system suggests and an employee who understands what it is based on makes and signs the decision. Customers must know how to ask for a review.
  3. Prepare an explanation customers can understand. One page: what data the system takes into account, which of it has the greatest effect on the result and what the customer can do to get a different result. Such an explanation protects the company better than saying “it is not AI”.

Our SAIE ecosystem helps with this. In the CDF methodology we describe the risk classification and the rules of human oversight for each use case separately, before the system goes live. PROXY:AI keeps a register of AI use, so you can see who used the models, when and for what. Assistants in TWIN:DESK suggest and give their sources, while the decision and the signature stay with a person. The company has a ready description of the system in case of an inspection.

Frequently asked questions

Is every scoring system an AI system?

No. A program that only adds up points according to rules entered by people is usually not one. A model trained on historical data that decides for itself what indicates risk usually is. When in doubt, it is better to describe the system as if it were an AI system.

Must a company show its algorithm to the customer?

It does not have to disclose its code or trade secrets. It must, however, explain in an intelligible way which data affected the assessment and how. That is what the Court of Justice of the EU ruled in 2025.

What if an employee approves the decision?

That is good, provided they really check it. Clicking “approve” on hundreds of applications a day is not human oversight. We wrote about how to change that in our post on the Human Competence Gate.

When do AI Act fines apply?

For high-risk systems from 2 December 2027: up to EUR 15 million or 3% of turnover, the lower of the two for small and medium-sized companies. GDPR fines apply already today: up to EUR 20 million or 4% of turnover.

Does this also apply to a small company?

Yes, if it uses an off-the-shelf tool that assesses customers or job candidates. The obligations apply not only to the tool’s developer but also to the company that uses it.

The opportunity

A system that can explain its assessment responds faster and more fairly than a person with fifteen applications on the desk. 41% of Poles trust AI, compared with 46% worldwide (KPMG). Trust grows when people know why a decision was made. A company that shows this will win a customer, not a complaint.

Sources