Warning for South African businesses using ChatGPT, Gemini, and Claude

South African businesses using AI tools such as ChatGPT, Gemini and Claude could face risks under POPIA if employee or applicant information is uploaded to these systems.

This is according to Bowmans’ partners Melissa Cogger and Talita Laubscher, who said businesses must understand how the Protection of Personal Information Act (POPIA) applies when AI is used in human resources.

AI tools are increasingly being used to screen job applications, assess performance, support recruitment and process employee information.

However, these tools can involve large amounts of personal information, including curricula vitae, interview recordings, health information and performance data.

Under POPIA, employers remain responsible for ensuring that this information is processed lawfully and securely.

Section 71 of POPIA is particularly important where AI is used to make decisions about employees or job applicants.

The section restricts decisions based solely on automated processing where the outcome has legal consequences for a person or substantially affects them.

This includes profiling relating to matters such as performance at work, health, reliability, location, personal preferences and conduct.

In an HR environment, this could apply where an AI system generates a performance rating, identifies an employee for promotion or demotion, or recommends rejecting a job candidate.

Where the decision is based solely on the AI system and has a significant effect on the person, the requirements of Section 71 may apply.

There are exceptions where automated decision-making is linked to the conclusion or performance of a contract, provided appropriate measures are in place to protect the data subject’s legitimate interests.

In such cases, the affected person must be given an opportunity to make representations. They must also be given sufficient information about the underlying logic of the automated processing to do so.

This poses a challenge for businesses using AI because some systems do not explain how they arrive at a result. This is often described as the “black box” problem.

The system receives information and produces an outcome, but the reasoning behind the outcome may be difficult to understand. This creates an accountability problem for employers.

A recruitment agency or HR department using AI to shortlist candidates needs to understand how the system reaches its conclusions.

Applicants who are excluded may need an opportunity to challenge the decision and engage with a human decision-maker where Section 71 applies.

Businesses should also consider whether the criteria used by the system are rational, fair and objectively justifiable.

POPIA requirements

POPIA also imposes information requirements on businesses that collect personal information, Cogger and Laubscher explained.

Section 18 requires parties to take practicable steps to ensure that people know what information is being collected, and why it is being collected, and

Where relevant, they must also ensure they know where it may be transferred. These requirements are particularly important when employers use third-party AI platforms.

Employees and job applicants should be informed that their personal information is being processed using AI and understand the purpose of that processing.

Businesses also need to consider whether personal information will be transferred across borders when using cloud-based AI systems.

Cogger and Laubscher added that POPIA’s general conditions for lawful processing create further obligations for employers.

The principle of minimality requires personal information to be adequate, relevant and not excessive for the purpose for which it is being processed.

This means businesses should question whether an AI system really needs access to large amounts of personal information.

The same applies to social media data and other information gathered through methods such as online scraping.

POPIA also requires that information be collected for a specific purpose and not later used for an unrelated purpose.

Businesses must also take technical and organisational measures to protect personal information from loss, damage or unauthorised access. Personal information should not be retained for longer than necessary, either.

According to Cogger and Laubscher, some types of personal information receive additional protection under POPIA.

This includes information about health, religious or philosophical beliefs, race or ethnic origin, trade union membership, political persuasion, sex life, biometric information and criminal behaviour.

AI systems that process or infer this type of information require particular care. For example, facial recognition technology may process biometric information or potentially infer information about race or ethnicity.

Similarly, workplace wellness systems may handle sensitive health information. Businesses need to ensure that there is a basis for processing this information and that the POPIA requirements are met.

What employers should do

Cogger and Laubscher noted that many employers use recruitment agencies or external AI providers rather than developing their own systems.

POPIA distinguishes between a “responsible party”, which determines the purpose and means of processing, and an “operator”, which processes information on behalf of the responsible party.

An employer is the responsible party even when a recruitment agency or AI provider performs the processing.

This means businesses must ensure that contracts with AI vendors and recruitment agencies include requirements covering data security, purpose limitations, retention and POPIA compliance.

Simply outsourcing the processing does not remove the employer’s responsibilities, Cogger and Laubscher stressed.

Another risk arises when employees use public AI tools in their jobs. For example, an employee could upload a colleague’s personal information, customer information or other confidential data into an AI platform.

This could create a POPIA compliance problem for the employer because the business remains responsible for personal information under its control.

Cogger and Laubscher said businesses should introduce clear AI-use policies setting out what employees may and may not upload to AI systems.

Employers should also consider using closed or enterprise AI systems where possible, particularly for sensitive information, or require personal information to be de-identified before it is entered into an AI tool.

Training is also important so employees understand the risks and the information they are allowed to provide to AI systems.