EU AI Act: Prohibitions & Risks

The EU Commission published Guidelines on prohibited artificial intelligence practices established by Regulation (EU) 2024/1689 (AI Act), stating that 'AI systems that threaten people’s safety, rights, or livelihoods are strictly prohibited, except in limited circumstances.' The ban on AI systems classified as posing unacceptable risk came into force in February 2025.

Practices and systems that are not defined as prohibited in the EU AI Act (2024), are categorised by the Act into different risk levels (High, Minimal and Limited Risk levels).

Key Requirements of UCC Researchers to mitigate risks & limitations of AI and GenAI use in order to comply with EU legislation

• Researchers must not use AI or GenAI systems that are prohibited by Guidelines on prohibited artificial intelligence practices established by Regulation (EU) 2024/1689 (AI Act), see Appendix D.
• Researchers must be aware of the risk level and limitations of GenAI systems associated with their specific practice and take steps to mitigate such risks (see Appendices D & F). It is important for researchers to understand the the risks and limitations of GenAI tools in order to safeguard against the potential negative ethical implications that arise with irresponsible use of GenAI systems as well as leading to unacceptable practices and research misconduct, thus undermining Research Integrity and Good Research Practice.
• Researchers should familiarise themselves and comply with the UCC Statement on AI, IT policies and all relevant policies pertaining to the responsible use of GenAI in their research.
• In line with UCC Research Security Framework and Due Diligence Process, researchers should ensure to protect sensitive knowledge and/or personal data and be aware of and comply with UCC’s cybersecurity measures, IT-Security, UCC IT Security Policy.

Appendix D summarises prohibited AI practices and risk levels of different AI practices relevant to research. 

Key Risks & Limitations

Nature’s 2023 study, surveying over 1,600 researchers (Section 3), reported on the concerns expressed by researchers globally on the risks, limitations and impacts of AI and GenAI use.

These risks, limitations and impacts can be understood in the context of how these systems work and the challenges and issues which can arise. Generative AI models can exhibit limitations that impact both their responses and usability.

It is critical that researchers know and understand the issues, challenges, limitations and risks associated with GenAI use – these include the following and are outlined in detail in Appendix E:

• Transparency
• Random Errors
• Systemic Errors
• Lack of Moral Agency
• Disconnect from the Physical World
• Plagiarism & Originality
• Privacy Infringement
• Reproducibility
• Sustainability

Research Misconduct

The intentional or unintentional misuse and irresponsible use of AI and GenAI tools are subject to Research Integrity principles and processes as set out in the UCC Code of Research Conduct, as well as the UCC Violations of Research Integrity Policy. Research Misconduct constitutes Fabrication, Falsification Plagiarism. Additionally, there are other types of unacceptable research practices which can damage the research record as they can be more prevalent and therefore (in the aggregate) potentially as damaging to the overall research process, the credibility of research and the reputation of the research community.

Research Misconduct relating to the use of Generative AI in research in UCC:

• Use AI or GenAI systems that are prohibited by the Guidelines on prohibited artificial intelligence practices established by Regulation (EU) 2024/1689 (AI Act), see Appendix D.
• Researchers are liable for Research Misconduct where they intentionally, knowingly, or recklessly use AI and/or GenAI to fabricate or falsify data or commit plagiarism.
• Content produced by GenAI tools does not represent original work. Therefore, use of GenAI without appropriate declaration, acknowledgement and / or notification would be considered a form of Research Misconduct.
• Reckless or inappropriately use AI and/or GenAI tools in a manner that is detrimental to institutional research security and pose risks for Research Quality, Integrity and Ethics.
• Data breach – inadvertent leak causing undue harm to individuals and breaches in data regulation and security (personal and/or sensitive data) (see Section 4.3, Appendices F).
• Where there is a claim that there has been a potential breach of the Responsible usage of Generative AI, UCC shall follow its Procedure in Cases of Suspected Research Misconduct/Disputes, as outlined in the UCC Code of Research Conduct and the UCC Violations of Research Integrity Policy.

Breaches will be handled under the UCC Procedure in Cases of Suspected Research Misconduct.