← Poisoning Generative AI:
What You Need To Know

Code Generation

AI Copilots risk licence contamination

GitHub’s safeguards against contaminating your code base with GPL-licenced code are insufficient. If the Copilot copies code under the GPL and modifies it in even the slightest way, GitHub’s safeguards no longer work, but your code will still be contaminated.

They are prone to insecure code

They seem to generate code that is at least as insecure as that written by a novice programmer.

They trigger our Automation and Anchoring Biases

Tools for cognitive automation trigger your automation bias. If you’re using a tool to help you think less, that’s exactly what you do, which compromises your judgement about what the tool is doing. We also have a bias that favours whatever “anchors” the current context, usually the first result, even if that result is subpar.

They are “stale”

Most of the code language models are trained on is legacy code. They will not be aware of deprecations, security vulnerabilities, new frameworks, new platforms, or updated APIs.

They reinforce bad ideas

Unlike research, an AI will never tell you that your truly bad idea is truly bad. This is a major issue in software development as programmers are fond of reinventing bad ideas.

They promote code bloat

The frequency of defects is software generally increases proportionally with lines of code. Larger software projects are also more prone to failure and have much higher maintenance costs. Code copilots promote code bloat and could that way increase not decrease development costs in the long term.


Cover for the book 'The Intelligence Illusion'

These cards were made by Baldur Bjarnason.

They are based on the research done for the book The Intelligence Illusion: a practical guide to the business risks of Generative AI .

“Analyzing the Legal Implications of GitHub Copilot - FOSSA.” Dependency Heaven, July 2021. https://fossa.com/blog/analyzing-legal-implications-github-copilot/.
“Anchoring Bias.” The Decision Lab. Accessed April 3, 2023. https://thedecisionlab.com/biases/anchoring-bias.
Atwell, Elaine. GitHub Copilot Isn’t Worth the Risk.” Kolide, February 2023. https://www.kolide.com/blog/github-copilot-isn-t-worth-the-risk.
“Don’t Believe ChatGPT - We Do NOT Offer a "Phone Lookup" Service,” February 2023. https://blog.opencagedata.com/post/dont-believe-chatgpt.
Fleury, Ryan. “The Gullible Software Altruist,” July 2022. https://www.rfleury.com/p/the-gullible-software-altruist.
Glaiel, Tyler. “Can GPT-4 *Actually* Write Code?” Substack newsletter. Tyler’s Substack, March 2023. https://tylerglaiel.substack.com/p/can-gpt-4-actually-write-code.
Howard, Jeremy. “Fast.ai - Is GitHub Copilot a Blessing, or a Curse?” July 2021. https://www.fast.ai/posts/2021-07-19-copilot.html.
Kuhn, Bradley M. “If Software Is My Copilot, Who Programmed My Software?” Software Freedom Conservancy, February 2022. https://sfconservancy.org/blog/2022/feb/03/github-copilot-copyleft-gpl/.
Mosier, Kathleen L., Linda J. Skitka, Mark D. Burdick, and Susan T. Heers. “Automation Bias, Accountability, and Verification Behaviors.” Proceedings of the Human Factors and Ergonomics Society Annual Meeting 40, no. 4 (October 1996): 204–8. https://doi.org/10.1177/154193129604000413.
Mosier, Kathleen L., Linda J. Skitka, Susan Heers, and Mark Burdick. “Automation Bias: Decision Making and Performance in High-Tech Cockpits.” The International Journal of Aviation Psychology 8, no. 1 (January 1998): 47–63. https://doi.org/10.1207/s15327108ijap0801_3.
Mosier, K., and L. Skitka. “Human Decision Makers and Automated Decision Aids: Made for Each Other?” 1996. https://www.semanticscholar.org/paper/Human-Decision-Makers-and-Automated-Decision-Aids%3A-Mosier-Skitka/ffb65e76ac46fd42d595ed9272296f0cbe8ca7aa.
Parasuraman, Raja, and Victor Riley. “Humans and Automation: Use, Misuse, Disuse, Abuse.” Human Factors: The Journal of the Human Factors and Ergonomics Society 39, no. 2 (June 1997): 230–53. https://doi.org/10.1518/001872097778543886.
Pearce, Hammond, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. “Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions.” arXiv, December 2021. https://doi.org/10.48550/arXiv.2108.09293.
Perry, Neil, Megha Srivastava, Deepak Kumar, and Dan Boneh. “Do Users Write More Insecure Code with AI Assistants?” arXiv, December 2022. https://doi.org/10.48550/arXiv.2211.03622.
“Questions Around Bias, Legalities in GitHub’s Copilot.” PWV Consultants, July 2021. https://www.pwvconsultants.com/blog/questions-around-bias-legalities-in-githubs-copilot/.
Rossi, Luca. “On AI Freshness, the Pyramid Principle, and Hierarchies 💡,” April 2023. https://refactoring.fm/p/on-ai-freshness-the-pyramid-principle.