FDI : Attack neural code generation systems through user feedback channel

• Main Authors: SUN, Zhensu, DU, Xiaoning, LUO, Xiapu, SONG, Fu, LO, David, LI, Li
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2024
• Subjects: Code generation; Data poisoning; User feedback; Security and privacy; Feedback data injection; Artificial Intelligence and Robotics; Information Security
• Online Access: https://ink.library.smu.edu.sg/sis_research/9885; https://ink.library.smu.edu.sg/context/sis_research/article/10885/viewcontent/2408.04194v1.pdf
• Institution: Singapore Management University

Neural code generation systems have recently attracted increasing attention to improve developer productivity and speed up software development. Typically, these systems maintain a pre-trained neural model and make it available to general users as a service (e.g., through remote APIs) and incorporate a feedback mechanism to extensively collect and utilize the users' reaction to the generated code, i.e., user feedback. However, the security implications of such feedback have not yet been explored. With a systematic study of current feedback mechanisms, we find that feedback makes these systems vulnerable to feedback data injection (FDI) attacks. We discuss the methodology of FDI attacks and present a pre-attack profiling strategy to infer the attack constraints of a targeted system in the black-box setting. We demonstrate two proof-of-concept examples utilizing the FDI attack surface to implement prompt injection attacks and backdoor attacks on practical neural code generation systems. The attacker may stealthily manipulate a neural code generation system to generate code with vulnerabilities, attack payload, and malicious and spam messages. Our findings reveal the security implications of feedback mechanisms in neural code generation systems, paving the way for increasing their security.