Agentic AI Security & Autonomous Red-Teaming
Keywords:
Agentic AI Security, Autonomous Red-Teaming, AI Vulnerability AssessmentAbstract
Recent progress in foundation models and multi-agent orchestration systems has increased their capabilities and also their attack surface. Cyber-physical systems and edge devices serve both as a target of deployment and as an enabler of operation. The security issues surrounding these enabling mechanisms are already becoming a reality, but the implications of these issues on AI-driven ecosystems are under-researched. In contrast to traditional security areas, threats in agentic AI environments are difficult to anticipate due to their dynamic execution contexts, lack of standardized operational baselines, and the unpredictable behaviors arising from autonomous and emergent agent strategies. This paper examines these challenges and proposes a forward-looking security approach centered on continuous model verification, alignment assurance, and transparency tooling tailored to agentic systems. The framework emphasizes early, automated, and lifecycle-integrated security validation, augmented by autonomous red-teaming to proactively surface weaknesses. The findings suggest that embedding self-assessing security mechanisms into agentic AI pipelines enables more resilient, adaptive, and accountable intelligent systems.
References
[1] A. Patil, N. Patel, and S. Deshpande, "Ethical Decision-Making in Sustainable Autonomous Transportation: A Comparative Study of Rule-Based and Learning-Based Systems," Cogent Engineering, vol. 11, no. 12s, 2025. [Online]. Available: https://doi.org/10.64252/cgzh6r94
[2] M. DeBellis and R. Neches, "Knowledge Representation and the Semantic Web: An Historical Overview of Influences on Emerging Tools," Recent Advances in Computer Science and Communications, vol. 16, no. 6, pp. 22–36, Jul. 2023. [Online]. Available: https://doi.org/10.2174/2666255815666220527145610
[3] Cloud Security Alliance and OWASP AI Exchange, "Agentic AI Red Teaming Guide," 2025. [Online]. Available: https://cloudsecurityalliance.org/artifacts/agentic-ai-red-teaming-guide
[4] A. Dawson, R. Mulla, N. Landers, and S. Caldwell, "AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models," arXiv:2506.14682, 2025. [Online]. Available: https://arxiv.org/abs/2506.14682
[5] HiddenLayer, "Indirect Prompt Injection of Claude Computer Use," HiddenLayer Innovation Hub, 2024. [Online]. Available: https://hiddenlayer.com/innovation-hub/indirect-prompt-injection-of-claude-computer-use/
[6] K. Huang, "Agentic AI Threat Modeling Framework: MAESTRO," Cloud Security Alliance, Feb. 6, 2025. [Online]. Available: https://cloudsecurityalliance.org/blog/2025/02/06/agentic-ai-threat-modeling-framework-maestro
[7] K. Huang, G. Huang, Y. Duan, and J. Hyun, "Utilizing Prompt Engineering to Operationalize Cybersecurity," in Generative AI Security: Theories and Practices, K. Huang, Y. Wang, B. Goertzel, Y. Li, S. Wright, & J. Ponnapalli, Eds., Springer, 2024, pp. 271–303. [Online]. Available: https://doi.org/10.1007/978-3-031-54252-7_9
[8] K. Huang, V. Manral, and W. Wang, "From LLMOps to DevSecOps for GenAI," in Generative AI Security: Theories and Practices, K. Huang, Y. Wang, B. Goertzel, Y. Li, S. Wright, & J. Ponnapalli, Eds., Springer, 2024, pp. 241–269. [Online]. Available: https://doi.org/10.1007/978-3-031-54252-7_8
[9] K. Huang, J. Huang, and C. Hughes, "AI Agents in Offensive Security," in Agentic AI: Theories and Practices, K. Huang, Ed., Springer, 2025, pp. 167–205. [Online]. Available: https://doi.org/10.1007/978-3-031-90026-6_6
[10] Invariant Labs, "MCP Security Notification: Tool Poisoning Attacks," Invariant Labs Blog, May 26, 2025. [Online]. Available: https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks
[11] Z. Wang, C. Q. Knight, J. Kritz, W. E. Primack, et al., "A Red Teaming Roadmap Towards System-Level Safety," arXiv preprint, 2025. [Online]. Available: https://arxiv.org/abs/xxxx.xxxxx
[12] M. Feffer, A. Sinha, W. H. Deng, Z. C. Lipton, et al., "Red-teaming for Generative AI: Silver Bullet or Security Theater?," Proceedings of the AAAI Conference, 2024. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/xxxx
[13] S. Ghosh, B. Simkin, K. Shiarlis, S. Nandi, D. Zhao, et al., "A Safety and Security Framework for Real-World Agentic Systems," arXiv preprint, 2025. [Online]. Available: https://arxiv.org/abs/xxxx.xxxxx
[14] A. Sinha, K. Grimes, J. Lucassen, M. Feffer, et al., "From Firewalls to Frontiers: AI Red-Teaming is a Domain-Specific Evolution of Cyber Red-Teaming," arXiv preprint, 2025. [Online]. Available: https://arxiv.org/abs/xxxx.xxxxx
[15] B. Ren, E. J. Cheon, and J. Li, "Organization Matters: A Qualitative Study of Organizational Dynamics in Red Teaming Practices for Generative AI," Proceedings of the ACM on Human-Computer Interaction, 2025. [Online]. Available: https://dl.acm.org/doi/xxxx
[16] I. Wicaksono, Z. Wu, R. Patel, T. King, et al., "Mind the Gap: Comparing Model-vs Agentic-Level Red Teaming with Action-Graph Observability on GPT-OSS-20B," arXiv preprint, 2025. [Online]. Available: https://arxiv.org/abs/xxxx.xxxxx
[17] V. Saarainen, "Red Teaming: Regulatory and Non-Regulatory Frameworks Used in Adversarial Simulations," Theseus.fi, 2021. [Online]. Available: https://www.theseus.fi/handle/10024/xxxx
[18] B. Challita and P. Parrend, "RedTeamLLM: An Agentic AI Framework for Offensive Security," arXiv preprint arXiv:2505.06913, 2025. [Online]. Available: https://arxiv.org/abs/2505.06913
[19] R. Singh, B. Blili-Hamelin, and C. Anderson, "Red-Teaming in the Public Interest," Data & Society, 2025. [Online]. Available: https://ranjitsingh.me/red-teaming-public-interest
[20] K. Huang and C. Hughes, "Agentic AI Red Teaming," in Securing AI Agents, Advances in Data Analytics, AI, and Smart Systems (ADAASS), pp. 207–252, 2025.
[21] S. Majumdar, B. Pendleton, and A. Gupta, "Red Teaming AI Red Teaming," in Conference on Applied Machine Learning for Information Security (CAMLIS) 2025, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2507.05538
Downloads
Published
Issue
Section
License
Copyright (c) 2026 American Scientific Research Journal for Engineering, Technology, and Sciences
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who submit papers with this journal agree to the following terms.
