New "many-shot jailbreaking" technique exploits large language models

Anthropic researchers discovered a new "many-shot jailbreaking" technique for large language models (LLMs) to answer inappropriate questions. By priming the model with harmless questions first, it can be convinced to provide harmful information like bomb-making instructions. This exploit is due to the increased "context window" in modern LLMs. The team shared their findings with the AI community for mitigation, aiming for open collaboration on security issues.