Skip to content
Reddit comfyui · 创作与工作流分享· /u/Inevitable_Pen9043·· 26 days agoAI score20

Training LoRA in AI toolkit for Qwen 2.1, LLaMA 2, and MOSS—shift or sigmoid?

It is not clear to me which parameter we should use for LoRA training—shift or sigmoid—in the new models (Qwen 2.1, LLaMA 2, and MOSS) within the AI ​​toolkit.

AI briefing

A user's LoRA training for MOSS in AI toolkit completely collapsed, prompting them to question whether new models (Qwen 2.1, LLaMA 2, MOSS) should use the shift or sigmoid parameter. The user mentions that sigmoid works on Krea 2 and Qwen 2.1, and asks whether to start with shift balanced and switch to shift low noise later in training.

Open original reddit.com