What's the downside? Don't they stop when they hit diminishing returns? | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

andai 3 months ago | parent | context | favorite | on: Liquid AI reveals 8B-A1B MoE trained on 38T

What's the downside? Don't they stop when they hit diminishing returns?

Ifkaluva 3 months ago | [–]

You’d think so, but I haven’t seen it explicitly discussed in their papers, and nobody else that I know of trains on that many tokens

hgoel 3 months ago | [–]

Wouldn't the model start overfitting at some point? Degrading generalization for accuracy on the training set.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact