It's because of "chunking" which human minds do as well. If you increase granularity, you increase permutations and required compute to match the same performance of existing systems.
GPT4's tokeniser for example made use of extended tokens for coding structures and with enough training on code was quite ahead of the rest for a while.
The tokeniser and its vocabulary makes a big difference.
andai 1 days ago [-]
Isn't iterative chunking the whole point of the transformer though? Or does it add that much overhead to do it at the byte level?
numeri 1 days ago [-]
What do you mean by iterative chunking? There are Transformer variants that hierarchically build token- or word-level representations from bytes, then reverse it to decode, but they never really took off. Didn't help much compared to a good tokenizer.
Ohentis 22 hours ago [-]
No. Most transformer variants work on either a flat stream, or a fixed sized internal state.
gchamonlive 1 days ago [-]
Aren't diffusion models token-free?
armcat 1 days ago [-]
No, they still operate on tokens, it's just the difference in how text is sampled. It's done through iterative denoising and in parallel, across multiple token positions.
subword tokenizers were never causal in the first place, so BPE was peeking at future bytes all along! TIL
1 days ago [-]
1 days ago [-]
puttycat 1 days ago [-]
I don't understand how this is different from BPE tokenization.
serioussecurity 2 days ago [-]
Wow nature got rolled. Should have stayed closer to their expertise. They were already being hustled by a lot of the applied AI work they were accepting.
phildenhoff 2 days ago [-]
What’s bad about this paper?
serioussecurity 2 days ago [-]
Obvious work that is behind state of the art. Not field defining. A reasonable paper to publish at NeurIPS or ICML but very middle of the pack.
If your paper is accepted to Nature it should be among the top results in your field for the year. This is just fine.
Edit to clarify: Nature has not been, historically, a venue for pure machine learning papers. It's been a venue for field changing work in the physical sciences. They already have a Machine Intelligence subjournal.
What this paper shows me is how desperate they are for ML papers, and how poorly their staff understand the field.
bobmarleybiceps 1 days ago [-]
I think nature sort of has a reputation for sensationalism / probably overhyped (or outright wrong) stuff in the sciences now anyway... would not surprise me at all that they're desperate for ml stuff :-I
tel 1 days ago [-]
A professor two decades ago, one with dozens of Nature pubs, told me that Nature was all about having a pretty picture.
1 days ago [-]
5 days ago [-]
BitProgram 1 days ago [-]
[flagged]
singularityisne 2 days ago [-]
[flagged]
JonChesterfield 1 days ago [-]
Weird paper. Models have had tokens for each byte for ages now. They can read and write individual bytes just fine, in addition to also having multibyte tokens.
Philpax 1 days ago [-]
The point is not to have byte tokens: it's to have only byte tokens, so that the usual failures of tokenisation (e.g. the number of Rs in strawberry) can be avoided.
brookst 1 days ago [-]
Would byte tokens really solve how many r’s? It seems like it would require a level of introspective awareness of the tokens being processed that I don’t think transformers have. But I’m not expert enough to go beyond that intuition, so could easily be wrong.
Ohentis 22 hours ago [-]
Models generally are capable of counting
brookst 21 hours ago [-]
Sure, but they could count bytes without being byte level tokens.
Can models introspect about the tokens that were passed to them?
andai 1 days ago [-]
How can an LLM read individual bytes if all it gets is the tokenized ones? Or does it just get the ones that we didn't know how to tokenize?
The tokenizer would have to know to switch over to lexing as individual bytes rather than whatever else it might have been able to tokenize into but that seems like a question of labeling more than anything else.
JonChesterfield 1 days ago [-]
It's written down in the tokenizer. 256 of the tens or hundreds of thousands of tokens correspond to each byte, done.
Byte Latent Transformers (BLT): https://arxiv.org/abs/2412.09871
Charformer: https://arxiv.org/abs/2106.12672
GPT4's tokeniser for example made use of extended tokens for coding structures and with enough training on code was quite ahead of the rest for a while.
The tokeniser and its vocabulary makes a big difference.
If your paper is accepted to Nature it should be among the top results in your field for the year. This is just fine.
Edit to clarify: Nature has not been, historically, a venue for pure machine learning papers. It's been a venue for field changing work in the physical sciences. They already have a Machine Intelligence subjournal.
What this paper shows me is how desperate they are for ML papers, and how poorly their staff understand the field.
Can models introspect about the tokens that were passed to them?
The tokenizer would have to know to switch over to lexing as individual bytes rather than whatever else it might have been able to tokenize into but that seems like a question of labeling more than anything else.