It was to be expected: GPT-4 is the best informed about Dutch culture, although it was an unequal battle. It’s like playing Mario kart against someone you know is much better. You always have a chance to win, but in the vast majority of cases you will still lose.
It is unknown how many parameters GPT-4 has, but in absolute numbers it is certainly much, much more training data than the entire model of Llama 3 Dutch and GEITje. This numerical dominance is clearly visible, in addition to the fact that OpenAI can of course have a lot of data about what Dutch people ask and can therefore better tailor the answers. Furthermore, GPT-4 is a source of income for OpenAI, a company with hundreds of employees.
| Which language model wins? | GPT-4 | Llama 3 Dutch | GOAT Ultra |
| Circle birthday snacks | 1 | 3 | 2 |
| Birthday greeting | 1 | 1 | 1 |
| Germans on the beach | 1 | 3 | 2 |
| Eating stew | 1 | 2 | 3 |
| Station bike | 1 | 2 | 2 |
| Carnival Bergen op Zoom | 1 | 2 | 2 |
| Carnival Flanders | 1 | 1 | 1 |
| Royal day | 1 | 2 | 2 |
Between Llama 3 Dutch and GEITje Ultra, the victory goes to GEITje. The Dutch language model sometimes seems to understand the Dutch context better, although it often depends on preference which answer is best. The assessment is of course subjective; everyone can make their own assessment based on the answers in this article.
Rijgersberg believes that as a society we should think about what happens next. “I have received a lot of positive reactions to GEITje and also a lot of questions for a follow-up. But my hobby budget for this has now run out, so I don’t have a follow-up for the time being. There comes a point when we as a country and as a language community have to decide how important we find the existence of these types of Dutch open models. And whether we want to keep up with developments in this way. If we all think that is important, we then have to decide who should do that.”
The publicly available Dutch language models may not win, but it is nice that there are Dutch language models that are freely available and can be run locally. In any case, competition is a great thing and it would be bad if we became dependent on large, closed AI models from the big tech companies that have dominated for more than a decade.
