ETH and EPFL publish large language model

(CONNECT) Researchers at Swiss universities such as the Swiss Federal Institutes of Technology in Zurich (ETH) and Lausanne (EPFL) have developed a Large Language Model (LLM) together with the Swiss National Supercomputing Center (CSCS). According to a statement of the ETH, the LLM will be published in late summer and will offer multilingual competence in over 1000 languages. In the future, it will be completely publicly available under an open source license, enabling broad use and promoting innovation in science, society and industry.

The model will be published in two sizes, with 8 and 70 billion parameters. According to ETH, the larger version will be one of the world's most powerful, fully open models. The model was trained on the Alps supercomputer at the CSCS in Lugano using 100 percent climate-neutral electricity. By training with over 15 billion high-quality training tokens and a text dataset of over 1500 languages, the model achieves a high level of reliability.

The LLM was developed in completely open and transparent processes. The source code and weights will be publicly available, and the training data will be transparent and reproducible. This is intended to offer a trustworthy alternative to commercial systems, most of which have been developed using non-traceable processes. 

"Fully open models enable applications that ensure a high degree of trustworthiness and are necessary to advance research into the risks and opportunities of AI," says Imanol Schlag, researcher at the ETH AI Center and co-leader of the project. "Transparent processes also enable compliance with legal regulations." Swiss data protection laws, Swiss copyright law and the transparency obligations of the EU's AI Regulation were therefore taken into account during development. ce/nta