Foundation Models No Longer Need Prompts or Labels: EPFL Researchers Introduce a Joint Inference Framework for Fully Unsupervised Adaptation Using Fine-Tuning and In-Context Learning

Foundation models, often massive neural networks trained on extensive text and image data, have significantly shifted how artificial intelligence systems handle language and vision tasks. These models are not designed for a single task but generalize across a wide variety by leveraging their pretraining knowledge. Once trained, they can generate coherent responses, classify images, or solve problems without needing new task-specific training. Their scalability and reuse across domains make them a cornerstone of AI development.

Despite their broad capabilities, a persistent issue lies in how these models are adapted for new, unseen tasks. In most scenarios, achieving strong performance requires providing them with handcrafted prompts or labeled examples that guide the model on how to behave. This process, however, introduces overhead, as crafting prompts involves trial and error, and collecting labeled examples can be expensive and time-consuming. Moreover, in real-world applications, such support data may not always be readily available, limiting the usability of foundation models in zero-shot settings.

Several strategies have been used to bridge this gap between generality and task-specific performance. In-context learning enables models to mimic a task by including example input-output pairs during inference, while supervised fine-tuning adjusts model weights using labeled data. Another method, prompt engineering, involves crafting prompts that steer the model toward desired outputs. Though these tools have been successful in boosting performance, each relies on external support—either human input or labeled data—making them less viable in completely unsupervised settings.

Swiss Federal Institute of Technology Lausanne (EPFL) researchers introduced a joint inference framework that supports unsupervised adaptation. This framework enables foundation models to perform coordinated predictions over multiple inputs without requiring ground truth data or manual prompts. The research team presented two specific techniques under this framework: unsupervised fine-tuning and unsupervised in-context learning. These methods allow models, including closed-weight ones like GPT-4, to improve accuracy without external guidance.

The approach of unsupervised fine-tuning works by letting the model iteratively improve its predictions using only its feedback. It formulates an optimization objective where predictions for a batch of inputs are generated together, and their joint probability is maximized. This method uses LoRA (Low-Rank Adaptation) for efficient weight updates and introduces a regularization step to avoid trivial solutions, such as predicting the same answer for all inputs. The researchers developed unsupervised in-context learning for situations where weight access isn’t available, such as with GPT-4. This method mimics the effect of labeled ICL by using previously generated outputs as pseudo-labels, refining predictions over multiple iterations without human annotations. Each iteration involves conditioning the model on prior examples and developing a more accurate answer, simulating a supervised learning loop through self-generated data.

The performance improvements from these unsupervised methods were substantial. On the GSM8K dataset, designed for math reasoning, unsupervised ICL applied to the Qwen2.5-Math model achieved a 39.2% absolute improvement over the standard zero-shot baseline. Similarly, for the Llama-3.1-8B model tested across 13 natural language processing tasks, unsupervised fine-tuning delivered a 23% average gain in accuracy. It matched the performance of fully supervised fine-tuning in 6 out of the 13 tasks. In vision-language tasks, unsupervised ICL also demonstrated strong results—showing a 23% gain on the Food101 dataset and significant improvements across other benchmarks. The research even extended to GPT-4o, a closed-weight model, where a 3% improvement was observed on ImageNet, reinforcing the framework’s versatility.

This work reveals a meaningful shift in how foundation models can adapt. The researchers successfully addressed the core limitation—reliance on labeled data and manual configuration—by introducing a robust and scalable self-supervised strategy. Their joint inference framework is a practical, generalizable approach that redefines the boundaries of unsupervised learning for large-scale AI models.

Check out Paper and Project. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and don’t forget to join our 85k+ ML SubReddit.

Nikhil is an intern consultant at Marktechpost. He is pursuing an integrated dual degree in Materials at the Indian Institute of Technology, Kharagpur. Nikhil is an AI/ML enthusiast who is always researching applications in fields like biomaterials and biomedical science. With a strong background in Material Science, he is exploring new advancements and creating opportunities to contribute.