Enterprise AI Analysis

The fine-tuned LLM (FT-LLM) demonstrates a remarkable learning efficiency, achieving accuracy comparable to optimized deep neural networks (DNNs) for forward prediction, particularly on larger datasets (1,000 samples and above). In terms of Mean Absolute Relative Error (MARE), the FT-LLM outperforms all traditional machine learning models and achieves similar accuracy to DNNs across various dataset sizes.

However, for Mean Squared Error (MSE), the FT-LLM performs worse than other models in low-data scenarios (<10,000 samples). Its accuracy improves rapidly with larger datasets, eventually surpassing all baseline models except the DNN at 40,000 samples, and even showing a faster improvement rate, suggesting it could surpass the DNN with more data. This metric discrepancy is attributed to MARE's sensitivity to errors in low-value regions versus MSE's uniform performance across spectral magnitudes.

The influence of prompt templates on predictive accuracy was found to be minimal, with both concise vector and detailed descriptions yielding similar results. Temperature settings during inference showed that lower temperatures generally yield better MSE for larger training datasets, ensuring more consistent and probable outputs.

Despite its impressive predictive capabilities, the FT-LLM did not demonstrate significantly better interpretability than the original, unmodified ChatGPT 3.5 model. When asked to explain the physical principles behind metamaterial responses to geometric changes, both fine-tuned and un-trained LLMs provided similar, general responses. The fine-tuning on geometry-spectra pairs did not provide the model with a deeper, actionable understanding of underlying physical relationships.

A stylistic difference was observed, with the FT-LLM generating responses in single paragraphs, likely influenced by the numerical list format of its training data. This suggests that while LLMs can generate text, their 'understanding' of physics from purely numerical input via textual encoding is limited without more diverse, contextual training data (e.g., research papers or textbooks related to physics explanations).

Fine-tuning ChatGPT 3.5 for this research incurred substantial computational costs, estimated at over $4,000 for all experiments conducted via the OpenAI API. This high cost is a significant barrier for academic researchers and highlights the resource-intensive nature of working with large language models, even for task-specific fine-tuning.

The prompt template's influence on prediction accuracy was minimal, indicating that the LLM's ability to process numerical data, once encoded, is robust regardless of whether the input is a concise vector or a detailed description. However, temperature settings are crucial for larger datasets: lower temperatures (e.g., 0.5) tend to minimize MSE by reducing output variability and selecting more probable predictions, which is important for precise scientific applications.