Machine Learning Models for Short-Term Solar Power Forecasting

Introduction

Solar energy. Electricity for our personal use that is harvested from radiation from our very own star. Solar energy is increasingly critical and becoming more and more integrated into our lives, with 7% of homes in the United States having solar panels as of May 2024, a number that’s set to double by 2030 and triple by 2034. Beyond its merits solar power has a fundamental issue that has yet to be addressed: its consistency. Clouds, weather patterns, and everyday cycles cause fluctuations in energy production rate of solar panels that challenge grid stability and even market efficiency. Therefore, researchers have turned to the newest generation of research technology - artificial intelligence and machine learning - for assistance in forecasting solar output with accuracy. However, there are still significant engineering challenges before these models can be properly implemented into the field at a large scale.

Challenges of Solar Intermittency

Why the issue exists

In order to understand why solar forecasting matters, we need to understand what solar variability really is and its consequences. Solar irradiance - the intensity of sunlight hitting the Earth’s surface - is constantly changing due to cloud movement, atmospheric conditions, and daily rotation of the planet. The “duck curve” is one of those challenges. During midday, power floods the grid, but in the evening, when the power demand is highest, solar output drops. Managing these swings in output requires expensive operational fixes. Because of the lack of ability to anticipate changes in solar output, generators must always maintain expensive reserves that always have to be available for use at a moment’s notice in case output drops.

Who the issue affects

Intermittency in solar power causes what is known as a “duck curve,” where solar power floods during midday and noon but then rapidly drops in the evening when electricity demand is at its highest. Maintenance of these ramps requires the use of very accurate predictive systems or very costly systemic adjustments. Speaking of costly, the economic implications can’t be ignored. NASA’s Space Launch System (SLS) is estimated to take around $2 million to launch, yet solar forecasting inaccuracies cause comparable losses. Grid operators must maintain expensive reserve capacity and on top of that are penalized severely when the forecasted energy output differs from the power produced. 

The impact, of course, extends beyond just those who work within the grid. A 2024 study in Nature Communications notes that accurate forecasting enables for “a more effective allocation of operating reserves than conventional deterministic approaches, highlighting the potential of probabilistic machine learning to enhance market efficiency and grid stability in increasingly decarbonized power systems." In simpler terms, better forecasting doesn’t just reduce costs but also makes the entire energy system more economically efficient and stable. 

Forecasting Methods: Statistics to Machine Learning

For decades, people have used statistics to predict solar power production. However, regression lines and equations simply don’t have the power to safely and consistently predict outcomes outside the domain of the experiment: that is, it can’t properly predict/capture something that hasn’t happened yet based on a trend because of the complex and non-linear relationship between solar output and weather. Searching for other options to predict solar power output, researchers looked to machine learning.

Machine learning directly uses the data that it’s trained on to predict future outcomes based on a vast amount of variables, making it much better at recognizing patterns and predicting trends than statistics are. It can also recognize patterns and relationships that humans may miss, being able to predict protein structure, used in early disease detection, identify previously overlooked hypotheses by linking scientific research together, and so much more. Scientists used models such a XGBoost, LightGBM, and CatBoost. However, machine learning still struggles with extreme weather events due to a lack of training data, as models generally require a large amount of data in order to perform analyses with accuracy.

Deep Learning Architectures (Most of info?)

To better manage solar power’s complexity, researchers began to use a deep learning - neural networks based on how brains work - method called LSTM (Long Short-Term Memory). 

Typical Neural Networks struggle with retaining information when they are processing data, but LSTMs use memory cells that decide what information the model will keep and forget. This allows them to learn patterns from much larger datasets. Specifically, “in solar power time series forecasting, the LSTM model has outperformed the MLP algorithm in all major metrics. 

While LSTMs are incredibly useful, they aren’t enough. They can accurately predict the solar output over time but can’t account for variables such as moving cloud cover. To remedy this, researchers combined LSTMs and CNNs (Convolutional Neural Networks), which are great for extracting patterns in image/visual data. Together, the models can predict solar output through space and time. One LSTM and CNN hybrid model “achieved impressive results with an R2 score of .979 for 1-hour forecasts, demonstrating near-perfect accuracy by accounting for 97.9% of the variation in data. 

Why this matters

With accurate forecasts, grid operators can simply plan ahead rather than being forced to react to the results. During the 2024 solar eclipse, ERCOT used ML forecasts to predict the sudden drop in solar output. Then, because they knew what was coming, they adjusted their backup power in advance, preventing grid problems. Now, the best forecasting models have improved accuracy for predictions by 25% compared to the older models. For solar companies, accuracy matters financially, and with forecasts hitting up to 98% accuracy, financial penalties and losses for companies have dropped dramatically.

Implementation Challenges

While there has been significant progress, challenges still present themselves. Deep learning models still require lots of quality training data to produce accurate results. Most solar data comes from sunnier regions, like California. Using data from California to train a model to predict solar output in a cloudy region, like Germany, won’t work too well. Go into how deep learning is a black box here. Write on the price of investing in artificial intelligence training.

At the moment, the most practical solution is to use multiple forecasting methods together. Satellites handle the next 3 hours, machine learning handles the medium-term forecasts, and weather models are used to longer predictions. This way efficiency and cost are balanced and the results are better than relying on any single method.

Future Work/Outlook

Solar forecasting has improved dramatically. Statistical methods weren’t working, so we turned to deep learning, which has worked better at nearly 98% accuracy. With cheaper grid operations, lower penalties for solar companies, and more efficient markets, the benefits are very clear. On the other hand, however, it is essential to note that obstacles still remain with data bias, interpretability, and the cost of training and maintaining models. For now, hybrid approaches make the most sense. In the future, these limitations will likely shrink as models become more powerful and machine learning will become cheaper, becoming the norm for solar forecasting.

Works Cited

Nature Communications. "Probabilistic Day-Ahead Forecasting of System-Level Renewable Energy and Electricity Demand." Nature Communications, vol. 17, 28 Feb. 2026, www.nature.com/articles/s41467-026-69015-w.

PLOS One. "Solar Energy Prediction Through Machine Learning Models: A Comparative Analysis of Regressor Algorithms." PLOS One, 2 Jan. 2025, journals.plos.org/plosone/article?id=10.1371/journal.pone.0315955.

ResearchGate. "Forecasting of Photovoltaic Solar Power Production Using LSTM Approach." ResearchGate, Apr. 2020, www.researchgate.net/publication/340265680_Forecasting_of_Photovoltaic_Solar_Power_Production_Using_LSTM_Approach.

Scientific Reports. "Hybrid Deep Learning CNN-LSTM Model for Forecasting Direct Normal Irradiance: A Study on Solar Potential in Ghardaia, Algeria." Scientific Reports, vol. 16, 2 May 2025, www.nature.com/articles/s41598-025-94239-z.

Scientific Reports. "Hybrid Deep Learning Models for Time Series Forecasting of Solar Power." Scientific Reports, vol. 15, 22 Feb. 2024, link.springer.com/article/10.1007/s00521-024-09558-5.

Scientific Reports. "Short-Term and Long-Term Solar Irradiance Forecasting with Advanced Machine Learning Techniques in Zafarana, Egypt." Scientific Reports, vol. 16, 12 Nov. 2025, www.nature.com/articles/s41598-025-24853-4.