NREL MIDC station network#
The Measurement and Instrumentation Data Center (MIDC) is operated by NREL and provides irradiance and meteorological data from a number of ground stations in the U.S. The stations vary in quality, with some stations measuring all three components with high-quality instruments and other stations featuring a rotating shadow band pyranometer.
The most notable station is the Baseline Measurement System (BMS) at NREL’s Solar Radiation Research Laboratory (SRRL) outside of Denver, Colorado. The BMS features the world’s largest collection of operating pyranometers and pyrheliometers. A number of sky imagers, PV reference cells, and spectral radiometers are also located at the site. Instruments at the BMS are cleaned each weekday and frequently calibrated. Due to the large collection of co-located and well maintained instruments, the BMS data is ideal for comparing different types of instruments.
Note, the MIDC includes several inactive stations. Also, several of the active stations are no longer cleaned or calibrated frequently. For these reasons, the SolarStations.Org catalog only includes the SRRL BMS, SOLARTAC, and Flatirons M2 sites, as these measures all three irradiance components and are active. See the map below for the locations of the stations.
---------------------------------------------------------------------------
RemoteDisconnected Traceback (most recent call last)
Cell In[1], line 6
3 init_notebook_mode(all_interactive=True)
5 stations_midc_url = 'https://midcdmz.nrel.gov/apps/data_api_doc.pl?_idtextlist_'
----> 6 stations = pd.read_csv(stations_midc_url)
7 stations = stations.rename(columns={
8 'STATION_ID': 'Station Identifier',
9 'STATION_FULLNAME': 'Station name',
(...)
13 'ELEVATION_M': 'Elevation',
14 'ACTIVE': 'Active'})
16 del stations['RESERVED']
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/pandas/io/parsers/readers.py:1026, in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, date_format, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, on_bad_lines, delim_whitespace, low_memory, memory_map, float_precision, storage_options, dtype_backend)
1013 kwds_defaults = _refine_defaults_read(
1014 dialect,
1015 delimiter,
(...)
1022 dtype_backend=dtype_backend,
1023 )
1024 kwds.update(kwds_defaults)
-> 1026 return _read(filepath_or_buffer, kwds)
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/pandas/io/parsers/readers.py:620, in _read(filepath_or_buffer, kwds)
617 _validate_names(kwds.get("names", None))
619 # Create the parser.
--> 620 parser = TextFileReader(filepath_or_buffer, **kwds)
622 if chunksize or iterator:
623 return parser
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/pandas/io/parsers/readers.py:1620, in TextFileReader.__init__(self, f, engine, **kwds)
1617 self.options["has_index_names"] = kwds["has_index_names"]
1619 self.handles: IOHandles | None = None
-> 1620 self._engine = self._make_engine(f, self.engine)
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/pandas/io/parsers/readers.py:1880, in TextFileReader._make_engine(self, f, engine)
1878 if "b" not in mode:
1879 mode += "b"
-> 1880 self.handles = get_handle(
1881 f,
1882 mode,
1883 encoding=self.options.get("encoding", None),
1884 compression=self.options.get("compression", None),
1885 memory_map=self.options.get("memory_map", False),
1886 is_text=is_text,
1887 errors=self.options.get("encoding_errors", "strict"),
1888 storage_options=self.options.get("storage_options", None),
1889 )
1890 assert self.handles is not None
1891 f = self.handles.handle
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/pandas/io/common.py:728, in get_handle(path_or_buf, mode, encoding, compression, memory_map, is_text, errors, storage_options)
725 codecs.lookup_error(errors)
727 # open URLs
--> 728 ioargs = _get_filepath_or_buffer(
729 path_or_buf,
730 encoding=encoding,
731 compression=compression,
732 mode=mode,
733 storage_options=storage_options,
734 )
736 handle = ioargs.filepath_or_buffer
737 handles: list[BaseBuffer]
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/pandas/io/common.py:384, in _get_filepath_or_buffer(filepath_or_buffer, encoding, compression, mode, storage_options)
382 # assuming storage_options is to be interpreted as headers
383 req_info = urllib.request.Request(filepath_or_buffer, headers=storage_options)
--> 384 with urlopen(req_info) as req:
385 content_encoding = req.headers.get("Content-Encoding", None)
386 if content_encoding == "gzip":
387 # Override compression based on Content-Encoding header
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/pandas/io/common.py:289, in urlopen(*args, **kwargs)
283 """
284 Lazy-import wrapper for stdlib urlopen, as that imports a big chunk of
285 the stdlib.
286 """
287 import urllib.request
--> 289 return urllib.request.urlopen(*args, **kwargs)
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/urllib/request.py:215, in urlopen(url, data, timeout, cafile, capath, cadefault, context)
213 else:
214 opener = _opener
--> 215 return opener.open(url, data, timeout)
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/urllib/request.py:515, in OpenerDirector.open(self, fullurl, data, timeout)
512 req = meth(req)
514 sys.audit('urllib.Request', req.full_url, req.data, req.headers, req.get_method())
--> 515 response = self._open(req, data)
517 # post-process response
518 meth_name = protocol+"_response"
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/urllib/request.py:532, in OpenerDirector._open(self, req, data)
529 return result
531 protocol = req.type
--> 532 result = self._call_chain(self.handle_open, protocol, protocol +
533 '_open', req)
534 if result:
535 return result
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/urllib/request.py:492, in OpenerDirector._call_chain(self, chain, kind, meth_name, *args)
490 for handler in handlers:
491 func = getattr(handler, meth_name)
--> 492 result = func(*args)
493 if result is not None:
494 return result
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/urllib/request.py:1392, in HTTPSHandler.https_open(self, req)
1391 def https_open(self, req):
-> 1392 return self.do_open(http.client.HTTPSConnection, req,
1393 context=self._context)
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/urllib/request.py:1348, in AbstractHTTPHandler.do_open(self, http_class, req, **http_conn_args)
1346 except OSError as err: # timeout error
1347 raise URLError(err)
-> 1348 r = h.getresponse()
1349 except:
1350 h.close()
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/http/client.py:1430, in HTTPConnection.getresponse(self)
1428 try:
1429 try:
-> 1430 response.begin()
1431 except ConnectionError:
1432 self.close()
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/http/client.py:331, in HTTPResponse.begin(self)
329 # read until we get a non-100 response
330 while True:
--> 331 version, status, reason = self._read_status()
332 if status != CONTINUE:
333 break
File /opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/http/client.py:300, in HTTPResponse._read_status(self)
296 print("reply:", repr(line))
297 if not line:
298 # Presumably, the server closed the connection before
299 # sending a valid response.
--> 300 raise RemoteDisconnected("Remote end closed connection without"
301 " response")
302 try:
303 version, status, reason = line.split(None, 2)
RemoteDisconnected: Remote end closed connection without response
Data retrieval#
Data from the MIDC can be retrieved from the MIDC website or using the MIDC raw data API.
Note
If you use data from the MIDC in any publication, make sure to cite it. As an example, the citation for the BMS site is:
Andreas, A.; Stoffel, T.; (1981). NREL Solar Radiation Research Laboratory (SRRL): Baseline Measurement System (BMS); Golden, Colorado (Data); NREL Report No. DA-5500-56488. http://dx.doi.org/10.5439/1052221
Data can be downloaded and parsed conveniently using the pvlib-python iotools module using the function pvlib.iotools.read_midc_raw_data_from_nrel. If you use pvlib iotools for published work, please Jensen et al. (2023) which provides additional background information. The use of the function is shown below, demonstrating how to retrieve five days of data:
import pvlib
data = pvlib.iotools.read_midc_raw_data_from_nrel(
site='BMS', # station identifier
start=pd.Timestamp(2020,6,1),
end=pd.Timestamp(2020,6,5))
# show a subset of the data
show(data.iloc[:5, 5:10], dom="tpr")
C:\Users\arajen\Anaconda3\envs\solarstations\lib\site-packages\scipy\__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.3
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
| CR3000 Zen Angle [degrees] | Global LI-200 [W/m^2] | Global CMP22 (vent/cor) [W/m^2] | Global RG780 PSP [W/m^2] | Global CM3 (cor) [W/m^2] | |
|---|---|---|---|---|---|
| Loading... (need help?) |
The retrieved BMS dataset contains numerous instruments measuring the same irradiance component. Let’s, for example, compare the global horizontal irradiance (GHI) measured by a high-quality CMP22 pyranometer with that of a low-cost CM3 pyranometer:
Show code cell source
import matplotlib.pyplot as plt
fig, axes = plt.subplots(ncols=2, figsize=(10,4))
# plot both measurement as a time-series
data[['Global CMP22 (vent/cor) [W/m^2]', 'Global CM3 (cor) [W/m^2]']].plot(
ax=axes[0], alpha=0.8, ylim=[-20, 1500])
# compare measurements with a scatter plot
data.plot.scatter(ax=axes[1], s=1, grid=True,
x='Global CMP22 (vent/cor) [W/m^2]',
y='Global CM3 (cor) [W/m^2]',
xlim=[-20, 1300], ylim=[-20, 1300])
fig.tight_layout()
References#
Adam R. Jensen, Kevin S. Anderson, William F. Holmgren, Mark A. Mikofski, Clifford W. Hansen, Leland J. Boeman, and Roel Loonen. Pvlib iotools—open-source python functions for seamless access to solar irradiance data. Solar Energy, 266:112092, 2023. doi: 10.1016/j.solener.2023.112092.