News and Notes

The Iowa Environmental Mesonet (IEM) collects environmental data from a number of different "networks". The term "network" is ambiguous and there are various conventions out there for what it means. For the IEM, the term is used to attempt to group together similar data collection platforms within some common geography. The most common grouping is by state. For example, "IA_ASOS" represents a network of airport weather stations maintained by the FAA/NWS within Iowa. Presently, the IEM has 527 networks labelled, gasp!

The most problematic network labeling exists between what the IEM calls "COOP" and "DCP". These terms are used by the National Weather Service (NWS), but not in the same way the IEM uses them. Oh this gets confusing. The IEM processes a large fire hose of SHEF formatted products that use a station identifier that do not easily translate back into robust metadata indicating where the data is coming from. For example, the IEM has a page listing out station identifiers used that have unknown metadata. Some of the station identifiers have an embedded code that properly associates the site with some geography, thankfully those are almost always right.

But even when the station has metadata, they are mapped back to a NWS "Program Acronym" that can be sometimes inaccurate, vague, or dual listed. For example, some identifiers are shared between disparate observation platforms. From the IEM's processing end, all we see is the raw data reports associated with a station identifier. The frequency and type of reports are used to infer what type of platform they are. For example, humans typically do not measure river stage every 15 minutes!

For most end-users of this SHEF data, they are either interested in reports coming from the once-daily COOP human observers or in the reports coming from everywhere else! For that purpose, the IEM groups the stations into either "COOP" or "DCP" where "DCP" is the label covering everything that is not-COOP. But again, there is no definitive and accurate database showing which identifiers go with which networks.

So over the past 18 years, the IEM has written various buggy codes and decisions algorithms attempting to assign networks to these station identifiers. This has yielded many false positives and stations that are improperly labelled as COOP. Thankfully, a few NWS offices have kindly supplied local office listing of their COOP sites, which have been merged into the IEM metadata database to provide better listings.

The purpose of this news item is to document this fun and provide a reference point for others to review. I am always happy to discuss this situation with interested folks and would love to improve the accuracy of the station / network metadata.