The Automatic Processing of Ionospheric Data: New Possibilities Even with Old Data?

: The automatic interpretation of ionograms provides an opportunity to establish a large quantity of data at reduced costs and also makes it possible to recover measurements that could otherwise be lost to posterity. The quality of the automatically obtained data is still not completely defined, which induces comparison with the traditional method conducted by experienced operators.


Introduction
In 1925, Edward Appleton together with his student Miles Barnett performed a series of experiments that demonstrated the "existence of a layer of electrically charged particles" capable of reflecting radio waves, now known as the ionosphere. In the following year Breit and Tuve came to the same conclusions in completely independent experiments (Bible, 1998). These experiments used two different techniques to demonstrate the existence of the ionosphere and to measure its height, inventing what is now known as an ionosonde. The first technique is called the "chirp method" and the second the "pulse method" (Bible, 1998). These two alternative techniques are still used in radar applications and the ionosonde can effectively be considered the first radar ever developed. Over the following years radar was greatly developed due to the military importance of radio detection of targets.
In subsequent years, the ionosonde technique was refined to the point of recording a graph t(f) of the time t elapsed for a radio frequency f to travel the ionosphereearth-ionosphere path onto a film, the result being called an ionogram. It is assumed that the first ionogram was recorded in the station of Slough (UK), December 27, 1933 at 10.30 (Rishbeth, 1991).
The prediction of usable frequencies for radio links became extremely important during the Second World War and ionospheric soundings were conducted in great numbers during the war years. Users became familiar with the characteristic shapes of the ionogram traces, shapes that depend on the physical state of the ionosphere and in particular on its vertical electron density profile N e (h). The relative maxima in N e (h) were highlighted by cusps in the traces, showing regular diurnal and seasonal variations as well as the eleven-year sunspot cycle.
However, ionograms often exhibit poorly defined features, with continuous transitions between one physical state and another. The effects of perturbative phenomena and various other irregularities are also observed, giving rise to multiple traces or spread echoes. These kind of phenomena immediately made it clear that the interpretation of an ionogram is to some extent subjective.
There was a strongly felt need for standardization and so during the International Geophysical Year (IGY) 1957-58, "The World Wide Soundings Committee" produced rules and conventions to be applied when deriving the physical quantities from ionograms, which were then to be published in bulletins. In 1961, the Union Radio-Scientifique Internationale (URSI) took steps to implement this set of rules, publishing the first URSI Handbook of Ionogram Interpretation and Reduction (Piggott and Rawer, 1972). In principle, all the data stored in worldwide databases and used for the development of models up to now, were obtained from ionograms on the basis of the rules set out in this manual.
In the meantime, developments in digital electronics brought to an end the era of analog ionosondes and ionograms recorded directly on film. Various models of digital ionosondes were produced and marketed. Initially, these provided output in paper form, but with the development of information technology this was rapidly replaced by output of a data file, processed and stored on static memories.

The Current Situation of Ionospheric Data
In parallel with the above developments, modern technological applications have highlighted the need for real-time monitoring of the rapid evolution of certain phenomena, like ionospheric storms. In response to these newly emerging needs, in the 1980s the Center for Atmospheric Research (UMLCAR) of the University of Massachusetts Lowell, produced the Automatic Real-Time Scaling Ionogram with True-height (ARTIST) system, capable of automatic interpretation of ionograms and also providing N e (h) (Reinisch and Huang, 1983). The problem of automatic ionogram interpretation is highly complex and consequently the only other widely tested system alongside ARTIST to date is the Autoscala program (Scotto and Pezzopane, 2002), produced by the Istituto Nazionale di Geofisica e Vulcanologia group in Italy. The quality of the data produced by these programs is generally satisfactory for real-time ionospheric monitoring applications, but the reliability of the data is generally considered inferior to that produced by skilled human operators (Pezzopane and Scotto, 2005).
Consequently, the current situation of ionospheric data is that there is accurate (manually interpreted) historical data, plus a large quantity of recent data produced automatically. It is also important to note that the 4 world data centers have around 200 million ionograms stored on paper or film, not digitalized, accounting for up to 60 years of ionospheric climatology and meteorology. Of these only 25% were interpreted and reduced to standard features and only a very small percentage of them were used to subsequently obtain N e (h) (Wright and Conkright, 2002). Without a concerted effort to scan the images into a suitable file format, followed by automatic ionogram interpretation, this large volume of information is likely to be lost to posterity, also due to the degradation of the paper and film media. This large quantity of information could be useful, among other things, for scientific studies such as the observation of long-term trends in the ionosphere. In some circumstances, this type of study has failed to reach definitive conclusions as a consequence of scarcity of available data (Scotto, 2013).

Conclusion
In general, while automatic interpretation is the only possibility for real-time applications, the question arises whether the large amount of data produced automatically can also be used for scientific purposes. Using programs for automatic interpretation can generate considerable amounts of data, with excellent continuity and without the subjectivity introduced by an operator. These aspects certainly deserve due consideration, also bearing in mind the scarcity of funds for research, especially in Western countries.
Manual data, on the other hand, is still more accurate, the human brain exhibiting an, as yet, unsurpassed capacity to appropriately recognize fading and evanescent traces, dispersed echoes and where necessary extrapolating traces to obtain the correct values for the ionospheric characteristics.
Whether the less accurate, but more abundant, realtime data could replace manual data for scientific applications is an issue that needs be discussed by the ionospheric community. This could also be a way to recover the enormous amount of historical data currently available on paper or film, which may contain useful information to answer questions extending beyond the interest of ionospheric science.