[600MRG] WSPRnet database

[John Langridge]

Larry,

The CSV record filesize is a nightmare.  It might not be so bad if the breakdown were by band, by day, etc.  I had similar aspirations in the past but found that the processing load on the machines I have here was such that the machines would lock up when trying to convert the CSV to relational database table(s).  In short, its a tough go to use MS Access to write SQL to grab desired data and build specific tables with that many records and consumer grade PC junk.


Someone with some horsepower and an implementation of SQL Server would be the way to go.  You could easily come up with stored procedures and the queries to get just about anything you wanted and it would run pretty quickly particularly with good indexing.   The key is getting the CSV data into a relational database to play with and both access and SQL server have facilities to accomplish that exact task.

Perhaps someone here with some drive space and machine horse power could load a database with the parsed csv data on a regular basis.  There are some good web query tools out there that would allow multiple user to access the data and write queries via a web portal so that many of us can look at the data in a variety of ways.  I'm not aware of what freeware might be available.  Perhaps there is an IT manager on here with access to the hard ware and software to get this done.  It would just have to be someone's pet project to do it... 

73 and maybe someone with interest in DP will jump on this as a project.  I think a lot of us would like to look at numbers outside of how they are presented on wsprnet.org.


John XIQ





From: Larry <larry at w7iuv.com>

To: 600 MRG <600MRG at w7ekb.com> 
Sent: Wednesday, October 29, 2014 12:07 PM
Subject: [600MRG] WSPRnet database
 

The 630 propagation over the last couple months has interested me quite 
a bit. I decided to do some analysis using the spot database that can be 
downloaded from WSPRnet.

Unfortunately the monthly CSV data files are now up to over 6 million 
spots each. None of the database tools I have available and know how to 
use will handle that size file.

At one time I think I remember being smart enough to write a script that 
would parse the data and create a useable file but that was decades ago 
and the old brain just ain't what it used to be.

Has anyone out there solved this problem? Any suggestions? Best solution 
would be an existing tool that would do the job.

Searching the WSPRnet forums and Google has not helped.

Larry - W7IUV / WH2XGP

_______________________________________________
600MRG mailing list
600MRG at w7ekb.com
http://w7ekb.com/mailman/listinfo/600mrg_w7ekb.com