[600MRG] WSPRnet database

[microcode at zoho.com]

Hi,

On Fri, Oct 31, 2014 at 09:06:05AM -0700, Larry wrote:
> 
> On 31-Oct-14 1:05 AM, microcode at zoho.com wrote:
> >
> >No, as far as I saw in my testing, OO has the same 1 mega-record limit as
> >the other utilities we've been talking about. That's why I suggested
> >sqlite. If you want to do ad-hoc queries it's perfect. If you can find a GUI
> >it will make life much simpler for individuals. If there is a web-enablement
> >add-on then it can serve the same role as suggested to be filled by SQL
> >server, but be much easier to set up and lighter on resources.
> >
> >Even if not, it would be much easier for individuals to download a single
> >sqlite database that works on every platform that contains everything and is
> >manageable without slicing and dicing. As it is with the CSV much additional
> >processing is required before you can even get started looking at what you
> >want. The problem with Excel and OO etc. is not the monthly size it is a
> >general problem of your query returning more than one mega rows.
> >
> 
> 
> I don't think I understand exactly what you are trying to tell me.
> 
> Yes, 64 bit implementations of Open Office and Microsoft Office have a 1
> mega row limit (32 bit implementations much less). That's a given and the
> reason this discussion started.

I have been talking about 32 bit OO and Libre Office, which is a fork of
OO. You suggested OO after I had already posted my findings Libre Office has
the same 1 megarow limit as Microsoft Office. My point was only that OO is
not any better than Libre Office, ie is not a solution for those issuing
queries that return more than that number of rows.  

> But by using the "grep" and/or "find" commands, we can pre-process the 6
> mega row monthly file and strip out the 630 meter data leaving us with a CSV
> file with only a half million rows, well within the 1 mega row limit.

True. Now what if monthly samples are not the only subject of interest? I had
understood that people want to run various queries and if any of those
exceed the limit than the Microsoft, Open Office, and LibreOffice won't work
for them.

> Are you suggesting that the spreadsheet sort/count process will add rows and
> increase the data size beyond the application limits? That's not the process
> I envisioned but maybe it is necessary with the bloated inefficient
> applications we have these days.

No, I didn't mean that at all. I was just thinking ahead that eventually if
somebody wants something more than filtering a month's worth of data, or
when a month's worth of data gets beyond the 1 megarows limit they all seem
to have copied from each other than we're SOL if we have decided to rely on
tools that have fixed limits. So we should probably get away from the
spreadsheet idea and think about something light that can:

a) provide people with a way download the data in one usable chunk and not
be required to export or convert it further unless desired;

b) support ad-hoc queries so the guy can slice and dice the data however he
wants and not how somebody else thinks he wants

> I agree that using SQL of some sort (SQLite, MySQL, etc) would be the
> correct way to do the analysis. But very few of us 630 meter guys have the
> resources to pull that off and those that do are seemingly not willing to
> get involved. I for sure can not spend the time and effort required to learn
> all this (or re-learn as the case may be) and won't even consider installing
> all this database stuff on my desktop.

I only suggested sqlite as an alternative to SQL Server and it's a good
alternative to MySQL, Postgres, etc because of the stuff I mentioned
before. I realize from reading the posts over the last couple of days my
post didn't show up in anybody's mailbox but yours and mine ;-) 

My suggestion also addresses your concern about "installing all this database
stuff on my desktop" because sqlite is one executable and can even be run
from a USB stick. It's a zero-install, as I said. I share your concerns
about installing tons of crap and all that goes with it. This ain't
it. sqlite is something that doesn't require much processing power. The
issues are only if people are willing to fool around with SQL language to
extract the data they want (and it's not that hard to do basic queries) or
whether they want a graphical interface that helps with that somewhat but
not entirely (if we can find one.) I am coming from the angle of suggesting
a hammer that is just big-enough and not coming from the bigger hammer angle.

> So, as far as I can see, if it's not able to be done on a desktop
> spreadsheet, it is not going to be done at all.

Well I think something like sqlite would alleviate most of the need for a
spreadsheet but still allow you to use one if you wanted to manipulate the
data there instead of directly in sqlite. For what we have been talking
about sqlite seems like the lowest impact solution that meets all the
requirements and doesn't lock people into to artificial limits that will
cause this discussion to come up again in some number of months, years, etc.

People have said they are not interested/willing/etc. to learn how to use
spreadsheet macros and I would guess these same people are not willing to
learn how to use SQL either. Basically for those people they're going to
have to rely on some fixed form of reports that other people are willing to
provide for them. Because the only way to get the data presented the way you
want is to use some tools to select, sort, etc. They are not hard to learn.