Post #391,127
6/24/14 3:21:24 PM
|
So, PivotTables are great.
They're a rapid and effective way of summarising and (via the PivotChart tool) visualising data.
However.
I've got a lot of data.
When I say "a lot", I mean "eight hundred and ninety thousand rows". A month. For a fragment of the whole dataset. And I want to do some sums on the entire year. I'm really talking about tens of millions of datapoints.
So, Excel ain't it. Neither is Access (but then, whatever the question was, I'm not sure that "Access" was ever the answer).
What do folk munge big datasets with? Bear in mind this is for work, so corporate-friendly output is a must.
|
Post #391,130
6/24/14 3:26:46 PM
|
Tangent, I still like Access
It's a fabulous rapid prototyping tool. I've pointed it at tables in SQL Server on the network and ginned up reports and data entry screens in minutes, then handed that off to a web developer as a UI spec for a web app that would handle things like concurrency, thread safety, scalability.
|
Post #391,134
6/24/14 3:43:02 PM
|
key phrase
"pointed it at tables in SQL server on the network"
It's a pile of dung for local storage.
Might give it a go, though. Watching Excel seize up is getting old, fast.
|
Post #391,137
6/24/14 4:01:07 PM
|
Oh yeah, the MDB sucks donkey wang
When you use the SQL Server tables I don't know where the search logic actually executes, but I was joining multiple tables with millions of rows each and getting results back sub-second.
|
Post #391,139
6/24/14 4:10:14 PM
|
It depends on the ODBC connection...
Since it is a Micrsoft product, the sorts/queries mostly hit the DB.
For things like Oracle, MySQL or PostGreSQL it is a mixture.
Sometimes it is painfully slow... since Access will use the LEAST capable SQL queries... due to "safety." We sped up a query/Join by manually editing it for a customer. Result went from several hours for a report to run clawing through millions and millions of records... sorted locally and manipulated locally. To results from a much smarter initial query with everything returned in the right format plus reduced the return rows to only those needed... reduced it to about 30 seconds fully rendered and "playable"
This was using a SSH tunnel with s local port forward on the "server" using the built in Access MySQL connector. --
greg@gregfolkert.net
"No snowflake in an avalanche ever feels responsible." --Stanislaw Jerzy Lec
|
Post #391,229
6/27/14 11:03:14 AM
|
Synchronicity
|
Post #391,230
6/27/14 11:31:27 AM
|
"You'd write email in excel if you could figure out how to do it in a font you like."
Any opinions expressed by me are mine alone, posted from my home computer, on my own time as a free American and do not reflect the opinions of any person or company that I have had professional relations with in the past 59 years. meep
|
Post #391,142
6/24/14 8:18:30 PM
|
Corporate friendly output
A while back I looked at Gephi to visualize the impact of BYOD on our Exchange server. Unfortunately, other things became more pressing so I never got beyond scratching the surface.
At the base level, Gephi is not a data crunching tool. For that, you do need to look elsewhere. However, once you have reduced your data to the dimensions you want, this thing is a lot of fun.
|
Post #391,175
6/25/14 2:45:54 PM
|
Do you have an Oracle instance you can load your data into?
|
Post #391,177
6/25/14 2:57:12 PM
|
Not easily available.
Might be doable with a couple of days' lead time, though...
|
Post #391,178
6/25/14 3:03:01 PM
|
Have you used Oracle in the past?
|
Post #391,180
6/25/14 3:06:45 PM
|
Yeah.
About 14 years ago :)
I do, however, have a tame DBA who's not massively busy right now.
|
Post #391,181
6/25/14 3:23:02 PM
|
Very important to gather stats.
|
Post #391,179
6/25/14 3:06:04 PM
|
Also, do you have any pictures of kittens?
Because the scary Oracle man is still looking at me, even after I clicked "back".
|
Post #391,183
6/25/14 4:27:31 PM
|
you have splunk in the house?
Any opinions expressed by me are mine alone, posted from my home computer, on my own time as a free American and do not reflect the opinions of any person or company that I have had professional relations with in the past 59 years. meep
|
Post #391,193
6/26/14 2:23:32 AM
|
Eeenteresting.
Will have a look, thanks.
|
