IWETHEY v. 0.3.0 | TODO
1,095 registered users | 1 active user | 1 LpH | Statistics
Login | Create New User

Welcome to IWETHEY!

New Pivoting data for gnuplot
Measuring some aspect of system behaviour results in a CSV file containing data of the form:

dd-mmm-yyyy hh:ss:mm,process_name,some_metric1,some_metric2

What I want to do is make a line chart for some_metric1 over time, categorised by process_name. This will give me a lovely graph with lots of coloured lines, one per process.

I'd do this in Excel (man, I love me a PivotTable), but I'm bouncing off performance issues because my source data is 325K lines, and Excel ain't built to chart this kind of volume of data, at least not the way I'm doing it. Also, there's a manual import issue and I'd like to eliminate that.

I looked at R and then quickly got over myself (it's weird and I don't have time to learn it) and remembered about gnuplot, which seems entirely suited to the task at hand.

But I can't figure out how to pivot the data to provide multiple data series (i.e. the metrics for processes over time) on a single chart.

Clue me?
New I'm probably the only person on the planet who...
doesn't think in terms of "pivot"s and so forth.

I use gnuplot a lot, but haven't done what you're asking.

Maybe this? http://blog.lick-me.org/ (search for "pivot") will help?

Good luck.

New Great URL :-/

New You really do have zero impulse control
public static final drook(webpage)
timesink = getalllinks(webpage);
for(link l in timesink)
new webpage wp = follow(l);
Or something. Anyway, just close the goddamn window and get back to work, wage slave.
Expand Edited by pwhysall Feb. 14, 2014, 12:46:58 PM EST
New Nope, didn't click this one

Expand Edited by drook Feb. 14, 2014, 12:54:40 PM EST
New I honestly hadn't paid much attention to it.
It does have programming information though and nothing offensive jumped out at me.


31 DEC 12

Creating graphs in LibreOffice is a nightmare. They're ugly, nearly impossible to customize and creating pivot tables with data is bloody tedious work. In this post, I'll show you how I took the output of a couple of performance test scripts and turned it into reasonably pretty graphs with a few standard command line tools (gnuplot, awk, a bit of (ba)sh and a Makefile).

The Data

I ran a series of query performance tests against data sets of different sizes. The sets contain 10k, 100k, 1M, 10M, 100M and 500M documents. One of the basic constraints is that it has to be easy to add/remove sets. I don't want to faff about with deleting columns or updating pivot tables. If I add a set to my test data, I want it automagically show up in my graphs.

The output of the test script is a simple tab separated file, and looks like this:



New Easily done in a perl multi-level hash
Suck all the data in, bucket it via the desired output keys, spit it out via different sorts.

If you want to give me a sample data file and a spec, happy to code it for you.
New Thanks, but I can't
Commercially confidential and all that.

I'll have a prod at the method in Python, though.
New yup. python easy as well for this.
     Pivoting data for gnuplot - (pwhysall) - (8)
         I'm probably the only person on the planet who... - (Another Scott) - (4)
             Great URL :-/ -NT - (drook) - (3)
                 You really do have zero impulse control - (pwhysall) - (1)
                     Nope, didn't click this one -NT - (drook)
                 I honestly hadn't paid much attention to it. - (Another Scott)
         Easily done in a perl multi-level hash - (crazy) - (2)
             Thanks, but I can't - (pwhysall) - (1)
                 yup. python easy as well for this. -NT - (crazy)

Ugh. Too bony for soup.
70 ms