IWETHEY v. 0.3.0 | TODO
1,095 registered users | 0 active users | 0 LpH | Statistics
Login | Create New User
IWETHEY Banner

Welcome to IWETHEY!

New trying to get a running total, not increment
sorry bad explanation of what I was trying to do. Im trying to add up the storage used in a particular directory of all files over 7 days old. There will be hundreds of millions of them,
If we torture the data long enough, it will confess. (Ronald Coase, Nobel Prize for Economic Sciences, 1991)
New Here, feed this filenames.
Yes, I can do it in 1 line in your AWK example. Or in Perl. But I won't unless required, very few people can read, understand, and then CHANGE one liners, so they are landmines waiting to be stepped on.


##########################################
#!/usr/bin/perl -w
#
use strict;

my $size = 0;
while(<>){
chomp;
next unless (-f $_);
my $file_size = -s $_;
$size += $file_size;
}
print "Total: $size\n";
New I dont quite get this
if I feed it filenames how does it determine the size?
If we torture the data long enough, it will confess. (Ronald Coase, Nobel Prize for Economic Sciences, 1991)
New -s
Perl has most of the unix scripting test operators:
http://perldoc.perl....functions/-X.html

This checks to see if it is a regular file and goes to the top of the current loop if not:
next unless (-f $_);

This declares a variable and assigns it the size of the filename contained in the default operator.

my $file_size = -s $_;
New Ah, got it. thanx!
If we torture the data long enough, it will confess. (Ronald Coase, Nobel Prize for Economic Sciences, 1991)
New That was all an inside joke, right?
There weren't enough characters there to actually do anything useful.
--

Drew
New no works as advertized
If we torture the data long enough, it will confess. (Ronald Coase, Nobel Prize for Economic Sciences, 1991)
New Sigh
Must be pebkac.

Current dir:
$ l
total 12
-rwxr-xr-x 1 broom broom 162 2010-03-26 19:55 add_file_size.pl
-rwxr-xr-x 1 broom broom 608 2010-03-26 19:55 make_self_contained.pl
-rwxr-xr-x 1 broom broom 657 2010-03-26 19:55 maketests.pl

Execute it:
$ find . -print | ./add_file_size.pl
Total: 1427

Figure what the sizes should be:
$ bc
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
162+608+657
1427

It matches.

The script file contains the following:

$ cat add_file_size.pl

#!/usr/bin/perl -w
#
use strict;

my $size = 0;
while(<>){
chomp;
next unless (-f $_);
my $file_size = -s $_;
$size += $file_size;
}
print "Total: $size\n";


New sorry, no, works as advertized. My bad forgot comma
If we torture the data long enough, it will confess. (Ronald Coase, Nobel Prize for Economic Sciences, 1991)
New Don't show it to Ben.
He'd do it in one line. More line noise than characters. And it would be brilliant.
New Or like this... (using find)
find /my/dir -ctime +7 -type f -name "*?b*" -printf "%sn" | paste -sd+ - | bc

-ctime +7: matches everything that was changed > 7*24 hrs ago.
-printf %sn: size in bytes, one per line
-name "*?b*": replaces fgrep .b

paste -sd+: replaces the line breaks with '+' signs

bc: execute the sum

I don't have a directory with hundreds of millions of files in it, so I can't how well it will scale. It goes through 7000 files nested ~3 levels deep in a blink.
New Yup, find is magic
I know sysadmins who will craft the most amazing find scripts.
Hate them.
Hate them.

The scripts, not the admins.

I could spend an hour reading the find manual, and still not be sure of the exact final results of what it will pull, especially when modifying someone else's script.

Find is sooooo coool. Especially when we went from standard unix find to gnu find. And when you start using those booleans, joined with escaped parens and quotes, oh my it gets pretty. There were a few years we had to have both the solaris utils and the gnu utils in different paths. If a system packaged scripts used a gnu util, we'd be fucked. And the same vice versa.

Can you tell I've been burned and try to opt for local coded black box utils (which log their decisions and drop a stat file for whatever they did), even if a system util in a multistage pipe will work?

I'm not saying I don't use find, sure I do. But I shy away from multiple decisions to choose or exclude a file in a single find command.

I never want to look at my boss and say: Yeah, the script that was supposed to archive off and delete those files REALLY got everyone, and here's the file counts.

And by the way, the fact someone in the other building said it deleted but did not really save the file must be bullshit, because this code looks fine.

Really, it's fine. No, I didn't write it, but the guy who did was very good. Huh, you want me to explain it to you?

Sure, gimme a sec.....

(uh oh, I'm fucked, no way to to explain this little snippet, I'm not really sure what it is doing....)

Um, never mind, I can't be sure, give me an hour to run some tests....
New s/magic/magic mushrooms/
I love and hate find in equal measure.

It's an abusive relationship we have.
New My feelings exactly aboout Perl ;-)
Just like Arabic, where you clear your throat and discover you accidentally ordered a cab to the airport? It is the only languange I know where you can bang your head against the keyboard and produce valid code...
New I've tried to learn Perl several times.
*Every* time, I just about give up because I can already do the task in bash, awk, php or icon so why am I bothering with perl? :-/ Still, great respect for what's been achieved and all that. Just not my cup-of-tea.

Wade.

Q:Is it proper to eat cheeseburgers with your fingers?
A:No, the fingers should be eaten separately.
New Learning Perl is like learning a core language facility
Sure, you CAN type line noise.
Or you can use it for a far higher level of grunting.
Or it can be poetry.
Pretty much like any very expressive language.

Those that have spent years programming in straight jackets (or constricted pipe lines with an occasional escape to some 'C') it is a godsend. Kind of like AWK.

I spent years grokking cut, paste, sort, uniq, tr, etc, etc. But the AWK man pages sucked, and each time I tried to use it, I found myself falling back into what I knew.

And then I got the MS-DOS MKS toolkit, which includes really good docs for AWK. 2 years later, I fell into Perl, with an AWK mindset. That worked.

Keep in mind, Perl is an organic aggregation of many unix utilities, then tied together by a single mind. Some utilities, like find, have been turning into a callback engine.

http://perldoc.perl.org/File/Find.html
New What's the point?
When there are scripting languages that are just as expressive, just as powerful, and a hell of a lot easier to read and understand?
Regards,
-scott
Welcome to Rivendell, Mr. Anderson.
New Ah, welcome to the expanding fray
The point isn't to use Perl. The point is to use something better than a stack of error prone pipes, when possible, for the big tasks that they were never envisioned for.

I use perl, and am happy to help in that direction, because it is what I know.

I am NOT positioning it as the best of the class.

Given a fresh start, it would be python or ruby for me.
New I hopped from Perl to Python...
...when I had to recycle some well-commented and neatly written Perl that was a few years old. I had to do a code walkthrough just to work out what the hell I was on about. I believe the words "fuck this shit" may have slipped out.

I can pick up my old Python code and understand it straight away, by contrast.

Not that I've written code of any description for a couple of years, mind...
New Yeah, but
I was under the impression you didn't consider yourself a programmer.

So any code switch is to be taken with a lot of questions. Which I don't feel like asking, so I'll drop it.
New No, it's fine. I don't consider myself a programmer.
I switched because the occasional, sporadic nature of that sort of work meant that I often had to get back up to speed with a bit of code with as little difficulty as possible, in as short a timeframe as possible.

I was better at writing Perl than I was at Python, but I could read Python code after the fact better than I could Perl.

I could probably have circumvented it by commenting my Perl properly, and using the long version of the inbuilt operators, but hey. That'd have been too easy.

I did find Python's regex stuff nicer to use than Perl's, though. Which is not what I expected at all.
New Quite.
I've been known to say that if you know a complex tool like Perl well, then go right ahead and use it for your own scripting purposes.

Most of my pipe-like processing is one-off and involves SQL and numerous trips through an editing session in vi (regexp replace FTW! when they work...). If it has to become something re-usable, then it gets turned into something much more robust, like a PHP script or maybe a bash script. I leave little pipe tools to things they are good at (grep comes to mind).

Wade.

Q:Is it proper to eat cheeseburgers with your fingers?
A:No, the fingers should be eaten separately.
New I like
find /my/dir -ctime +7 -type f -name "*?b*" -printf "%sn" | paste -sd+ - | bc
what is the - sign after the -sd+ is that the same as $i?
then after the - sign I could do a /1024 to get kilobytes?
If we torture the data long enough, it will confess. (Ronald Coase, Nobel Prize for Economic Sciences, 1991)
New The - means "standard input"
You can trade %s for %k to have find report in 1k blocks
New oh well doesnt work in solaris
bash-3.00$ find /export/home/imail/msgfiles/ -atime +21 -type f -name "?.b" -printf "%sn" | paste -sd+ - | bc
find: bad option -printf
find: [-H | -L] path-list predicate-list
bash-3.00$ find /export/home/imail/msgfiles/ -atime +21 -type f -name "?.b" -print "%sn" | paste -sd+ - | bc
find: bad option %sn
If we torture the data long enough, it will confess. (Ronald Coase, Nobel Prize for Economic Sciences, 1991)
New apt-get install ubuntu FTW!
--

Drew
New sorry, we have standards, snicker
If we torture the data long enough, it will confess. (Ronald Coase, Nobel Prize for Economic Sciences, 1991)
New GNU find from blastwave?
New That's how I'd go
though blastwave has some issues if you're using Solaris zones. Not horrible, but you get seriously inefficient in the use of disk space... they need a cleaner split between binaries and configs to make it better on a system with zones.
New Solaris style...
find /export/home/imail/msgfiles/ -atime +21 -type f -name "?.b" -ls | awk '{print $2}' | paste -sd+ - | bc

Column 2 in the ls output is the size in kB. Column 7 gives it in bytes. (No Solaris to test with, but I did use the Solaris man page this time.)
New thanx
banging the find thru crazies script is working but definitely bookmarked
If we torture the data long enough, it will confess. (Ronald Coase, Nobel Prize for Economic Sciences, 1991)
New Happy to help
And note the difficulty you have in the Solaris VS GNU utils? As I pointed out above, been there, been burned by that.

Yes, real admins and programmers are supposed to keep track of the many variations of arguments in the many implementations of the utils, both BSD vs AT&T, and then the individual computer manufacturer variations as well. Not me, and not anyone I've ever met. Who is this mythical person that can read and write shell scripts across the Unix variations? Sure, we each have the common ones, such as "ps -ef vs ps aux", but move into the line spoolers, and you're screwed unless you spend your time with nothing else.

Also, unix utils have internal limitations that I often hit. They are not meant to deal with my data (yes, pure text, but often too damn big), and puke (hopefully), or sometimes die silently, having written a short file (sort comes to mind, and in this case, yes, I knew to set the work space area.) I pulled the source for a bunch of the utils, and did an occasional hack years ago to make them work, but I hate working on other people's C code, escpecially filled with unrolled loops (lots of gotos).

And when I'm scripting a pipeline, checking the error return of the individual programs is a pain in the ass.

So, I'll lean to a Perl script that leaves lots of stats information in a log. I even have them email me on failure.

     aarrgg, brain dead - (boxley) - (33)
         {print $5+1}? Or did I totally misread the question? -NT - (scoenye) - (32)
             trying to get a running total, not increment - (boxley) - (31)
                 Here, feed this filenames. - (crazy) - (8)
                     I dont quite get this - (boxley) - (7)
                         -s - (crazy) - (6)
                             Ah, got it. thanx! -NT - (boxley) - (5)
                                 That was all an inside joke, right? - (drook) - (4)
                                     no works as advertized -NT - (boxley) - (2)
                                         Sigh - (crazy) - (1)
                                             sorry, no, works as advertized. My bad forgot comma -NT - (boxley)
                                     Don't show it to Ben. - (crazy)
                 Or like this... (using find) - (scoenye) - (21)
                     Yup, find is magic - (crazy) - (10)
                         s/magic/magic mushrooms/ - (pwhysall)
                         My feelings exactly aboout Perl ;-) - (scoenye) - (8)
                             I've tried to learn Perl several times. - (static) - (7)
                                 Learning Perl is like learning a core language facility - (crazy) - (6)
                                     What's the point? - (malraux) - (5)
                                         Ah, welcome to the expanding fray - (crazy) - (4)
                                             I hopped from Perl to Python... - (pwhysall) - (2)
                                                 Yeah, but - (crazy) - (1)
                                                     No, it's fine. I don't consider myself a programmer. - (pwhysall)
                                             Quite. - (static)
                     I like - (boxley) - (1)
                         The - means "standard input" - (scoenye)
                     oh well doesnt work in solaris - (boxley) - (7)
                         apt-get install ubuntu FTW! -NT - (drook) - (3)
                             sorry, we have standards, snicker -NT - (boxley) - (2)
                                 GNU find from blastwave? -NT - (pwhysall) - (1)
                                     That's how I'd go - (jake123)
                         Solaris style... - (scoenye) - (2)
                             thanx - (boxley) - (1)
                                 Happy to help - (crazy)

We have a moral imperative.
210 ms