Edupe seems pretty good. (More details.)

IWETHEY v. 0.3.0 | TODO

1,095 registered users | 0 active users | 0 LpH | Statistics
Login | Create New User

Welcome to IWETHEY!

Post #283,919

by Another Scott

5/4/07 5:11:53 PM

5/26/07 9:38:32 AM

Edupe seems pretty good. (More details.)

At least it works pretty quickly to give a list of duplicate files sorted by file size. It's written in [link|http://www.erlang.org/|Erlang].

[link|http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/|Writing a Duplicate File Finder in Erlang].

Instructions on compiling and using it are [link|https://bohr.diginux.net/wiki/index.php/EDupe|here]. Remember to pipe the output to a file (e.g. C:\\Program Files\\erl5.5.4\\bin>erl -noshell -run edupe start c:/ -s erlang halt > 070526-DriveC.txt).

[edit:] Note that (on Windows) you'll need the DLL - libeay32.dll. It's easily found. You can get a copy of it in the WGet package - [link|http://www.christopherlewis.com/WGet/WGetFiles.htm|here]. Simply copy the DLL to the c:\\Program Files\\erl5.5.4\\bin\\ directory. Also, don't forget to compile the edupe package by running "erlc edupe.erl" before running the erl - noshell ... script.

HTH.

Cheers,
Scott.

Edited by Another Scott May 26, 2007, 09:38:32 AM EDT

Expand All History

Post #286,922

by Another Scott

6/12/07 11:11:54 PM

Edupe crashes on some of my machines. Trying another...

On some systems I get an error dump with comments about being unable to create enough virtual memory, but it seems to be related to very deep directories with very long directory names. I don't know enough to try to debug it.

I'm looking at [link|http://www.tucows.com/preview/373411|Duplicate File Finder 1.1.0.0] right now. It's very well done, very fast (even when calculating CRC32s), has a nice GUI, and has lots of output options. It seems nearly perfect for Win32. The price is right too - it's free.

It had no problem with my T41 laptop. I'm having it search my Opteron box now - it's got ~ 2 M files with a sizable fraction being duplicates...

Cheers,
Scott.

Best duplicate finder/deleter for Win32? - (Another Scott) - (19) - Nov. 23, 2005, 09:50:10 AM EST

That's a Perl problem if ever I saw one. - (pwhysall) - (9) - Nov. 23, 2005, 10:38:26 AM EST

ICLRPD: That's a Perl problem if ever I saw one. (new thread) - (Steve Lowe) - Nov. 23, 2005, 10:57:11 AM EST

Baz? What's a Baz? - (broomberg) - (6) - Nov. 23, 2005, 12:17:48 PM EST

Alternatively... - (pwhysall) - (3) - Nov. 23, 2005, 01:35:25 PM EST

Any gotchas? - (Another Scott) - (2) - Nov. 23, 2005, 10:45:23 PM EST

The sample program in the documentation... - (ben_tilly) - (1) - Nov. 23, 2005, 11:01:27 PM EST

Thanks Ben. I appreciate it. -NT - (Another Scott) - Nov. 23, 2005, 11:02:48 PM EST

You're a Baz. -NT - (pwhysall) - Nov. 23, 2005, 01:35:55 PM EST

If you want to be really cautious.... - (ben_tilly) - Nov. 23, 2005, 09:52:06 PM EST

Yes - (ben_tilly) - Nov. 23, 2005, 09:51:11 PM EST

I'm trying a Python solution now. - (Another Scott) - (2) - Dec. 7, 2005, 02:36:50 PM EST

Any reason you can't use gnu/sort ? -NT - (drewk) - (1) - Dec. 7, 2005, 07:19:24 PM EST

Dunno. I'll check into it. Thanks. -NT - (Another Scott) - Dec. 7, 2005, 09:47:02 PM EST

I'm trying DUFF now. - (Another Scott) - (3) - April 8, 2007, 11:12:04 PM EDT

<drool> Mm, Duff... </homer> -NT - (CRConrad) - (1) - April 9, 2007, 10:05:32 AM EDT

Ha! :-) -NT - (Another Scott) - April 9, 2007, 11:49:23 AM EDT

It's called an alpha for a reason. - (Another Scott) - April 11, 2007, 09:22:02 PM EDT

Edupe seems pretty good. (More details.) - (Another Scott) - (1) - May 26, 2007, 09:38:32 AM EDT

Edupe crashes on some of my machines. Trying another... - (Another Scott) - June 12, 2007, 11:11:54 PM EDT

iwethey.org

Her patriotism is as genuine as her hair color.
91 ms