Post #71,246
12/28/02 5:01:08 PM
12/28/02 5:02:03 PM
|
Once again, Todd's explanation
You are halfway to the problem. The problem is that sometimes MacOS uses special files, and sometimes not. tar does not understand this.
We have two filesystems. One of which (HFS) stores a file with lots of hooks for metadata. One of which (UFS) only directly supports the metadata that any Unix system has.
Going from HFS to UFS, what do you do with the important metadata? Apple decided to create new files for it. Therefore you cannot move a file from HFS to UFS without creating extra files.
The tar shipped with OS X doesn't understand this. And therefore does not create the needed extra files, and loses important metadata moving from HFS to UFS.
Personally (admittedly without much serious thought) what I would have been inclined to do in Apple's situation is have 2 sets of APIs for accessing files. One of which sees a traditional HFS file with metadata (on either system). The other of which sees the associated collection of files that are needed in UFS. Yes, there are problems with that as well. But then you would at least find it easy to get the appropriate consistent behaviour across filesystems.
BTW Linux may have a similar problem some day. (May already have in fact...) There is exploration of the idea of supporting streams within files in some filesystems. Copying a file with streams to a filesystem without streams has to be done...how? I haven't exactly been following discussion on this, but the following old post illustrates the issues that have to be thought through:
[link|http://web.gnu.walfield.org/mail-archive/linux-kernel/2000-August/1275.html|http://web.gnu.walfi...-August/1275.html]
Cheers,
Ben
"Career politicians are inherently untrustworthy; if it spends its life buzzing around the outhouse, it\ufffds probably a fly."
- [link|http://www.nationalinterest.org/issues/58/Mead.html|Walter Mead]
Edited by ben_tilly
Dec. 28, 2002, 05:02:03 PM EST
Once again, Todd's explanation
You are halfway to the problem. The problem is that sometimes MacOS uses special files, and sometimes not.
We have two filesystems. One of which (HFS) stores a file with lots of hooks for metadata. One of which (UFS) only directly supports the metadata that any Unix system has.
Going from HFS to UFS, what do you do with the important metadata? Apple decided to create new files for it. Therefore you cannot move a file from HFS to UFS without creating extra files.
The tar shipped with OS X doesn't understand this. And therefore does not create the needed extra files, and loses important metadata moving from HFS to UFS.
Personally (admittedly without much serious thought) what I would have been inclined to do in Apple's situation is have 2 sets of APIs for accessing files. One of which sees a traditional HFS file with metadata (on either system). The other of which sees the associated collection of files that are needed in UFS. Yes, there are problems with that as well. But then you would at least find it easy to get the appropriate consistent behaviour across filesystems.
BTW Linux may have a similar problem some day. (May already have in fact...) There is exploration of the idea of supporting streams within files in some filesystems. Copying a file with streams to a filesystem without streams has to be done...how? I haven't exactly been following discussion on this, but the following old post illustrates the issues that have to be thought through:
[link|http://web.gnu.walfield.org/mail-archive/linux-kernel/2000-August/1275.html|http://web.gnu.walfi...-August/1275.html]
Cheers,
Ben
"Career politicians are inherently untrustworthy; if it spends its life buzzing around the outhouse, itÂ’s probably a fly."
- [link|http://www.nationalinterest.org/issues/58/Mead.html|Walter Mead]
|
Post #71,269
12/28/02 8:05:12 PM
|
Using a hammer to drive screws will hurt
Ben wrote:
The tar shipped with OS X doesn't understand this. And therefore does not create the needed extra files, and loses important metadata moving from HFS to UFS.
If you understand how files are stored, then you immediately realise why GNU tar is an inappropriate tool for such operations. A "problem" exists only if you're part of a funky computer religion that implicitly assumes you shouldn't have to understand the fundamentals of what you're doing.
BTW Linux may have a similar problem some day. (May already have in fact...)
That wouldn't be a "problem" any more than the ability to overwrite files with the "cat" utility is a "problem": Using an obviously unsuitable tool for a particular task, or otherwise aiming a gun at your foot and pulling the trigger, will inevitably have suboptimal results or worse.
So Don't Do That, Then.
Rick Moen
rick@linuxmafia.com
If you lived here, you'd be $HOME already.
|
Post #71,338
12/29/02 5:23:15 PM
|
I understand your position perfectly
You believe that all abstractions should be useless, and that nobody should be able to trust in anything without knowing the whole system perfectly.
There is a name for code produced by programmers who think like that. That name is spaghetti. And systems that are built like that inevitably become a [link|http://www.laputan.org/mud/mud.html|Big Ball of Mud].
Yes, I am part of a funky computer religion that implicitly assumes that you shouldn't have to understand the fundamentals of what you're doing - at least not all of the time. Abstractions like files and directories serve an important purpose, and the abstractions should not be broken lightly. And I am proud to say that that is a Good Thing.
Cheers,
Ben
"Career politicians are inherently untrustworthy; if it spends its life buzzing around the outhouse, it\ufffds probably a fly."
- [link|http://www.nationalinterest.org/issues/58/Mead.html|Walter Mead]
|
Post #71,362
12/29/02 7:17:38 PM
|
Re: I understand your position perfectly
Ben wrote:
I understand your position perfectly. You believe that all abstractions should be useless, and that nobody should be able to trust in anything without knowing the whole system perfectly.
What an absolutely fabulous straw man you have there. May I take a whack at it, too, or do you have proprietary rights?
Rick Moen
rick@linuxmafia.com
If you lived here, you'd be $HOME already.
|
Post #71,572
12/30/02 6:16:31 PM
|
If it is not a straw man...
Then please explain how your position differs appreciably from what I described.
Regards,
Ben
"Career politicians are inherently untrustworthy; if it spends its life buzzing around the outhouse, it\ufffds probably a fly."
- [link|http://www.nationalinterest.org/issues/58/Mead.html|Walter Mead]
|
Post #71,811
12/31/02 10:14:16 PM
|
Re: If it is not a straw man...
Ben wrote:
If it is not a straw man, then please explain how your position differs appreciably from what I described.
If I could figure out how you derived the extremely strange view you described immediately before attributing it to me, and then went on to draw even more peculiar conclusions from it, I'd gladly tell you which wrong turn you took on that road. But it's way too damn bizarre for my tastes, and it should suffice to say "No, I most certainly don't believe that, nor does anyone, actually."
I'm willing to believe that you honestly thought that. I guess. But I'm not going to try to prove to you that I don't hold a rather odd and silly view that strike me as nothing at all like what I said.
Rick Moen
rick@linuxmafia.com
If you lived here, you'd be $HOME already.
|
Post #71,962
1/2/03 4:55:17 AM
|
Funny, I concluded the same as Ben
YTF should I have to understand how the file system works to use it?
Do I also need to know the voltages in my ram chips to make proper use of memory in my software?
And of course, I mustn't write anything using sockets unless I clearly understand the inner workings of the entire communications stack and what is actually happening on the wires.
Better not drive a car without being able to calculate the thermal energy per unit of fuel - otherwise you'll probably run out someplace. You certainly can't trust that little needle - especially if you don't know how it works.
At some point, you have to use *abstractions* in order to move forwards or you will never transcend your current level of minutiae.
Your assertion that you must know the inner workings of the file system in order to use it is stupid.
I am out of the country for the duration of the Bush administration.
Please leave a message and I'll get back to you when democracy returns.
|
Post #71,970
1/2/03 5:31:20 AM
|
Re: Funny, I concluded the same as Ben
ToddBlanchard wrote:
YTF should I have to understand how the file system works to use it?
Well, you needn't, of course. Just concentrate on watching the pretty pictures. Don't worry; be happy. Other people will take care of the technical stuff, and you can just wait for it to be delivered in idiotproofed form.
Rick Moen
rick@linuxmafia.com
If you lived here, you'd be $HOME already.
|
Post #72,164
1/3/03 4:27:19 AM
|
Little story - you might find it entertaining
I've been an avid skydiver since 1983. I've got over 24 hours of free fall time and over 1000 jumps (which means I've fallen something like 1500 miles).
When I started in the sport, before standardization of equipment, the number one cause of deaths wasn't "failed parachutes" (which was maybe 5% of problems - and today its less than a whole percent). It was "borrowed gear".
Every month I would eagerly rip open my new issue of Parachutist to read the accident reports - because its valuable (and safer) to learn from other people's mistakes.
In the early 80's skydiving was going through a transition period from outlaw practice to viable busineess. While lots of military surplus gear was still common, the first gear created strictly for pleasure jumping was beginning to appear on the market from a half dozen different companies.
The result was stuff shipped with a wide variety of "user interfaces" (handle placements). Some put the main deployment handle on a band crossing the belly. Other s on the left hip, or the right hip. The bottom of the pack was a common place while older gear had the main rip cord on the chest. Reserve ripcords were typically on the chest next to the cutaway handle for separating from the main parachute (if its trash you want it gone before opening old faithful) but left and right placement varied. Handle appearance varied as well. So you couldn't just grab a rig and know whats what. You had to ask the owner. I mean, if you don't know what it is, maybe you shouldn't jump it.
But people sometimes borrow gear because their stuff isn't packed when the plane is ready to go. So very knowledgeable skydivers would borrow something, get a quick explanation for what was where, (there's only 3 handles on a parachute, main, reserve, cutaway), and go make a jump.
And then they'd hit the ground pulling on the wrong thing. Lots of them. Gurus. Guys who had jumped everything under the sun and were as comfortable freefalling as lying in a hammock would bounce. The investigation would typically find that the guy was jumping something he had borrowed on the spur of the moment.
This cycle is repeating among the BASE jumping community as BASE gear evolves. You may remember the news story from a couple years ago of Jan Davis - one of the best and brightest - very publicly bouncing during a protest jump at Yosemite in 1999. The park officials were going to arrest the participants and confiscate their gear - so she borrowed something less nice than her regular rig. The unfamiliarity killed her (it wasn't lack of knowledge - she knew how it worked - she had hundreds of jumps on it).
The moral of this story is that variety kills. Today every single rig on the market has the handles laid out in exactly the same way and we don't have those kinds of accidents anymore. So for the same reason, I think its dangerous to mix file system types on a computer. At least until the tools evolve to properly handle the issues.
If the handles on the different file systems are different, accidents will happen.
Better knowledge is not the answer. Standardizing the interfaces is. All tools must become multi-file sytem aware or as Arkadiy says - we might as well keep sector maps on a pad of paper by our desk.
Of course, you seem to think you're bulletproof and your "knowledge" will protect you. It won't. You'll forget what file system you're on one day and slip. Like all those former skygods I used to know.
The dead ones.
I am out of the country for the duration of the Bush administration.
Please leave a message and I'll get back to you when democracy returns.
|
Post #72,212
1/3/03 10:21:21 AM
|
Excellent story. Going on the wall.
Regards,
-scott anderson
"Welcome to Rivendell, Mr. Anderson..."
|
Post #72,237
1/3/03 11:59:25 AM
|
This my dear friend... (new thread)
Created as new thread #72236 titled [link|/forums/render/content/show?contentid=72236|This my dear friend...]
[link|mailto:curley95@attbi.com|greg] - Grand-Master Artist in IT
[link|http://www.iwethey.org/ed_curry/|REMEMBER ED CURRY!]
[link|http://pascal.rockford.com:8888/SSK@kQMsmc74S0Tw3KHQiRQmDem0gAIPAgM/edcurry/1//|ED'S GHOST SPEAKS!] | The Heimatland Geheime Staatspolizei reminds:
[link|http://www.wired.com/news/wireless/0,1382,56742,00.html| Wi-Fi Terrorism] comes with an all inclusive
free trip to the local Hoosegow! | Please visit [link|http://z.iwethey.org/forums/render/board/show?boardid=1|iwethey.anti.anti++], providing
*THE* alternative to iwethey.anti-- since
June 18, 2001 22:00EST | | I'll never tell, my *overly-red* lips are sealed! *wink* *wink* |
|
Post #72,456
1/4/03 12:43:54 PM
|
Since you seem to be missing the point...
Here it is again.
Your attitude is that people should not use tools unless they know everything relevant to its proper usage. Your definition of what is relevant seems to be that if it might come up, then they need to know it. This is a circular definition of relevancy that justifies any particular dependency in hindsight as the user's fault for not knowing better.
Now I know that [link|http://www.joelonsoftware.com/articles/LeakyAbstractions.html|abstractions leak]. (That is one of the few articles by Joel that I agree with incidentally.) Furthermore I often find myself being in the position of being the person around who understands what abstractions leak, and why. There is no fundamental solution to that problem. Shit happens, and plumbers are needed for it.
However one mark of a programmer that you want to have around is the ability to see shit and recognize it for what it is. Leaking abstractions are signs of shit. While they may be inevitable, they are not something that you want to take lightly. Because a system built by people with too much tolerance for that inevitably degrades into a complete mess.
In this case the leaking abstraction is one of the worst kinds to have. Different filesystems use the same words for subtly different abstractions. This makes it hard for most people to even verbalize that there is a difference, let alone what it is. (When the same thing happens in speech you get classic threads where people talk past each other at length.) And when you try to combine them into one abstraction, the gaps keep on leaking past.
So yes, it happens for good reason. But it is a basic design flaw in the system. And it is not one to belittle people for getting tripped up by, nor is it one that has a simple right answer.
Cheers,
Ben
"Career politicians are inherently untrustworthy; if it spends its life buzzing around the outhouse, it\ufffds probably a fly."
- [link|http://www.nationalinterest.org/issues/58/Mead.html|Walter Mead]
|
Post #74,798
1/16/03 4:13:56 AM
|
Re: Since you seem to be missing the point...
Ben Tilly wrote:
Your attitude is that people should not use tools unless they know everything relevant to its proper usage.
Not quite. Rather: I mildly (and not at all insistently, given that it's really not my problem) suggest that people should take responsibility for what they do. If you choose to use tools without adequately understanding them, you can pound your thumb with a misaimed hammer. The choice of taking that risk may be reasonable; the point is to not expect pity when you screw up and hurt yourself.
Personally, I prefer education over "learning experiences". Less painful.
Remainder of your post duly ignored as furious and tedious pummeling of an irrelevant straw man. Therapeutic though, I'm sure.
Rick Moen
rick@linuxmafia.com
If you lived here, you'd be $HOME already.
|
Post #71,434
12/30/02 5:21:30 AM
12/30/02 7:52:16 AM
|
I do not think that word means what you think it means.
"Using an obviously unsuitable tool for a particular task"
One day you will have to perhaps define your version of this word.
Firing up a terminal on OS X and navigating to an HFS file system, I find that I can:
ls
cat
mv
cp
chown
chgrp
chmod
ln
rm
Within an HFS volume and across HFS volumes just fine.
IOW, it appears that the developers of OS X have taken some pains to make sure that these tools work as expected on both UFS and HFS volumes. I fact, in the Fred Sanchez paper I cited awhile back, it describes the interesting strategies used to emulate hard links, soft links, file ownership, etc, that were taken to make the system behave as expected.
Its unfortunate that the illusion is incomplete (though I remain hopeful that this will be rectified with a future release).
But sometime you are going to have to share your definition of obvious with us.
Because what is clearly obvious to me is that what you claim to be obvious is not at all obvious to a daily user of the system.
It had been some months of daily command line development (I generally use vi and makefiles when I work - so is all in the shell) before I was bitten by the tar/cp/mv issue.
I am out of the country for the duration of the Bush administration.
Please leave a message and I'll get back to you when democracy returns.
|
Post #71,440
12/30/02 8:21:08 AM
|
Hello, you must be going.
Hi there! I said we were done, and I meant that.
However, on your way back to whatever it is that you do, you might consider the merits of consulting manpages before just blithely assuming that someone has custom-modified a standard tool like GNU tar and GNU cpio/afio to perform operations they do nowhere else on the planet.
Or, if you prefer, you can ignore such considerations and bitch when reality bites you in the ass. Feel free to work out your options, somewhere.
Rick Moen
rick@linuxmafia.com
If you lived here, you'd be $HOME already.
|
Post #71,534
12/30/02 3:47:08 PM
|
This is obvious?
You wanna point out the obvious part?
Yeah, right. I don't see the letters UFS in there anywhere.
I don't see the letters HFS in there anywhere.
I imagine what you meant by "man" must have been the source code, eh?
>man tar
TAR(1) System General Commands Manual TAR(1)
NAME
tar - tape archiver
SYNOPSIS
tar [-]{crtux}[befhmopvwzHLPXZ014578] [archive] [blocksize] [-C directory
] [-s replstr ] file1 [file2...]
DESCRIPTION
The tar command creates, adds files to, or extracts files from an archive
file in tar format. A tar archive is often stored on a magnetic tape,
but can be a floppy or a regular disk file.
One of the following flags must be present:
-c Create new archive, or overwrite an existing archive,
adding the specified files to it.
-r Append the named new files to existing archive. Note that
this will only work on media on which an end-of-file mark
can be overwritten.
-t List contents of archive. If any files are named on the
command line, only those files will be listed.
-u Alias for -r
-x Extract files from archive. If any files are named on the
command line, only those files will be extracted from the
archive. If more than one copy of a file exists in the
archive, later copies will overwrite earlier copies during
extration.
In addition to the flags mentioned above, any of the following flags may
be used:
-b blocking factor
Set blocking factor to use for the archive, tar uses 512
byte blocks. The default is 20, the maximum is 126.
Archives with a blocking factor larger 63 violate the POSIX
standard and will not be portable to all systems.
-e Stop after first error.
-f archive Filename where the archive is stored. Defaults to
/dev/rst0
-h Follow symbolic links as if they were normal files or
directories.
-m Do not preserve modification time.
-O Write old-style (non-POSIX) archives.
-o Don't write directory information that the older (V7) style
tar is unable to decode. This implies the -O flag.
-p Preserve user id, group id, file mode, access and modifica-
tion times if possible. The user id and group id will only
be set if the user is the superuser (unless these values
correspond to the user's user and group ids).
-s replstr Modify the file or archive member names specified by the
pattern or file operands according to the substitution
expression replstr, using the syntax of the ed(1) utility
regular expressions. The format of these regular expres-
sions are:
/old/new/[gp]
As in ed(1), old is a basic regular expression and new can
contain an ampersand (&), \\n (where n is a digit) back-ref-
erences, or subexpression matching. The old string may
also contain <newline> characters. Any non-null character
can be used as a delimiter (/ is shown here). Multiple -s
expressions can be specified. The expressions are applied
in the order they are specified on the command line, termi-
nating with the first successful substitution. The
optional trailing g continues to apply the substitution
expression to the pathname substring which starts with the
first character following the end of the last successful
substitution. The first unsuccessful substitution stops the
operation of the g option. The optional trailing p will
cause the final result of a successful substitution to be
written to standard error in the following format:
<original pathname> >> <new pathname>
File or archive member names that substitute to the empty
string are not selected and will be skipped.
-v Verbose operation mode.
-w Interactively rename files. This option causes tar to
prompt the user for the filename to use when storing or
extracting files in an archive.
-z Compress archive using gzip.
-C directory This is a positional argument which sets the working direc-
tory for the following files. When extracting, files will
be extracted into the specified directory; when creating,
the specified files will be matched from the directory.
-H Follow symlinks given on command line only.
-L Follow all symlinks.
-P Do not strip leading slashes (``/'') from pathnames. The
default is to strip leading slashes.
-X Do not cross mount points in the file system.
-Z Compress archive using compress.
The options [-014578] can be used to select one of the compiled-in backup
devices, /dev/rstN.
FILES
/dev/rst0 The default archive name
SEE ALSO
pax(1), cpio(1)
AUTHOR
Keith Muller at the University of California, San Diego
ERRORS
tar will exit with one of the following values:
0 All files were processed successfully.
1 An error occured.
Whenever tar cannot create a file or a link when extracting an archive or
cannot find a file while writing an archive, or cannot preserve the user
ID, group ID, file mode or access and modification times when the -p
options is specified, a diagnostic message is written to standard error
and a non-zero exit value will be returned, but processing will continue.
In the case where tar cannot create a link to a file, tar will not create
a second copy of the file.
If the extraction of a file from an archive is prematurely terminated by
a signal or error, tar may have only partially extracted the file the
user wanted. Additionally, the file modes of extracted files and direc-
tories may have incorrect file bits, and the modification and access
times may be wrong.
If the creation of an archive is prematurely terminated by a signal or
error, tar may have only partially created the archive which may violate
the specific archive format specification.
BSD June 11, 1996 BSD
I am out of the country for the duration of the Bush administration.
Please leave a message and I'll get back to you when democracy returns.
|
Post #71,812
12/31/02 10:20:55 PM
|
Re: This is obvious?
ToddBlanchard wrote:
You wanna point out the obvious part?
It's the part where you understand the basics of how files are stored on your filesystems, understand how GNU tar / GNU cpio work, and avoid assuming (without confirmation) that those tools perform special functions, there, that they carry out nowhere else on the planet.
Khendon's Law duly invoked. Va t'en.
Rick Moen
rick@linuxmafia.com
If you lived here, you'd be $HOME already.
|
Post #71,868
1/1/03 3:14:02 PM
|
At no time did you utter the word "obvious" - nice dodge
thus you did not answer the question.
What is obvious to me is your lack of knowledge and use of bluster to compensate.
I am out of the country for the duration of the Bush administration.
Please leave a message and I'll get back to you when democracy returns.
|
Post #71,891
1/1/03 5:34:29 PM
|
Re: At no time did you utter the word "obvious" - nice dodge
ToddBlanchard wrote, before reverting to argumentum ad hominem:
thus you did not answer the question.
Indeed, Chuckles. I was addressing the UFS/FFS topic at hand, which does not necessarily mean "answering your questions". You have perhaps confused me with some paid sidekick.
Now, run along.
Rick Moen
rick@linuxmafia.com
If you lived here, you'd be $HOME already.
|
Post #71,965
1/2/03 5:02:05 AM
|
Topic at hand - point out what you claim is "obvious" (new thread)
Created as new thread #71964 titled [link|/forums/render/content/show?contentid=71964|Topic at hand - point out what you claim is "obvious"]
I am out of the country for the duration of the Bush administration.
Please leave a message and I'll get back to you when democracy returns.
|
