mtree(8) keyword clarification: "device"

[Jean-Yves Migeon]

Hi lists,

I am actually working on making libarchive's mtree format and our
NetBSD's mtree(8) interchangeable, but I am facing a documentation issue
that dates back /a priori/ to 2001, and that concerns the "device"
keyword. Copy paste from NetBSD's mtree(8):


[...]
device The device number to use for block or char file
types.
The argument must be one of the following forms:

format,major,minor
A device with major and minor fields, for an
operating system specified with format. See
below
for valid formats.

format,major,unit,subunit
A device with major, unit, and subunit
fields, for
an operating system specified with format.
(Currently this is only supported by the
bsdos
format.)

number
Opaque number (as stored on the file
system).

The following values for format are recognized:
native,
386bsd, 4bsd, bsdos, freebsd, hpux, isc, linux,
netbsd,
osf1, sco, solaris, sunos, svr3, svr4, and ultrix.


The actual implementation in C never goes beyond the point of printing
(long long)st_rdev from stat(2) in %llx, and use it "as-is" for
mknod(8).

Libarchive does it differently and log it as
"device=native,<major>,<minor>" format.

Questions arise:
- what was the intent of this keyword, especially the format part?
mtree(8) was expected to receive some kind of "-k device=netbsd"
parameter? Or does it only specify the argument of the resulting mtree
evaluation?
- the actual code in libarchive does not follow NetBSD's format, it
prints "device" by default as "native,<major>,<minor>". Would it be
acceptable for libarchive to accept raw, opaque number with the
convention that it starts with "Ox"?

Thanking you in advance for your clarifications

Cheers,

[Christos Zoulas]

Post by Jean-Yves Migeon
Hi lists,
I am actually working on making libarchive's mtree format and our
NetBSD's mtree(8) interchangeable, but I am facing a documentation issue
that dates back /a priori/ to 2001, and that concerns the "device"
[...]
device The device number to use for block or char file
types.
format,major,minor
A device with major and minor fields, for an
operating system specified with format. See
below
for valid formats.
format,major,unit,subunit
A device with major, unit, and subunit
fields, for
an operating system specified with format.
(Currently this is only supported by the
bsdos
format.)
number
Opaque number (as stored on the file
system).
native,
386bsd, 4bsd, bsdos, freebsd, hpux, isc, linux,
netbsd,
osf1, sco, solaris, sunos, svr3, svr4, and ultrix.
The actual implementation in C never goes beyond the point of printing
(long long)st_rdev from stat(2) in %llx, and use it "as-is" for
mknod(8).
Libarchive does it differently and log it as
"device=native,<major>,<minor>" format.
- what was the intent of this keyword, especially the format part?
mtree(8) was expected to receive some kind of "-k device=netbsd"
parameter? Or does it only specify the argument of the resulting mtree
evaluation?
- the actual code in libarchive does not follow NetBSD's format, it
prints "device" by default as "native,<major>,<minor>". Would it be
acceptable for libarchive to accept raw, opaque number with the
convention that it starts with "Ox"?
Thanking you in advance for your clarifications

NetBSD's mtree understands different encodings for major,minor (how
many bits are used for major and minor). See the mknod(8) sources.
Native means NetBSD, as well as it can understand a single number
which is interpreted as raw.

christos

[Jean-Yves Migeon]

Post by Christos Zoulas
NetBSD's mtree understands different encodings for major,minor (how
many bits are used for major and minor). See the mknod(8) sources.
Native means NetBSD, as well as it can understand a single number
which is interpreted as raw.

For whatever reason I read mknod(2) and not mknod(8). I was not looking
at the correct place (e.g. the parsedev() function).

Which raises a remark here: while NetBSD's mtree can decode a
"format,major,minor" value thanks to parsedev, it will encode the value
as raw fts entry ("number") in the spec.

AFAICT that part is missing in libarchive, so I'll have to import it.

Thanks!

[Christos Zoulas]

Post by Jean-Yves Migeon
Which raises a remark here: while NetBSD's mtree can decode a
"format,major,minor" value thanks to parsedev, it will encode the value
as raw fts entry ("number") in the spec.

I guess nobody implemented this yet.

Post by Jean-Yves Migeon
AFAICT that part is missing in libarchive, so I'll have to import it.

Ok.

christos

[Tim Kientzle]

Post by Jean-Yves Migeon
Hi lists,
I am actually working on making libarchive's mtree format and our NetBSD's mtree(8) interchangeable,

Excellent!

Post by Jean-Yves Migeon
[...]
device The device number to use for block or char file types.
format,major,minor
A device with major and minor fields, for an
operating system specified with format. See below
for valid formats.
format,major,unit,subunit
A device with major, unit, and subunit fields, for
an operating system specified with format.
(Currently this is only supported by the bsdos
format.)
number
Opaque number (as stored on the file system).
The following values for format are recognized: native,
386bsd, 4bsd, bsdos, freebsd, hpux, isc, linux, netbsd,
osf1, sco, solaris, sunos, svr3, svr4, and ultrix.
The actual implementation in C never goes beyond the point of printing (long long)st_rdev from stat(2) in %llx, and use it "as-is" for mknod(8).
Libarchive does it differently and log it as "device=native,<major>,<minor>" format.

Libarchive's mtree support has evolved quite a bit in order to
try to be more compatible with other implementations. We're
happy to get help in continuing to improve this.

I believe the current code was heavily influenced by the FreeBSD
implementation of mtree(8), which is a little different from NetBSD's
implementation.

Post by Jean-Yves Migeon
- what was the intent of this keyword, especially the format part? mtree(8) was expected to receive some kind of "-k device=netbsd" parameter? Or does it only specify the argument of the resulting mtree evaluation?

I don't know the history; digging through source control history
might suggest something.

Post by Jean-Yves Migeon
- the actual code in libarchive does not follow NetBSD's format, it prints "device" by default as "native,<major>,<minor>". Would it be acceptable for libarchive to accept raw, opaque number with the convention that it starts with "Ox"?

Libarchive tries to be intelligent about handling variant formats.
We would be happy to see patches that made this more
flexible.

It looks like it should be relatively simple to expand parse_device()
in libarchive/archive_read_support_format_mtree.c to handle other
formats. I would suggest first checking for the number of comma
characters.

Tim

[Jean-Yves Migeon]

Post by Tim Kientzle

Post by Jean-Yves Migeon
Libarchive does it differently and log it as
"device=native,<major>,<minor>" format.

Libarchive's mtree support has evolved quite a bit in order to
try to be more compatible with other implementations. We're
happy to get help in continuing to improve this.
I believe the current code was heavily influenced by the FreeBSD
implementation of mtree(8), which is a little different from NetBSD's
implementation.

christos@ answered my question to this, and I pushed the code that does
the format,major,minor parsing as well as "packing" (e.g. converting the
format to the appropriate dev_t).

See https://github.com/libarchive/libarchive/pull/47

This would need a review though, I am familiar with NetBSD KNF, and not
with libarchive's and its cmake :)

Post by Tim Kientzle
Libarchive tries to be intelligent about handling variant formats.
We would be happy to see patches that made this more
flexible.

See above.

Post by Tim Kientzle
It looks like it should be relatively simple to expand parse_device()
in libarchive/archive_read_support_format_mtree.c to handle other
formats. I would suggest first checking for the number of comma
characters.

That's what I did.

Another question then: I am willing to extend mtree(5) format so it can
store st_rdev (the resident device ID) and the inode so that I can
obtain a "unique" node ID based on the device number on which the node
reside combined with its own inode.

"device" being already used, I went with "resdevice" (for "resident
device") and "inode".

My initial patch was against NetBSD's mtree(5) [1]; I turned the binary
into a library for a $DAYJOB project, but joerg@ then suggested using
libarchive, which already has mtree(5) support in addition to being,
well... a library, with tons of other archives/compression formats
recognized.

Indeed, the advice was good, and I moved to working with libarchive,
but as our team already use mtree(5) quite extensively, I would like to
make a drop-in replacement and have an mtree(8) binary based on
libarchive (like bsdtar and cpio).

That could also provide the same mtree for both NetBSD and FreeBSD
(through libarchive), but this is more "pie in the sky" for now. There's
still some code to write before I achieve that purpose. Does that fit
libarchive's goals too?

Thanks!