Small tip proposal for headless systems boot resiliency

Discussion:

Mathieu Lubrano

2013-12-30 23:46:30 UTC

Hello,

The headless systems, or boxes that may be far away are a pain to get back online if stuck during the boot sequence... (embedded/headless do not have IP-KVM or remote control).

* The point is : with a headless system you want it back online, really. Even if it needs maintenance, you need a ssh or whatever to get your hands dirty. So don't stop the boot sequence. If it's damaged beyond basic/rescue usability, it makes no difference anyway.

Example: The stop_boot() function of /etc/rc.subr got it to stop a few time here... and these scripts may call it:
grep stop_boot /etc/rc.d/*
/etc/rc.d/ipfilter: stop_boot
/etc/rc.d/ipsec: stop_boot
/etc/rc.d/pf: stop_boot
/etc/rc.d/pf_boot: stop_boot
(see, these are network related scripts, ones that you may play with remotely and get stuck for many reasons, let's tackle this one in particular).

A failed fsck may call it too, but this can be moderated with fsck_flags="-p -y -P" in your rc.conf.

* Proposal:
Add a "headless" flag to rc.conf, and alter stop_boot function this way:

/etc/rc.conf:
headless=yes

diff -u /mnt/sd0d/backup/rc.subr /etc/rc.subr
--- /mnt/sd0d/backup/rc.subr 2013-12-21 23:29:01.000000000 +0100
+++ /etc/rc.subr 2013-12-31 00:10:18.000000000 +0100
@@ -100,14 +100,24 @@
# If booting directly to multiuser, send SIGTERM to
# the parent (/etc/rc) to abort the boot.
# Otherwise just exit.
+# OR
+# If this is a headless system, just send a warning, pause to give a hint,
+# and try resuming the boot sequence.
#
stop_boot()
{
- if [ "$autoboot" = yes ]; then
- echo "ERROR: ABORTING BOOT (sending SIGTERM to parent)!"
- kill -TERM ${RC_PID}
+ if [ "$headless" = yes ] || [ "$headless" = YES ]; then
+ echo "WARNING: BOOT *SHOULD* HAVE BEEN STOPPED"
+ echo "Resuming boot sequence in 15s, the System may be unusable."
+ sleep 30
+ touch /CHECK_BOOT_LOG.warn
+ else
+ if [ "$autoboot" = yes ]; then
+ echo "ERROR: ABORTING BOOT (sending SIGTERM to parent)!"
+ kill -TERM ${RC_PID}
+ fi
+ exit 1
fi
- exit 1
}

Happy end of year with your unstoppable NetBSD systems ;)
Kind regards,
Mat.

Dave McGuire

2013-12-30 23:59:43 UTC

Permalink

Post by Mathieu Lubrano
The headless systems, or boxes that may be far away are a pain to get
back online if stuck during the boot sequence... (embedded/headless
do not have IP-KVM or remote control).
* The point is : with a headless system you want it back online,
really. Even if it needs maintenance, you need a ssh or whatever to
get your hands dirty. So don't stop the boot sequence. If it's
damaged beyond basic/rescue usability, it makes no difference
anyway.

This is, to put it simply, very dangerous.

In fact it DOES make a difference, potentially a very large one. If
I've, say, put a NetBSD system in an industrial control environment
(which is not an unlikely thing to do, and not even an unlikely thing
for ME to do), and it's damaged so it's "partially" functional, I'd
rather have it STOP COMPLETELY than, say, become confused and drive a
gantry crane in the wrong direction.

If something like this is done, it should be a configurable option.

-Dave

--
Dave McGuire, AK4HZ
New Kensington, PA

Thor Lancelot Simon

2013-12-31 00:36:35 UTC

Permalink

I couldn't agree less.

Thor

Paul Goyette

2013-12-31 00:45:35 UTC

Permalink

John Hawkinson

2013-12-31 02:27:16 UTC

Permalink

Post by Paul Goyette

Post by Mathieu Lubrano
* The point is : with a headless system you want it back online,

...

Post by Paul Goyette
It is certainly much better to have the predictable known behavior
of "no machine" than to have a machine whose behaviour is unkown and
unknowable.

It seems to me that perhaps a better solution would be to stop booting
and configure a network interface and start an sshd. Now, sometimes
configuring a network interface is hard, but for those of us for whom
it isn't, this seems a path forward.

--***@mit.edu
John Hawkinson

p.s.: Can we drop tech-embed?

Robert Elz

2013-12-31 04:14:29 UTC

Permalink

Date: Mon, 30 Dec 2013 21:27:16 -0500
From: John Hawkinson <***@MIT.EDU>
Message-ID: <***@athena.dialup.mit.edu>

| It seems to me that perhaps a better solution would be to stop booting
| and configure a network interface and start an sshd. Now, sometimes
| configuring a network interface is hard, but for those of us for whom
| it isn't, this seems a path forward.

Seems like a useful solution for some cases to me. And probably not even all
that hard (though whether or not it is productive would depend upon how the
sshd is configured, and whether or not at least one non-root user's .ssh/*
files were available.)

| p.s.: Can we drop tech-embed?

Done.

For what it is worth, for some time now, I have been considering a different
idea to solve a different, but related, problem - I'm not so worried about
systems that I cannot get at, but I have systems that really need to be running
all the time, and need to boot and operate if they possibly can, even if
80% of the resulting system is not there - the rest can wait until I wake up,
or return from wherever, or whatever prevented me from immediately fixing
things.

My issue is largely with fsck and filesystem repair - most of the time
the fsck -p (and/or log recovery) works fine, and the system reboots fully
by itself, but occasionally, there's some filesystem error that needs
assistance. If that's on a crucual filesystem (root /usr /var etc) there's
little that can be done safely, but for most other filesystems the
system will run (with some reduced functionality) just fine - eg /home
isn't really needed on a system that is primarily a nameserver, and pkgsrc
and distfiles, and stuff (and filesystems for building updated NetBSDs)
aren't needed at all until they're actually needed (which for me certainly
means after I can fix them).

My basic strategy would be to enhance fstab in some yet to be determined way
to indicate which filesystems were optional, them simply ignore (without
mounting obviously) any optional filesystem that has a problem (or whose
parent filesystem could not be mounted) and only abort the boot if a
non-optional filesystem failed to check & mount. Since in my case my
non-optional filesystems are (in practice) all remarkably stable (as in
nothing much important normally changes on them - the only significant
changes are the newsyslog updates, and for me, they don't often do anything,
so they are unlikely to have fsck problems) this would allow the system to
get up and running in situations where now it stops and waits for
attention.

I have also considered allowing all the optional filesystem repair and
mounting to happen after the system is running with the essential
filesystems - not useful if the filesystem is clean, or a simple log replay,
is all that is needed, but even fsck -p can be slow, and so delay booting,
if the filesystem in question is a big beast that isn't really essential,
then whatever needs it could wait to start (if it ever starts) until a little
later, while the promary OS functions are running sooner.

I have a gross hack (or more correctly, had on a system I just replaced,
which has not yet been duplicated on the replacement system - but will if
I don't come up with something better) where the most problematic filesystem
was simply listed as noauto, and fsck pass 0, in fstab, then a script run
later (could be from @boot in crontab, or backgrounded from rc.local, or
anything like that) fsck'd it and mounted if OK, or simply newfs'd and mounted
if it was damaged - it gets mounted -o async so if it is active when the
system crashes, it is expected to be damaged - the damage is fine, hanging
the boot because of it would not be.

Because of this kind of thing, the "extended fstab" really needs a way to
specify a recovery script, that is run if the normal fsck -p recovery indicates
failure. I could even imagine that being used for essential filesystems.
On a system where uptime is crucial, if an essential filesystem is damaged,
I could see a script that would newfs and restore from a network fileserver
(a locally tailored script of course) recovering the most recent backup, for
situations where "run now" is more important than losing the minimal possible
data,

kre

Brett Lymn

2013-12-31 06:54:13 UTC

Permalink

Post by Robert Elz
Seems like a useful solution for some cases to me. And probably not even all
that hard (though whether or not it is productive would depend upon how the
sshd is configured, and whether or not at least one non-root user's .ssh/*
files were available.)

Yes, I think so too. Dare I say it but the Solaris developers have done
just this with SMF, it tries to bring up as much of the system as
possible - if you get sshd then, bargain, you can get in and fix the
other faults remotely. An all or nothing approach may preclude this.
As long as the flags are optional then all it is doing is giving the
administrator the traditional UNIX
shoot-yourself-in-the-foot-if-you-want-to option that we are used to but
providing the ability to get a machine up far enough to permit remote
administration.

--
Brett Lymn
Staple Guns: because duct tape doesn't make that KerCHUNK sound - xkcd.com

Hubert Feyrer

2013-12-31 16:27:29 UTC

Permalink

Post by Robert Elz
My basic strategy would be to enhance fstab in some yet to be determined way
to indicate which filesystems were optional, them simply ignore (without
mounting obviously) any optional filesystem that has a problem (or whose
parent filesystem could not be mounted) and only abort the boot if a
non-optional filesystem failed to check & mount. Since in my case my
non-optional filesystems are (in practice) all remarkably stable (as in
nothing much important normally changes on them - the only significant
changes are the newsyslog updates, and for me, they don't often do anything,
so they are unlikely to have fsck problems) this would allow the system to
get up and running in situations where now it stops and waits for
attention.

from /etc/defaults/rc.conf:

# Filesystems to mount early in boot-up.
# Note that `/var' is needed in $critical_filesystems_local (or
# implied as part of `/') as certain services that need /var (such as
# dhclient) may be needed to get the network operational enough to mount
# the $critical_filesystems_remote. Prepending "OPTIONAL:" means it
# will not be an error if that file system is not present in fstab(5).
#
critical_filesystems_local="OPTIONAL:/var"
critical_filesystems_remote="OPTIONAL:/usr"

Maybe that can serve as starting point...

- Hubert

2013-12-31 18:24:40 UTC

Permalink

Post by Hubert Feyrer

# Filesystems to mount early in boot-up.
# Note that `/var' is needed in $critical_filesystems_local (or
# implied as part of `/') as certain services that need /var (such as
# dhclient) may be needed to get the network operational enough to mount
# the $critical_filesystems_remote. Prepending "OPTIONAL:" means it
# will not be an error if that file system is not present in fstab(5).
#
critical_filesystems_local="OPTIONAL:/var"
critical_filesystems_remote="OPTIONAL:/usr"
Maybe that can serve as starting point...
- Hubert

Regarding filesystem issues, it seems fair to use rc.conf settings and some scripting rather than modifying fstab (it may be read and parsed by others). And it may help recovering for most common boot issues.

On the stop_boot side, still I don't like the idea of rc scripts possibly halting the whole boot sequence just for their own concern.
Depending on the use case it's à good or bad thing, yet to me it should be a rc.conf and sysadmin decision.

Bye,
Mathieu Lubrano.

David Holland

2013-12-31 18:53:26 UTC

Permalink

Post by M
Regarding filesystem issues, it seems fair to use rc.conf settings
and some scripting rather than modifying fstab (it may be read and
parsed by others). And it may help recovering for most common boot
issues.

Adding something to the options column of fstab (akin to the existing
"noauto") should not cause a problem in this regard, and it makes more
sense to put this setting there than tucked away somewhere else.

FWIW.

--
David A. Holland
***@netbsd.org

Christos Zoulas

2013-12-31 21:03:22 UTC

Permalink

Post by David Holland

Absolutely, what people are asking for is to change:

fs_spec fs_file fs_vfstype fs_mntops fs_freq fs_passno

to:

fs_spec fs_file fs_vfstype fs_mntops fs_fsckops fs_freq fs_passno

This could be implemented without too much hassle in many different ways.
A hacky way would be to use a separator such as : to split between the
mount and fsck options in fs_mntops and have both in the same string.

Since fs_freq and fs_passno are just integers, we can detect old and
new fstab files so we don't have to put fs_fsckops last.

christos

Erik Berls

2014-03-03 01:04:47 UTC

Permalink

Chiming in late on this:
Instead of just trying heroics or additional path to set up networking,
etc, how about a "booted_cleanly" that critical processes can depend on?
Then the current proposed scheme can blindly try to bring the system back
up, but anything critical can gate.

Post by Christos Zoulas

Post by David Holland

fs_spec fs_file fs_vfstype fs_mntops fs_freq fs_passno
fs_spec fs_file fs_vfstype fs_mntops fs_fsckops fs_freq fs_passno
This could be implemented without too much hassle in many different ways.
A hacky way would be to use a separator such as : to split between the
mount and fsck options in fs_mntops and have both in the same string.
Since fs_freq and fs_passno are just integers, we can detect old and
new fstab files so we don't have to put fs_fsckops last.
christos

m***@lubrano.at

2014-03-03 08:52:40 UTC

Permalink

Hi Erik,=0A=0Awhatever brings a headless or remote system online look goo=
d to me. In a safe fashion, of course.=0APlus, your proposal avoid editin=
g the fstab file.=0A=0ABye,=0AMathieu Lubrano.=0A=0A3 mars 2014 03:06 "Er=
ik Berls" a =C3=A9crit: =0A=0A> Chiming in late on this:=0A> Instead of =
just trying heroics or additional path to set up networking,=0A> etc, how=
about a "booted_cleanly" that critical processes can depend on?=0A> Then=
the current proposed scheme can blindly try to bring the system back=0A>=
up, but anything critical can gate.=0A> =0A> On Tue, Dec 31, 2013 at 1:0=
3 PM, Christos Zoulas wrote:=0A> =0A> =0A>> In article ,=0A>> David Holla=
nd wrote: =0A>> =0A>>> On Tue, Dec 31, 2013 at 07:24:40PM +0100, M wrot=
e: =0A>>> =0A>>>> Regarding filesystem issues, it seems fair to use rc.co=
nf settings=0A>>>> and some scripting rather than modifying fstab (it may=
be read and=0A>>>> parsed by others). And it may help recovering for mos=
t common boot=0A>>>> issues.=0A>>>> =0A>>> =0A>>> =0A>>> Adding something=
to the options column of fstab (akin to the existing=0A>>> "noauto") sho=
uld not cause a problem in this regard, and it makes more=0A>>> sense to =
put this setting there than tucked away somewhere else.=0A>>> =0A>> =0A>>=
=0A>> Absolutely, what people are asking for is to change:=0A>> =0A>> fs=
_spec fs_file fs_vfstype fs_mntops fs_freq fs_passno=0A>> =0A>> to:=0A>> =
=0A>> fs_spec fs_file fs_vfstype fs_mntops fs_fsckops fs_freq fs_passno=
=0A>> =0A>> This could be implemented without too much hassle in many dif=
ferent ways.=0A>> A hacky way would be to use a separator such as : to sp=
lit between the=0A>> mount and fsck options in fs_mntops and have both in=
the same string.=0A>> =0A>> Since fs_freq and fs_passno are just integer=
s, we can detect old and=0A>> new fstab files so we don't have to put fs_=
fsckops last.=0A>> =0A>> christos=0A>> =0A>

Dave McGuire

2013-12-31 00:38:18 UTC

Permalink

Post by Thor Lancelot Simon

I couldn't agree less.

Thor is a man of few words today. ;)

-Dave

--
Dave McGuire, AK4HZ
New Kensington, PA

Mathieu Lubrano

2013-12-31 00:40:55 UTC

Permalink

Post by Dave McGuire

This is, to put it simply, very dangerous.
In fact it DOES make a difference, potentially a very large one. If
I've, say, put a NetBSD system in an industrial control environment
(which is not an unlikely thing to do, and not even an unlikely thing
for ME to do), and it's damaged so it's "partially" functional, I'd
rather have it STOP COMPLETELY than, say, become confused and drive a
gantry crane in the wrong direction.
If something like this is done, it should be a configurable option.

Right, it makes sense to be safe in embedded systems and stop right away. Even for large servers (DB) you want to stop and fall back to a shell and manual fsck.
But for many hosted systems, or headless systems you shouldn't stop the boot sequence for "minor" issues. Hands on operations may be avoided this way (think at colocation, VPS, old Servers and PCs revived with NetBSD...).

So like passing special arguments with fsck_flags, a "headless" flag or something like that in rc.conf, with a safe default setting (no change to the current behavior), seems fine to me.

I assumed it should be discussed from both "embedded" and "user" (like sysadmin) point of view, hence the cross-post...

Post by Dave McGuire
-Dave
--
Dave McGuire, AK4HZ
New Kensington, PA