How to debug systemd “init” freeze during boot?

up vote
3
down vote

favorite

I'm trying to get my own init script running on some ARM hardware and Debian Jessie with systemd. The overall boot concept and init script is almost identical to a x86 variant, which works fine. For both hardwares the whole SD card image is pre-built on a x86 host.

When running on the ARM hardware, having serial console access I see that my init script runs fine up to the point where switch_root is called:

exec switch_root -c /dev/console /newroot /sbin/init

Nothing happens after that. No error message is printed - which makes it difficult to find the problem.

The kernel command line is...

ro root=LABEL=IM_BOOT1 panic=10 mem=256M console=ttyS0,115200 systemd.log_level=debug

...and AFAIK this should make systemd print maximum debug messages. Unfortunately nothing is printed.

Any idea/hint what I could do to understand what could be causing the freeze, or with other words - understand what is going on?

The only strange detail are some warnings in the kernel log before calling switch_root:

<snip> ext4: Unknown symbol jbd2_journal_errno (err 0) ext4: Unknown symbol jbd2_journal_begin_ordered_truncate (err 0) ext4: Unknown symbol jbd2_journal_flush (err 0) ext4: Unknown symbol mb_cache_entry_find_next (err 0) squashfs: version 4.0 (2009/01/31) Phillip Lougher aufs 3.16-20150928 usbhid: Unknown symbol hid_output_report (err 0) usbhid: Unknown symbol hidinput_count_leds (err 0) usbhid: Unknown symbol hid_allocate_device (err 0) usbhid: Unknown symbol hid_destroy_device (err 0) usbhid: Unknown symbol hid_alloc_report_buf (err 0) usbhid: Unknown symbol hid_set_field (err 0) usbhid: Unknown symbol hid_check_keys_pressed (err 0) usbhid: Unknown symbol hid_input_report (err 0) usbhid: Unknown symbol hid_debug (err 0) usbhid: Unknown symbol __hid_request (err 0) usbhid: Unknown symbol hid_parse_report (err 0) usbhid: Unknown symbol hid_add_device (err 0) usbcore: registered new interface driver usbhid usbhid: USB HID core driver usb_storage: Unknown symbol scsi_report_device_reset (err 0) usb_storage: Unknown symbol scsi_remove_host (err 0) usb_storage: Unknown symbol scsi_report_bus_reset (err 0) usb_storage: Unknown symbol scsi_sense_desc_find (err 0) usb_storage: Unknown symbol scsi_eh_prep_cmnd (err 0) usb_storage: Unknown symbol scsi_host_put (err 0) usb_storage: Unknown symbol scsi_scan_host (err 0) usb_storage: Unknown symbol scsi_is_host_device (err 0) <snip>

...however these modules appear to work fine anyway. I don't know if that is related.

Some additional details:

Debian installation is prepared using debootstrap (same method works perfectly with x86)

Custom built Kernel: Linux (none) 3.16.7 #1 Mon Nov 16 08:32:55 UTC 2015 armv5tejl GNU/Linux, patched with AUFS and hardware-specific device tree

both Kernel and root fs are built with help of QEMU on a x86 host

initramfs boots with BusyBox, sets up the AUFS final root fs and finally invokes switch_root. The same method (nearly 100% identical init script) works fine on x86

I can run an interactive shell just before running switch_root and I can't find anything that isn't working as expected at that point

asked Nov 16 '15 at 11:00

Udo G

4602521

Cool toy. I am myself running here an ARM lamobo R1.Does it seems something should be compiled in the kernel not as module, or am I with the wrong impression? Try the armbian forum, it seems the guys have tried to make systemd work.
– Rui F Ribeiro
Nov 16 '15 at 11:03

What makes you think modules are missing? At least in the /newroot environment there should be all modules. There aren't all modules in the initramfs, though, however enough for my init script to run fine. BTW, the HW manufacturer provides a Debian Jessie image w/ systemd, but they are booting from the second partition directly, without an initramfs. BTW, Lamobo R1 looks interesting.
– Udo G
Nov 16 '15 at 12:59

The errors. But I have not dealt with kernel full compilations for a long time in Linux. Lamobo R1 is very interesting, internal switch a boon, pity the wifi is braindead. The HW manufacturer also provides Bananian and openwrt, which are too sloppy hacked for my taste. I prefer so much the work the armbian guys have done.
– Rui F Ribeiro
Nov 16 '15 at 13:47

Tried with a RAMdisk that includes all Kernel modules (19 MB compressed), however the same warnings keep appearing in Kernel log and systemd still freezes... :-( Any idea whatsoever?
– Udo G
Nov 17 '15 at 8:16

Besides one last try about compiling a kernel with them instead of using them as modules, not really.
– Rui F Ribeiro
Nov 17 '15 at 8:19

add a comment |

up vote
3
down vote

favorite

When running on the ARM hardware, having serial console access I see that my init script runs fine up to the point where switch_root is called:

exec switch_root -c /dev/console /newroot /sbin/init

Nothing happens after that. No error message is printed - which makes it difficult to find the problem.

The kernel command line is...

ro root=LABEL=IM_BOOT1 panic=10 mem=256M console=ttyS0,115200 systemd.log_level=debug

...and AFAIK this should make systemd print maximum debug messages. Unfortunately nothing is printed.

Any idea/hint what I could do to understand what could be causing the freeze, or with other words - understand what is going on?

The only strange detail are some warnings in the kernel log before calling switch_root:

...however these modules appear to work fine anyway. I don't know if that is related.

Some additional details:

Debian installation is prepared using debootstrap (same method works perfectly with x86)

Custom built Kernel: Linux (none) 3.16.7 #1 Mon Nov 16 08:32:55 UTC 2015 armv5tejl GNU/Linux, patched with AUFS and hardware-specific device tree

both Kernel and root fs are built with help of QEMU on a x86 host

initramfs boots with BusyBox, sets up the AUFS final root fs and finally invokes switch_root. The same method (nearly 100% identical init script) works fine on x86

I can run an interactive shell just before running switch_root and I can't find anything that isn't working as expected at that point

asked Nov 16 '15 at 11:00

Udo G

4602521

Cool toy. I am myself running here an ARM lamobo R1.Does it seems something should be compiled in the kernel not as module, or am I with the wrong impression? Try the armbian forum, it seems the guys have tried to make systemd work.
– Rui F Ribeiro
Nov 16 '15 at 11:03

What makes you think modules are missing? At least in the /newroot environment there should be all modules. There aren't all modules in the initramfs, though, however enough for my init script to run fine. BTW, the HW manufacturer provides a Debian Jessie image w/ systemd, but they are booting from the second partition directly, without an initramfs. BTW, Lamobo R1 looks interesting.
– Udo G
Nov 16 '15 at 12:59

The errors. But I have not dealt with kernel full compilations for a long time in Linux. Lamobo R1 is very interesting, internal switch a boon, pity the wifi is braindead. The HW manufacturer also provides Bananian and openwrt, which are too sloppy hacked for my taste. I prefer so much the work the armbian guys have done.
– Rui F Ribeiro
Nov 16 '15 at 13:47

Tried with a RAMdisk that includes all Kernel modules (19 MB compressed), however the same warnings keep appearing in Kernel log and systemd still freezes... :-( Any idea whatsoever?
– Udo G
Nov 17 '15 at 8:16

Besides one last try about compiling a kernel with them instead of using them as modules, not really.
– Rui F Ribeiro
Nov 17 '15 at 8:19

add a comment |

up vote
3
down vote

favorite

When running on the ARM hardware, having serial console access I see that my init script runs fine up to the point where switch_root is called:

exec switch_root -c /dev/console /newroot /sbin/init

Nothing happens after that. No error message is printed - which makes it difficult to find the problem.

The kernel command line is...

ro root=LABEL=IM_BOOT1 panic=10 mem=256M console=ttyS0,115200 systemd.log_level=debug

...and AFAIK this should make systemd print maximum debug messages. Unfortunately nothing is printed.

Any idea/hint what I could do to understand what could be causing the freeze, or with other words - understand what is going on?

The only strange detail are some warnings in the kernel log before calling switch_root:

...however these modules appear to work fine anyway. I don't know if that is related.

Some additional details:

Debian installation is prepared using debootstrap (same method works perfectly with x86)

Custom built Kernel: Linux (none) 3.16.7 #1 Mon Nov 16 08:32:55 UTC 2015 armv5tejl GNU/Linux, patched with AUFS and hardware-specific device tree

both Kernel and root fs are built with help of QEMU on a x86 host

initramfs boots with BusyBox, sets up the AUFS final root fs and finally invokes switch_root. The same method (nearly 100% identical init script) works fine on x86

I can run an interactive shell just before running switch_root and I can't find anything that isn't working as expected at that point

asked Nov 16 '15 at 11:00

Udo G

4602521

When running on the ARM hardware, having serial console access I see that my init script runs fine up to the point where switch_root is called:

exec switch_root -c /dev/console /newroot /sbin/init

Nothing happens after that. No error message is printed - which makes it difficult to find the problem.

The kernel command line is...

ro root=LABEL=IM_BOOT1 panic=10 mem=256M console=ttyS0,115200 systemd.log_level=debug

...and AFAIK this should make systemd print maximum debug messages. Unfortunately nothing is printed.

Any idea/hint what I could do to understand what could be causing the freeze, or with other words - understand what is going on?

The only strange detail are some warnings in the kernel log before calling switch_root:

...however these modules appear to work fine anyway. I don't know if that is related.

Some additional details:

Debian installation is prepared using debootstrap (same method works perfectly with x86)

Custom built Kernel: Linux (none) 3.16.7 #1 Mon Nov 16 08:32:55 UTC 2015 armv5tejl GNU/Linux, patched with AUFS and hardware-specific device tree

both Kernel and root fs are built with help of QEMU on a x86 host

initramfs boots with BusyBox, sets up the AUFS final root fs and finally invokes switch_root. The same method (nearly 100% identical init script) works fine on x86

I can run an interactive shell just before running switch_root and I can't find anything that isn't working as expected at that point

boot systemd init-script initramfs

asked Nov 16 '15 at 11:00

Udo G

4602521

asked Nov 16 '15 at 11:00

Udo G

4602521

asked Nov 16 '15 at 11:00

Udo G

4602521

asked Nov 16 '15 at 11:00

Udo G

4602521

asked Nov 16 '15 at 11:00

Udo G

4602521

Cool toy. I am myself running here an ARM lamobo R1.Does it seems something should be compiled in the kernel not as module, or am I with the wrong impression? Try the armbian forum, it seems the guys have tried to make systemd work.
– Rui F Ribeiro
Nov 16 '15 at 11:03

What makes you think modules are missing? At least in the /newroot environment there should be all modules. There aren't all modules in the initramfs, though, however enough for my init script to run fine. BTW, the HW manufacturer provides a Debian Jessie image w/ systemd, but they are booting from the second partition directly, without an initramfs. BTW, Lamobo R1 looks interesting.
– Udo G
Nov 16 '15 at 12:59

The errors. But I have not dealt with kernel full compilations for a long time in Linux. Lamobo R1 is very interesting, internal switch a boon, pity the wifi is braindead. The HW manufacturer also provides Bananian and openwrt, which are too sloppy hacked for my taste. I prefer so much the work the armbian guys have done.
– Rui F Ribeiro
Nov 16 '15 at 13:47

Tried with a RAMdisk that includes all Kernel modules (19 MB compressed), however the same warnings keep appearing in Kernel log and systemd still freezes... :-( Any idea whatsoever?
– Udo G
Nov 17 '15 at 8:16

Besides one last try about compiling a kernel with them instead of using them as modules, not really.
– Rui F Ribeiro
Nov 17 '15 at 8:19

add a comment |

Cool toy. I am myself running here an ARM lamobo R1.Does it seems something should be compiled in the kernel not as module, or am I with the wrong impression? Try the armbian forum, it seems the guys have tried to make systemd work.
– Rui F Ribeiro
Nov 16 '15 at 11:03

What makes you think modules are missing? At least in the /newroot environment there should be all modules. There aren't all modules in the initramfs, though, however enough for my init script to run fine. BTW, the HW manufacturer provides a Debian Jessie image w/ systemd, but they are booting from the second partition directly, without an initramfs. BTW, Lamobo R1 looks interesting.
– Udo G
Nov 16 '15 at 12:59

The errors. But I have not dealt with kernel full compilations for a long time in Linux. Lamobo R1 is very interesting, internal switch a boon, pity the wifi is braindead. The HW manufacturer also provides Bananian and openwrt, which are too sloppy hacked for my taste. I prefer so much the work the armbian guys have done.
– Rui F Ribeiro
Nov 16 '15 at 13:47

Tried with a RAMdisk that includes all Kernel modules (19 MB compressed), however the same warnings keep appearing in Kernel log and systemd still freezes... :-( Any idea whatsoever?
– Udo G
Nov 17 '15 at 8:16

Besides one last try about compiling a kernel with them instead of using them as modules, not really.
– Rui F Ribeiro
Nov 17 '15 at 8:19

Cool toy. I am myself running here an ARM lamobo R1.Does it seems something should be compiled in the kernel not as module, or am I with the wrong impression? Try the armbian forum, it seems the guys have tried to make systemd work.
– Rui F Ribeiro
Nov 16 '15 at 11:03

What makes you think modules are missing? At least in the /newroot environment there should be all modules. There aren't all modules in the initramfs, though, however enough for my init script to run fine. BTW, the HW manufacturer provides a Debian Jessie image w/ systemd, but they are booting from the second partition directly, without an initramfs. BTW, Lamobo R1 looks interesting.
– Udo G
Nov 16 '15 at 12:59

The errors. But I have not dealt with kernel full compilations for a long time in Linux. Lamobo R1 is very interesting, internal switch a boon, pity the wifi is braindead. The HW manufacturer also provides Bananian and openwrt, which are too sloppy hacked for my taste. I prefer so much the work the armbian guys have done.
– Rui F Ribeiro
Nov 16 '15 at 13:47

Tried with a RAMdisk that includes all Kernel modules (19 MB compressed), however the same warnings keep appearing in Kernel log and systemd still freezes... :-( Any idea whatsoever?
– Udo G
Nov 17 '15 at 8:16

Besides one last try about compiling a kernel with them instead of using them as modules, not really.
– Rui F Ribeiro
Nov 17 '15 at 8:19

add a comment |

1 Answer
1

active

oldest

votes

up vote
0
down vote

It turned out that the Kernel was missing some important settings, that are required by systemd.

http://cgit.freedesktop.org/systemd/systemd/tree/README shows what Kernel config settings were necessary.

Once configured, systemd started working immediately.

answered Dec 10 '15 at 15:11

Udo G

4602521

add a comment |

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f243288%2fhow-to-debug-systemd-init-freeze-during-boot%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

up vote
0
down vote

It turned out that the Kernel was missing some important settings, that are required by systemd.

http://cgit.freedesktop.org/systemd/systemd/tree/README shows what Kernel config settings were necessary.

Once configured, systemd started working immediately.

answered Dec 10 '15 at 15:11

Udo G

4602521

add a comment |

up vote
0
down vote

It turned out that the Kernel was missing some important settings, that are required by systemd.

http://cgit.freedesktop.org/systemd/systemd/tree/README shows what Kernel config settings were necessary.

Once configured, systemd started working immediately.

answered Dec 10 '15 at 15:11

Udo G

4602521

add a comment |

up vote
0
down vote

It turned out that the Kernel was missing some important settings, that are required by systemd.

http://cgit.freedesktop.org/systemd/systemd/tree/README shows what Kernel config settings were necessary.

Once configured, systemd started working immediately.

answered Dec 10 '15 at 15:11

Udo G

4602521

It turned out that the Kernel was missing some important settings, that are required by systemd.

http://cgit.freedesktop.org/systemd/systemd/tree/README shows what Kernel config settings were necessary.

Once configured, systemd started working immediately.

answered Dec 10 '15 at 15:11

Udo G

4602521

answered Dec 10 '15 at 15:11

Udo G

4602521

answered Dec 10 '15 at 15:11

Udo G

4602521

answered Dec 10 '15 at 15:11

Udo G

4602521

add a comment |

draft saved

draft discarded

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Cfrtjryk