[Solved] Manjaro update broke my os, I can't figure out how to fix it

[FalconFlurry]

I originally posted this question on Reddit, but I wasn't able to get the help I needed, so I'm really hoping someone can help me here. Here is my original conversation on Reddit for reference, below is a summary of most of the things I have tried:

I am running Manjaro XFCE dual boot with windows 10. Windows 10 was also broken by a forced update, so I can't use my computer at all right now, but that's another topic.
I installed the 5.10 kernel update and when I rebooted I got a black screen with this error:
Warning: /lib/modules/5.10.2.-2-MANJARO/modules.devname not found - ignoring mount: /new_root: unknown filesystem type ‘ext4’ You are not being dropped into an emergency shell. sh: can’t access tty: job control turned off [rootfs ]#

I googled the problem and it sounds like this was a common problem for people with nvidia drivers. I looked for solutions and I found a few options that others said worked for them, but I couldn't get them to work and now I have no idea how to fix this. Obviously reinstalling Manjaro is an option, but I want to leave that as a last resort because it will be a lot of work to get everything back the way I had it.
One option I tried was to load an older kernel from the GRUB menu and reinstall the update from there. I tried that, 5.4 was available, but when I loaded it it got hung up indefinitely on the Aorus logo.
I tried loading a live usb, then reinstalling the update by chroot. This is what I tried to do:
sudo manjaro-chroot -a sudo pacman -Syyu pacman -S linux510 exit

But after entering the first line it said:
sudgrub-probe: error: cannot find a GRUB drive for /dev/sdc1. Check your device map. grub-probe: error: cannot find a GRUB drive for /dev/sdc1. Check your device map. o ==> Detected systems: --> 0) ManjaroLinux ==> Select system to mount [0-0] :

I entered "0" to see what would happen and it said:
==> ERROR: You can't mount 0!

It seemed like the partitions were not mounted, so I tried creating a directory to mount into:

$ sudo mkdir /rescue $ sudo mount /dev/sda3 /rescue $ sudo mount /dev/sda1 /rescue/boot $ sudo mount -t proc proc /rescue/proc $ sudo mount -t sysfs sys /rescue $ sudo mount -o bind /dev /rescue/dev $ sudo mount -t devpts pts /rescue/dev/pts $ sudo chroot /rescue # sudo pacman -Syuu # sudo pacman -S linux510

It went through the update and installation process and at the end it said the kernel has been updated, but there were a number of errors and warnings. It said:

(4/5) Updating Grub-Bootmenu /usr/bin/grub-editenv: error: cannot open `//boot/grub/grubenv.new': No such file or directory. /usr/bin/grub-mkconfig: line 259: /boot/grub/grub.cfg.new: No such file or directory error: command failed to execute correctly

It also had hundreds of lines with warnings that read:

warning: could not get file information for usr/lib/modules/5.10.2-2-MANJARO/kernel/...

with the rest of the directory being different on every line, something like .../sound/soc/intel/boards/snd-soc-sof-sdw.ko.xz

I typed exit and rebooted and nothing had changed.

After booting into the live usb again I tried something else:

$ sudo mount /dev/sda3 /mnt $ sudo mount /dev/sda1 /mnt/boot $ sudo chroot /mnt # sudo pacman -Syyu

it said there is nothing to do. I typed

# sudo pacman -S linux510

It said everything is up to date, after proceeding with the installation I got the following errors:

error: could not open file: /etc/mtab: No such file or directory error: could not determine filesystem mount points error: not enough free disk space error: failed to commit transaction (unexpected error) Errors occurred, no packages were upgraded.

I don't know what else to try and I don't know what I'm doing wrong. If anyone can help me I would really appreciate it.

 

[wizardfromoz]

Ouch, painful. You've certainly given it your best shot.

G'day @FalconFlurry and welcome to linux.org.

I'll put together a few thoughts and run them by you on my tomorrow (I am in Australia), as I run half a dozen Manjaroes and half a dozen other Arch-based Distros. I may not have the solution, but should be able to clarify for you and offer ideas in what direction to proceed.

BTW I note from your Reddit attempt

I'll try to get help from the linux forum...

If that is us, I should just let you know

Just in case of any misapprehension on your part, we are not an official arm nor organ of Linux, just scored the dot org name - we are manned by volunteer staff who share a love of Linux and have varying skills in various departments.

That being said, we will do our best.

Cheers

Chris Turner
wizardfromoz

 

[Deleted member 108694]

Have you tried re-installing the kernel - remove anything that is related to 5.10 first then try for a re-install if that still fails then revert back to kernel 5.4 probably will have to wait for NVIDIA to catch up with the new kernel - probably best to install kernel 5.4 first then reboot to that kernel then remove 5.10

 

[PedroM]

I used manjaro and it was very common that after an update it would stop working. It was always because it corrupted the boot. I was fixing it using a live usb and reinstalling without formatting the partitions. I'm tired of doing this, so I switched to Endeavor OS, which is another arch-based distribution, and is not complicated at all to install. Since then everything works perfectly. If you have to reinstall, give it a try. I'm sorry for my English. Good luck

 

[wizardfromoz]

I'm sorry for my English

Never be sorry for helping, Pedro, we wish we spoke better Spanish

This current issue appears to be connected with an update Manjaro issued 30 December where the 5.10 kernel removed support for certain nVidia drivers/cards.

I have the Manjaro Xfce the OP has, but my rig is AMD/Radeon.

Manjaro have a banner at the top of their Forum

The OP may have already seen this, as he says

I googled the problem and it sounds like this was a common problem for people with nvidia drivers

Is that so, @FalconFlurry ?

You might be best advised to have a read at their Forum

https://forum.manjaro.org

and if need be sign up and ask for help.

I'll have some other ideas a little later in my day. One of them is to see if you can use manjaro-chroot to change the drivers from nVidia to the generic Nouveau, which should be on your install, but maybe blacklisted at the moment.

Cheers

Wizard

 

[FalconFlurry]

Never be sorry for helping, Pedro, we wish we spoke better Spanish

This current issue appears to be connected with an update Manjaro issued 30 December where the 5.10 kernel removed support for certain nVidia drivers/cards.

I have the Manjaro Xfce the OP has, but my rig is AMD/Radeon.

Manjaro have a banner at the top of their Forum

The OP may have already seen this, as he says



Is that so, @FalconFlurry ?

You might be best advised to have a read at their Forum

https://forum.manjaro.org

and if need be sign up and ask for help.

I'll have some other ideas a little later in my day. One of them is to see if you can use manjaro-chroot to change the drivers from nVidia to the generic Nouveau, which should be on your install, but maybe blacklisted at the moment.

Cheers

Wizard

Hey, thanks for taking time to try and help me. I did see that banner Manjaro posted. I am just on my way to work, so I will take a look at the Manjaro forum and see if I can find anything after work

 

[FalconFlurry]

I used manjaro and it was very common that after an update it would stop working. It was always because it corrupted the boot. I was fixing it using a live usb and reinstalling without formatting the partitions. I'm tired of doing this, so I switched to Endeavor OS, which is another arch-based distribution, and is not complicated at all to install. Since then everything works perfectly. If you have to reinstall, give it a try. I'm sorry for my English. Good luck

So to fix the boot you basically just did a fresh install without formatting the partitions?

 

[wizardfromoz]

so I will take a look at the Manjaro forum

I am a Member there nominally, and have both received a little help from them, and helped them with one or two matters over the last 6 years, and they seem like a good crowd.

If you learn anything useful, maybe bring it back to us so we can add it to our wiki.

You could let us know if you know what series and drivers your nVidia uses, so we can scout around for compatibility, or just focus on the Manjaro forum for now. I appreciate it has been at least 8 days for you without access.

Cheers

Wizard

 

[FalconFlurry]

You could let us know if you know what series and drivers your nVidia uses, so we can scout around for compatibility, or just focus on the Manjaro forum for now. I appreciate it has been at least 8 days for you without access

It's actually been closer to two weeks as I tried to fix it myself before looking for help.

Anyway, I couldn't tell yeu exactly which drivers are installed without being able to access the os (if there is a way to do that please let me know), however I can tell you that the GPU I am running is an MSI GeForce GTX 1080. This version of Manjaro was fairly fresh as I just reinstalled it about a month ago. When I did the install, at the GRUB menu I went to the drivers option and changed it from free to non-free (I believe it was called something like that) and that automatically installed Nvidia's drivers.

 

[wizardfromoz]

I understand and ta for the info (ta is Aussie and British for thanks, pron. "tar").

We can probably track the drivers either through Google or else using a Manjaro Live USB stick if you still have it.

Just a little brainstorming in the meantime is that depending on the config of your rig (number of drives, capacity of each, where Windows and Manjaro are situated now, &c), you could always install another Linux Distro to run side by side with them.

If you like Xfce DE (Desktop Environment) for example, it could be a LInux Mint Xfce or other.

That would get you up and running after 2 weeks without, while you/we/Manjaro Forum work on the Manjaro issue.

If you go to Manjaro Forum, just tell them their Member @wizardfromoz sent you (of course, they may say "That ponced-up PITA? Oh well, we'll treat this guy at face value" ).

In that event, also link them to this Thread so that they can see what we have covered so far. That is not only a courtesy to them, but also sensible, so that none of us duplicates the other and wastes time.

Obviously reinstalling Manjaro is an option, but I want to leave that as a last resort because it will be a lot of work to get everything back the way I had it.

Yet another option - make space for and install another iteration (instance, install) of the same Manjaro. That will work.

If what you mentioned involved your settings, and any other data contained in your Home folder or partition, then you can simply copy that to the new install, and once it is established that it works, blow away the broken original and reclaim the space.

I should qualify that and say, if you have a Home folder, you can copy the folder and contents to the new. If you have a Home partition, you can simply point the install of the new Manjaro to the existing Home partition to work from.

Finally, once you have your Manjaro up and running, we can introduce you to Timeshift, which is like Windows restore but better, and it will save your bacon in future.

I could go on, but digest what is above first, and do anything when you are bright-eyed and bushy-tailed if you can.

Cheers

Wizard

 

[FalconFlurry]

Thanks again for the help @wizardfromoz

My current setup is this:
-1 500GB SSD dedicated to Manjaro
-1 2TB HDD split into two 1TB partitions, one side is for windows, the other is my "Archive" drive, it's formatted for data storage and has no OS so it can be accessed by both Windows and Linux. I also have the ability to add additional hard drives if necessary. Unfortunately i'm not sure how to adjust the partitions right now as I don't have access to either OS (unbelievable luck)

I actually tried Mint when I first went to linux, that is what I wanteed to use, but I was having major framerate issues with it that I couldn't figure out so I switched to Manjaro. In retrospect it was probably also a driver issue.

I currently have my entire Home folder backed up to my Archive drive, but my main issue with reinstalling was I had a very customized ui, I was running Cinnamon DE and had done a lot of tinkering to get it how I wanted it. Then I also have a lot of programs and games to install. It will probably take several days to get it back how I had it.

 

[wizardfromoz]

I currently have my entire Home folder backed up to my Archive drive, but my main issue with reinstalling was I had a very customized ui, I was running Cinnamon DE and had done a lot of tinkering to get it how I wanted it. Then I also have a lot of programs and games to install. It will probably take several days to get it back how I had it.

No that is really good, and proactive

If you already know this, some of The Viewers can benefit.

When you are looking at the contents of your Home folder through your File Manager (Nemo under Cinnamon), the default view only shows the tip of the iceberg.

Example (this from Xubuntu where I am currently working)

But if you press Ctrl-h it shows what is hidden (and you can set this in your preferences), as seen

The dot . before the name of files and folders is what makes them hidden.

So in the folders marked .config, .cache, .local &c are contained the configuration settings you have changed since the install. .mozilla has Firefox settings if you are using Firefox.

Here, too will likely be many of the data associated with your games.

I have more for you, but have some mundane personal tasks to address first and will be back when I can with more.

Cheers

Wizard
BTW if you do install another Manjaro before I can talk with you, don't run any updates on it yet, as it may simply repeat the problem. And start reading up on Timeshift, it will save your bacon in future.

 

[FalconFlurry]

@wizardfromoz I did know about the hidden folders, although I did not know that's what the dot meant.

So a few things have happened, it feels like everything is actively working against me, but I think I finally got it mostly figured. I kept track of what I did, as much as I could, sorry for the lack of formatting, I'm writing on my phone (obviously) and I don't have the energy or time to fix it. Here's what I did:

-Went to Manjaro forum to make an account and post question. Came across this article: https://forum.manjaro.org/t/how-to-save-your-manjaro-installation-when-it-breaks/3902 and noticed the instructions were slightly different than the things I had previously tried, so I read it through and followed the steps through my live usb.

-When I got to the step for installing manjaro-chroot I got four errors saying that the core, extra, community and multilib databases did not exist. I messed around for a while trying to figure out how to install them, eventually I got it figured out. I tried to install manjaro-chroot again and I got a whole host of fresh new errors.

-I tried installing from the graphical package manager instead, it ran the installation, when it seemed like it was complete I realized it actually just froze at 100%, I was forced to reboot.

-After reboot I needed to get into terminal again, but I noticed on the desktop there was a terminal icon, I thought I would click that instead.

-Instead of opening terminal it launched the terminal based Manjaro Architect Installer. It turns out this tool seemed to be designed specifically for doing what i've been trying to do. It allows you to mount your partitions and either do a fresh manjaro install or do a system rescue with chroot

-I went through the steps and chose these options:
5 System Rescue
> 7 Chroot into Installation
> 1 Prepare Installation
> 8 Mount Partitions
(Select ROOT partition) > /dev/sda3
> Do not format
(Select SWAP) > None
(Select additional partitions) > Done
(Select UEFI partition) > /dev/sda1
(Reformat UEFI partition?) > No
> (*) /boot/efi
> 12 Enable fsck hook
> Yes
> 13 Back
> 5 System Rescue
> 7 Chroot into Installation

-The utility seemed to take care of everything i've been trying to do until now. It successfully chrooted me into my root partition.

-I attempted my original steps again:
sudo pacman -Syyu

-there were a few errors i noticed in the log:
warning: could not get file information for boot/linux510 - x86_64.kver
Failed to open file "sys/devices/system/cpu/microcode/reload": Read-only file system
error: command failed to execute correctly
Besides those errors the process seemed to complete and then I went to the next step:
sudo pacman -S linux510
It told me it was already up to date but besides that it completed.

-It warned me that the computer hasn't been restarted for a while and that I'm running an old Kernel and asked to restart my computer. This was the last message I got when I originally updated my computer the time when it broke my OS. This time I was a little more hopeful.

-I entered mhwd-kernel -li

-It said I am currently running 5.8.16-2-MANJARO and that linux 510 and 54 are installed.

-exit

-Terminal had an aneurysm and completely spazzed out flickering between screens rapidly until eventually it closed. I rebooted.

-It didn't make it to the grub menu this time, I got a black screen with this error:
[Failed] Failed to start Light Display Manager

-I booted back into the live usb

-Loaded the Architect Installer again, did the same thing, but first I tried reinstalling the boot partition and the nvidia drivers. It installed 6 packages including nvidia-390.141 -1, -3, and -6

-I tried to install linux58, it didn't work. It said "target not found"

-I tried installing linux54, it worked, although I got the error: cannot find a grub drive for /devsdc1. Check your device.map.

-I typed exit, same spaz happened again. I closed it, relaunched it and did everything again except his time I didn't chroot since it was crashing on exit, instead I went into the option for uninstalling software and selected linux510 and the nvidia driver for linux510. (Sorry for the vague description, i rebooted before I recorded what I did by accident), hit enter, it took me back to the previous menu. I went back to the main menu and hit done. It said the base was not installed or something, asked if I was sure I wanted to exit. I said yes.

-I'm pretty sure all that did nothing, I have a feeling that the only thing that actually changed was rebuilding the boot partition.

-I rebooted and to my surprise it worked! Limited success however as there is no longer a grub menu, so I need to get that back so I can boot to windows. When I loaded my desktop I got a message saying something about my graphics drivers being old or missing and that I could experience major performance issues. I checked the nvidia control panel and it said v390 was installed. I'm not really sure how to check if those drivers are up to date. I checked the kernel version and it said 5.10, so that's good, not sure how that worked though. I also encountered the problem that prompted me to update originally: my internet only half works, some websites like Google work fine but others like amazon instantly get an error "The OCSP response is not yet valid". Idk if that's a Firefox issue or a driver issue or something, but that's probably a topic for a new thread and another day. I am burnt out, it's after midnight and I have to work early, so i'll have to deal with the rest of these problems tomorrow.

Thank you so much for the help!

Edit: I figured out the oscp response error. Because my computer had been unplugged for so long the BIOS clock was a week behind. I set the clock and that fixed it.