Restart Python3 scripts if they exit/fail

Linx1

New Member
Joined
Jan 28, 2022
Messages
15
Reaction score
0
Credits
153
Hi, I am trying to get 2 Python3 scripts to restart if there's an error or other occurrence. I have a Raspberry Pi running on Debian 24/7 - Buster.

Right now, I have the 2 Python scripts running in the "Screen" program because I didn't want to leave SSH terminals open (I use my Raspberry Pi set-up headless.)
For the most part, the scripts are pretty stable, but I've had them drop off of Screen before and didn't even know it.

I've tried various ways to get this working, including making the 2 Python scripts into services, but I would receive various exit codes and couldn't figure out what the problem was (one had to do with user authorization or something like that.) Both scripts work together. The only solution that gets them to work is either running them in the background with Screen or running them from a terminal window and just leave those windows open.

Also, both scripts do load on on re-boot, as they are listed within Crontab -e.

To recap, the main thing I am trying to do is get them to re-load themselves if they exit or have an error for any reason (getting an email would be great, too.) Very open to going back to making these as a systemd service if I can get some help with that part.

Both scripts are located at /home/pi/

Thank you
 


digitaltrails

Member
Joined
Dec 18, 2021
Messages
37
Reaction score
16
Credits
353
This tutorial outlines how to create a systemd user service, and how to have it start at boot:
https://www.unixsysadmin.com/systemd-user-services/

It seems straight forward enough. I've previously set up system systemd-services, and user systemd-timer jobs, but not user services. The linked tutorial does seem to accurately cover what I think would be required for a user service. The first part of the the tutorial is mainly concerned with setting up a user account, if you already done that, just skim through to the part where the services are created.
 
OP
L

Linx1

New Member
Joined
Jan 28, 2022
Messages
15
Reaction score
0
Credits
153
This tutorial outlines how to create a systemd user service, and how to have it start at boot:
https://www.unixsysadmin.com/systemd-user-services/

It seems straight forward enough. I've previously set up system systemd-services, and user systemd-timer jobs, but not user services. The linked tutorial does seem to accurately cover what I think would be required for a user service. The first part of the the tutorial is mainly concerned with setting up a user account, if you already done that, just skim through to the part where the services are created.
Appreciate it - I'll check it out. I'm probably going to need a bit more guidance with building the .service file, as I think that's the issue and what I am not doing right.
 

digitaltrails

Member
Joined
Dec 18, 2021
Messages
37
Reaction score
16
Credits
353
Appreciate it - I'll check it out. I'm probably going to need a bit more guidance with building the .service file, as I think that's the issue and what I am not doing right.

The following creates a user service that is started at boot. Should the python program exit for any reason, systemd will restart the service after delaying 60 seconds.

As the pi user...

Make the necessary service directories and use your prefered editor to create a pi.service file (or any name_you_like.service):
Code:
% mkdir -p /home/pi/.config/systemd/user/
% vi /home/pi/.config/systemd/user/pi.service

Content of pi.service:
Code:
[Unit]
Description=pi application

[Service]
ExecStart=/usr/bin/python3 /home/pi/pi_code.py
WorkingDirectory=/home/pi
Restart=always
RestartSec=60

[Install]
WantedBy=default.target

Tell systemctl to check for any new config files, then enable the service for start at boot, start the service now as well:
Code:
% systemctl --user daemon-reload
% systemctl --user status pi.service
Enable for start up boot
Code:
% systemctl --user enable pi.service
Start it going and check if it's OK:
Code:
% systemctl --user start pi.service
% systemctl --user status pi.service
Anything the service outputs to standard out/error will wind up in the journal:
Code:
% journalctl --boot

Note: a user service runs outside any login session and some login-session related items won't be available, things such as the X11/Wayland or DBUS. You could potentially make things such as DISPLAY or DBUS available via config files, lookups, etc.

I've verified that the above works except for rebooting (I'm not in a position to reboot at the moment).
 
OP
L

Linx1

New Member
Joined
Jan 28, 2022
Messages
15
Reaction score
0
Credits
153
The following creates a user service that is started at boot. Should the python program exit for any reason, systemd will restart the service after delaying 60 seconds.

As the pi user...

Make the necessary service directories and use your prefered editor to create a pi.service file (or any name_you_like.service):
Code:
% mkdir -p /home/pi/.config/systemd/user/
% vi /home/pi/.config/systemd/user/pi.service

Content of pi.service:
Code:
[Unit]
Description=pi application

[Service]
ExecStart=/usr/bin/python3 /home/pi/pi_code.py
WorkingDirectory=/home/pi
Restart=always
RestartSec=60

[Install]
WantedBy=default.target

Tell systemctl to check for any new config files, then enable the service for start at boot, start the service now as well:
Code:
% systemctl --user daemon-reload
% systemctl --user status pi.service
Enable for start up boot
Code:
% systemctl --user enable pi.service
Start it going and check if it's OK:
Code:
% systemctl --user start pi.service
% systemctl --user status pi.service
Anything the service outputs to standard out/error will wind up in the journal:
Code:
% journalctl --boot

Note: a user service runs outside any login session and some login-session related items won't be available, things such as the X11/Wayland or DBUS. You could potentially make things such as DISPLAY or DBUS available via config files, lookups, etc.

I've verified that the above works except for rebooting (I'm not in a position to reboot at the moment).

Very much appreciated. A big difference is I didn't create that directory that you created. One thing that I noticed is that one of the scripts was not finding a configuration file it needs, even though it is in the home/pi directory. I was putting the .service files under /etc/systemd/system, maybe that was at least one of the issues. I'll give this a try...thank you.
 
OP
L

Linx1

New Member
Joined
Jan 28, 2022
Messages
15
Reaction score
0
Credits
153
Hi, when I try to enable each of the 2 python3 scripts, I am getting the below message - note: I added (name) below just so you can see what it's saying (it would be one of the names of the 2 different services I named/created with nano):

Failed to enable unit: Unit file /home/pi/.config/systemd/user/default.target.wants/(name).service does not exist.

By the way, 1 of the 2 scripts requires a .cfg file which is located in the same directory (home/pi/) as the 2 python scripts. I assume this will be accessible to the 1 script that requires it when it runs?
I am wondering if both scripts can be run from the same .service?

I can start the services, but no output is coming from the logs. Status is showing ".service; disabled; vendor preset: enabled."

The interesting thing is a I can run these with simple commands such as "Python3 name.py" with no problem..but running these through systemd just isn't working.
 
Last edited:

digitaltrails

Member
Joined
Dec 18, 2021
Messages
37
Reaction score
16
Credits
353
One way you could get that message is if you are logged in as someone other than user who will be running the service. In my example I choose to run the service as a user called pi. So I would become the pi user by logging in via "su - pi" or ssh [email protected] or similar commands, then all the commands and edits would be issued by the pi user. None of the example requires you to be root or any other user. The idea behind a --user service is that it run under a relatively unprivileged user account.
 
OP
L

Linx1

New Member
Joined
Jan 28, 2022
Messages
15
Reaction score
0
Credits
153
Ok, interesting. I am logging in under "pi" via SSH. This is what I have struggled with for weeks to try to figure out: How the Python3 scripts work without any issues running the command directly (Python3 nameofscrip.py) or via the Screen method. What am I missing?:) Is there any log or output I could send that would help figuring out where the error(s) are? Alternatively, since Screen seems to work well with these scripts, is there any way to set up Screen to automatically re-load the scripts if they fail? If there is a way to do that, my problem is solved.
 
Last edited:

digitaltrails

Member
Joined
Dec 18, 2021
Messages
37
Reaction score
16
Credits
353
The output from the following command might provide some clues (replace pi with the name of your service):
Code:
systemctl --user status pi.service

A very simple alternative would be to embed the python command line inside a bash script that loops, something like:
Code:
#!/bin/bash
while : 
do
     python3 x.py
     echo "WARN: exited with return code $? restarting in 5 seconds"
     sleep 5
done

Then have screen run the bash script (make it executable with chmod +x my_bash_script).

I have to go trim some hedges now, so I will not reply again for a few hours.
 
OP
L

Linx1

New Member
Joined
Jan 28, 2022
Messages
15
Reaction score
0
Credits
153
The output from the following command might provide some clues (replace pi with the name of your service):
Code:
systemctl --user status pi.service

A very simple alternative would be to embed the python command line inside a bash script that loops, something like:
Code:
#!/bin/bash
while :
do
     python3 x.py
     echo "WARN: exited with return code $? restarting in 5 seconds"
     sleep 5
done

Then have screen run the bash script (make it executable with chmod +x my_bash_script).

I have to go trim some hedges now, so I will not reply again for a few hours.

Yes, when I do the status, that's where it shows the service is disabled (for both scripts.)

For the bash script, I have 2 scripts. Should I make a bash script for each one or somehow include it in one? Any way to have that bash script use sendmail to send the message? Also, what type of exiting would cause the bash script to restart it? What would I call the bash script(s) and where do they reside?

(I tried using this with 1 of the scripts and it kept loading the first script, it would disappear, then re-appear again.)
 
Last edited:

digitaltrails

Member
Joined
Dec 18, 2021
Messages
37
Reaction score
16
Credits
353
Yes, when I do the status, that's where it shows the service is disabled (for both scripts.)
You should post the actual output, otherwise I can only guess from your interpretation of it.

I suggest posting the output from the following:

Code:
# 1 service status:
systemctl --user status your.service

# 2 response from attempting to enable the service:
systemctl --user enable your.service

# 3 response from attempting to start the service:
systemctl --user start your.service

# 4 a few of diagnostics
whoami
ls -la /home/your_service_username/.config/systemd/user
ls -la /home/your_service_username/.config/systemd/user/default.target.wants

Plus the content of the service file might yield some clues, so post that as well.
 

digitaltrails

Member
Joined
Dec 18, 2021
Messages
37
Reaction score
16
Credits
353
...

For the bash script, I have 2 scripts. Should I make a bash script for each one or somehow include it in one? Any way to have that bash script use sendmail to send the message? Also, what type of exiting would cause the bash script to restart it? What would I call the bash script(s) and where do they reside?

(I tried using this with 1 of the scripts and it kept loading the first script, it would disappear, then re-appear again.)
I don't understand your last sentence (the one in brackets).

It would be easiest to write a script for each of them. You asked about having screen automatically rerun the python script, so I assumed you would use screen to run the bash script. You would need run two screens.

Using mail would depend on whether mail is properly setup for the host, if it is, then use the mail command instead of echo. Something like:
Code:
mail -s "some subject" username <<EOF
python script exited with return code $?
EOF
 
OP
L

Linx1

New Member
Joined
Jan 28, 2022
Messages
15
Reaction score
0
Credits
153
I don't understand your last sentence (the one in brackets).


I tried the script and what it was doing was starting one of the python scripts within screen, then it would disappear, then a new PID appeared. It repeated this cycle.
 

digitaltrails

Member
Joined
Dec 18, 2021
Messages
37
Reaction score
16
Credits
353
I'm still not clear on what you are attempting:

I tried the script and what it was doing was starting one of the python scripts within screen, then it would disappear, then a new PID appeared. It repeated this cycle.

I think you need to post actual code and actual error messages to the forum. Otherwise any attempt at help is just guessing in the dark and just ends in frustration.

The other approach you could take is to get a simplified test example coded and post that. Once you've been helped with the test example, you can use that as a basis for creating your real version. For example, you could take the service example I've outlined and get it to run a simple "hello world" python script (which is actually what I did when I tested the systemd user service).
 
OP
L

Linx1

New Member
Joined
Jan 28, 2022
Messages
15
Reaction score
0
Credits
153
A little context to what these scripts do: They detect WiFi on phones to determine presence for automation. The edgebridge is a server that passes on information to a home automation system and phonetrack is what passes on the information to edgebridge. Phonetrack works in tandem with a .cfg file for configuation.

# 1 service status:
systemctl --user status your.service

[email protected]:~ $ systemctl --user status phonetrack
● phonetrack.service - phonetrack
Loaded: loaded (/home/pi/.config/systemd/user/phonetrack.service; disabled; vendor preset: enabled)
Active: active (running) since Sun 2022-01-30 00:35:31 GMT; 1s ago
Main PID: 2287 (python3)
CGroup: /user.slice/user-1000.slice/[email protected]/phonetrack.service
└─2287 /usr/bin/python3 /home/pi/phonetrack_st.py
Jan 30 00:35:31 raspberrypi systemd[758]: Started phonetrack.

[email protected]:~ $ systemctl --user status edgebridge
● edgebridge.service - edgebridge
Loaded: loaded (/home/pi/.config/systemd/user/edgebridge.service; disabled; vendor preset: enabled)
Active: active (running) since Sun 2022-01-30 00:30:54 GMT; 9min ago
Main PID: 2241 (python3)
CGroup: /user.slice/user-1000.slice/[email protected]/edgebridge.service
└─2241 /usr/bin/python3 /home/pi/edgebridge.py
Jan 30 00:30:54 raspberrypi systemd[758]: Started edgebridge.

# 2 response from attempting to enable the service:
systemctl --user enable your.service

[email protected]:~ $ systemctl --user enable phonetrack
Failed to enable unit: Unit file /home/pi/.config/systemd/user/default.target.wants/phonetrack.service does not exist.

[email protected]:~ $ systemctl --user enable edgebridge
Failed to enable unit: Unit file /home/pi/.config/systemd/user/default.target.wants/edgebridge.service does not exist.


# 3 response from attempting to start the service:
systemctl --user start your.service

(no response, cursor moves to next line)

# 4 a few of diagnostics
whoami - pi

ls -la /home/your_service_username/.config/systemd/user:

[email protected]:~ $ ls -la /home/pi/.config/systemd/user
total 16
drwxr-xr-x 2 root root 4096 Jan 29 18:16 .
drwxr-xr-x 3 root root 4096 Jan 29 16:43 ..
-rw-r--r-- 1 root root 182 Jan 29 18:07 edgebridge.service
-rw-r--r-- 1 root root 184 Jan 29 17:27 phonetrack.service

ls -la /home/your_service_username/.config/systemd/user/default.target.wants:

[email protected]:~ $ ls -la /home/pi/.config/systemd/user/default.target.wants
ls: cannot access '/home/pi/.config/systemd/user/default.target.wants': No such file or directory
 
OP
L

Linx1

New Member
Joined
Jan 28, 2022
Messages
15
Reaction score
0
Credits
153
I'm still not clear on what you are attempting:



I think you need to post actual code and actual error messages to the forum. Otherwise any attempt at help is just guessing in the dark and just ends in frustration.

The other approach you could take is to get a simplified test example coded and post that. Once you've been helped with the test example, you can use that as a basis for creating your real version. For example, you could take the service example I've outlined and get it to run a simple "hello world" python script (which is actually what I did when I tested the systemd user service).
The bash script example you provided kept loading one of the python scripts over and over again.
 

digitaltrails

Member
Joined
Dec 18, 2021
Messages
37
Reaction score
16
Credits
353
Much better, there are some real clues here...
A little context to what these scripts do: They detect WiFi on phones to determine presence for automation. The edgebridge is a server that passes on information to a home automation system and phonetrack is what passes on the information to edgebridge. Phonetrack works in tandem with a .cfg file for configuation.

# 1 service status:
systemctl --user status your.service

[email protected]:~ $ systemctl --user status phonetrack
● phonetrack.service - phonetrack
Loaded: loaded (/home/pi/.config/systemd/user/phonetrack.service; disabled; vendor preset: enabled)
Active: active (running) since Sun 2022-01-30 00:35:31 GMT; 1s ago
Main PID: 2287 (python3)
CGroup: /user.slice/user-1000.slice/[email protected]/phonetrack.service
└─2287 /usr/bin/python3 /home/pi/phonetrack_st.py
Jan 30 00:35:31 raspberrypi systemd[758]: Started phonetrack.

[email protected]:~ $ systemctl --user status edgebridge
● edgebridge.service - edgebridge
Loaded: loaded (/home/pi/.config/systemd/user/edgebridge.service; disabled; vendor preset: enabled)
Active: active (running) since Sun 2022-01-30 00:30:54 GMT; 9min ago
Main PID: 2241 (python3)
CGroup: /user.slice/user-1000.slice/[email protected]/edgebridge.service
└─2241 /usr/bin/python3 /home/pi/edgebridge.py
Jan 30 00:30:54 raspberrypi systemd[758]: Started edgebridge.
From the above it seems the services are actually setup and running.
# 2 response from attempting to enable the service:
systemctl --user enable your.service

[email protected]:~ $ systemctl --user enable phonetrack
Failed to enable unit: Unit file /home/pi/.config/systemd/user/default.target.wants/phonetrack.service does not exist.

[email protected]:~ $ systemctl --user enable edgebridge
Failed to enable unit: Unit file /home/pi/.config/systemd/user/default.target.wants/edgebridge.service does not exist.


# 3 response from attempting to start the service:
systemctl --user start your.service

(no response, cursor moves to next line)

# 4 a few of diagnostics
whoami - pi

ls -la /home/your_service_username/.config/systemd/user:

[email protected]:~ $ ls -la /home/pi/.config/systemd/user
total 16
drwxr-xr-x 2 root root 4096 Jan 29 18:16 .
drwxr-xr-x 3 root root 4096 Jan 29 16:43 ..
-rw-r--r-- 1 root root 182 Jan 29 18:07 edgebridge.service
-rw-r--r-- 1 root root 184 Jan 29 17:27 phonetrack.service
From the above it appears some of the files and folders are owned by root, which is wrong. Running enable commands as the normal user will fail if the user lacks write permission to directories under /home/pi/.config/systemd. So I think you need to change the ownership of the pi user's .config/systemd hierarchy so that they belong to the pi user. A recursive chown would likely fix things up, it would have to be done as root to get around the current permission issue.
Code:
chown -R pi.user /home/pi/.config/systemd
Once done doing the enable as the pi user might then work.

ls -la /home/your_service_username/.config/systemd/user/default.target.wants:

[email protected]:~ $ ls -la /home/pi/.config/systemd/user/default.target.wants
ls: cannot access '/home/pi/.config/systemd/user/default.target.wants': No such file or directory
Again, it appears the the pi user did not have permission to create /home/pi/.config/systemd/user/default.target.wants, so we get the no such file or directory error.

Thanks for the detailed info, it's like the fog has lifted.
 

digitaltrails

Member
Joined
Dec 18, 2021
Messages
37
Reaction score
16
Credits
353
The bash script example you provided kept loading one of the python scripts over and over again.
Yes, I thought that was the objective: run the bash script as the screen command, and have the bash script restart the python if it exits.
Perhaps you mean the python script was exiting immediately, and then being started over and over, in which perhaps the bash code for starting the python has an error or is not providing something in the surrounding environment that the python code needs. It would be hard to guess what that is. Getting a simple python example to work first would be the way to move forward, something like:
Code:
# Simple python code:
import time
print("hello world")
time.sleep(30)
But from the name of you scripts, a service does seem an attractive way to go.
 
OP
L

Linx1

New Member
Joined
Jan 28, 2022
Messages
15
Reaction score
0
Credits
153
Much better, there are some real clues here...

From the above it seems the services are actually setup and running.

From the above it appears some of the files and folders are owned by root, which is wrong. Running enable commands as the normal user will fail if the user lacks write permission to directories under /home/pi/.config/systemd. So I think you need to change the ownership of the pi user's .config/systemd hierarchy so that they belong to the pi user. A recursive chown would likely fix things up, it would have to be done as root to get around the current permission issue.
Code:
chown -R pi.user /home/pi/.config/systemd
Once done doing the enable as the pi user might then work.


Again, it appears the the pi user did not have permission to create /home/pi/.config/systemd/user/default.target.wants, so we get the no such file or directory error.

Thanks for the detailed info, it's like the fog has lifted.
No problem at all, you're helping me, so I'm glad this shed some light. As usual, user error :) I tried the command "chown -R pi.user /home/pi/.config/systemd" and this is what I got:

[email protected]:/# chown -R pi.user /home/pi/.config/systemd
chown: invalid user: ‘pi.user’
 
OP
L

Linx1

New Member
Joined
Jan 28, 2022
Messages
15
Reaction score
0
Credits
153
Yes, I thought that was the objective: run the bash script as the screen command, and have the bash script restart the python if it exits.
Perhaps you mean the python script was exiting immediately, and then being started over and over, in which perhaps the bash code for starting the python has an error or is not providing something in the surrounding environment that the python code needs. It would be hard to guess what that is. Getting a simple python example to work first would be the way to move forward, something like:
Code:
# Simple python code:
import time
print("hello world")
time.sleep(30)
But from the name of you scripts, a service does seem an attractive way to go.

Yes, this is exactly what was happening :)
 
$100 Digital Ocean Credit
Get a free VM to test out Linux!


Top