Nginx is killed #1

New issue

Open

opened 2025-02-19 07:41:34 +01:00 by tdpeuter · 2 comments

tdpeuter commented

2025-02-19 07:41:34 +01:00

Owner

Nginx sometimes stops working at night. Looking at my uptime monitor, it consistently crashes at 02:00, but on seemingly random days:

Rebooting the VM often helps, but it did not help this morning.

○ nginx.service - Nginx Web Server
     Loaded: loaded (/etc/systemd/system/nginx.service; enabled; preset: ignored)
     Active: inactive (dead)

I manually started Nginx back up (systemctl start nginx), resulting in the following logs:

Feb 19 00:22:53 Ingress nginx[1433]: 2025/02/19 00:22:53 [error] 1433#1433: *310898 upstream prematurely closed connection while reading upstream, client: 213.119.56.17, server: jelly.depeuter.dev, request: "GET /Items/4532b304992a121e2969801d1c9d304c/Images/Primary HTTP/1.1", upstream: "http://192.168.0.94:8096/Items/4532b304992a121e2969801d1c9d304c/Images/Primary", host: "jelly.depeuter.dev"
Feb 19 02:00:02 Ingress systemd[1]: Stopping Nginx Web Server...
Feb 19 02:00:02 Ingress systemd[1]: nginx.service: Deactivated successfully.
Feb 19 02:00:02 Ingress systemd[1]: Stopped Nginx Web Server.
Feb 19 02:00:02 Ingress systemd[1]: nginx.service: Consumed 4min 12.951s CPU time, 93.2M memory peak, 4.6M read from disk, 203.2M written to disk, 10.5G incoming IP traffic, 10.7G outgoing IP traffic.
-- Boot c59be96b8d3549fb82612ef6d88e363b --
Feb 19 07:27:08 Ingress systemd[1]: Starting Nginx Web Server...
Feb 19 07:27:08 Ingress nginx-pre-start[1115]: nginx: the configuration file /nix/store/s650l7a2xk2jgm8g567wfn6azb4zxz48-nginx.conf syntax is ok
Feb 19 07:27:08 Ingress nginx-pre-start[1115]: nginx: configuration file /nix/store/s650l7a2xk2jgm8g567wfn6azb4zxz48-nginx.conf test is successful
Feb 19 07:27:08 Ingress systemd[1]: Started Nginx Web Server.

Nginx sometimes stops working at night. Looking at my uptime monitor, it consistently crashes at 02:00, but on seemingly random days: ![uptime monitor stats](/attachments/a1728f02-e2fc-4aa1-b74a-3d595ef34a91) Rebooting the VM often helps, but it did not help this morning. ``` ○ nginx.service - Nginx Web Server Loaded: loaded (/etc/systemd/system/nginx.service; enabled; preset: ignored) Active: inactive (dead) ``` I manually started Nginx back up (`systemctl start nginx`), resulting in the following logs: ``` Feb 19 00:22:53 Ingress nginx[1433]: 2025/02/19 00:22:53 [error] 1433#1433: *310898 upstream prematurely closed connection while reading upstream, client: 213.119.56.17, server: jelly.depeuter.dev, request: "GET /Items/4532b304992a121e2969801d1c9d304c/Images/Primary HTTP/1.1", upstream: "http://192.168.0.94:8096/Items/4532b304992a121e2969801d1c9d304c/Images/Primary", host: "jelly.depeuter.dev" Feb 19 02:00:02 Ingress systemd[1]: Stopping Nginx Web Server... Feb 19 02:00:02 Ingress systemd[1]: nginx.service: Deactivated successfully. Feb 19 02:00:02 Ingress systemd[1]: Stopped Nginx Web Server. Feb 19 02:00:02 Ingress systemd[1]: nginx.service: Consumed 4min 12.951s CPU time, 93.2M memory peak, 4.6M read from disk, 203.2M written to disk, 10.5G incoming IP traffic, 10.7G outgoing IP traffic. -- Boot c59be96b8d3549fb82612ef6d88e363b -- Feb 19 07:27:08 Ingress systemd[1]: Starting Nginx Web Server... Feb 19 07:27:08 Ingress nginx-pre-start[1115]: nginx: the configuration file /nix/store/s650l7a2xk2jgm8g567wfn6azb4zxz48-nginx.conf syntax is ok Feb 19 07:27:08 Ingress nginx-pre-start[1115]: nginx: configuration file /nix/store/s650l7a2xk2jgm8g567wfn6azb4zxz48-nginx.conf test is successful Feb 19 07:27:08 Ingress systemd[1]: Started Nginx Web Server. ```

image.png

29 KiB

tdpeuter added the

bug

label 2025-02-19 07:41:34 +01:00

tdpeuter commented

2025-02-19 07:42:34 +01:00

Author

Owner

Possibly related section on the NixOS wiki: SIGTERM received from 1

Possibly related section on the NixOS wiki: [SIGTERM received from 1](https://nixos.wiki/wiki/Nginx#SIGTERM_received_from_1)

tdpeuter commented

2025-03-06 09:40:52 +01:00

Author

Owner

Upon further investigation, it seems that daemon off; is already in the configuration file, so that is unlikely to be the problem.

Incidentally, 02:00:00 is also the time that a backup is taken of the VM in which Nginx is running. Maybe the problem is related to the backup process and/or reboot and Nginx is unable to start again.

Upon further investigation, it seems that `daemon off;` is already in the configuration file, so that is unlikely to be the problem. Incidentally, 02:00:00 is also the time that a backup is taken of the VM in which Nginx is running. Maybe the problem is related to the backup process and/or reboot and Nginx is unable to start again.

tdpeuter self-assigned this 2025-03-07 08:37:52 +01:00