Solved: vncserver service failed a configured resource limit was exceeded

In my last article I shared the step by step guide to tigervnc server on RHEL 7 Linux. Now let me help you troubleshoot one of the most common issue with tigervnc service. I got this error message vncserver service failed because a configured resource limit was exceeded while restarting the tigervnc service on my Red Hat Linux 7 setup.

As from the error message it seems like my system resource has drenched out required by vncserver which is why vncserver service failed to start. So the first thing one would do is try to look into ulimit. Hence the message (vncserver service failed because a configured resource limit was exceeded)looks a little misleading when the actual problem is something else.

How to fix "vncserver service failed because a configured resource limit was exceeded" ?

We get this error message (vncserver service failed because a configured resource limit was exceeded) while we attempt to start the vncserver service.

# systemctl start vncserver@:1.service
Job for vncserver@:1.service failed because a configured resource limit was exceeded. See "systemctl status vncserver@:1.service" and "journalctl -xe" for details.

Also if we check the status of the service, we observe PID file /root/.vnc/openstack.example:1.pid not readable (yet?) after start message which again was confusing to me as the permission on the PID file looked correct and it cannot change suddenly when I know the vnc service was working just fine.

[root@openstack tmp]# systemctl status vncserver@:1.service
● vncserver@:1.service - Remote desktop service (VNC)
Loaded: loaded (/etc/systemd/system/vncserver@:1.service; enabled; vendor preset: disabled)
Active: failed (Result: resources) since Sat 2018-09-22 23:16:03 IST; 4s ago
Process: 8381 ExecStart=/usr/sbin/runuser -l root -c /usr/bin/vncserver %i (code=exited, status=0/SUCCESS)
Process: 8376 ExecStartPre=/bin/sh -c /usr/bin/vncserver -kill %i > /dev/null 2>&1 || : (code=exited, status=0/SUCCESS)

Sep 22 23:16:00 openstack.example systemd[1]: Starting Remote desktop service (VNC)...
Sep 22 23:16:03 openstack.example systemd[1]: PID file /root/.vnc/openstack.example:1.pid not readable (yet?) after start.
Sep 22 23:16:03 openstack.example systemd[1]: Failed to start Remote desktop service (VNC).
Sep 22 23:16:03 openstack.example systemd[1]: Unit vncserver@:1.service entered failed state.
Sep 22 23:16:03 openstack.example systemd[1]: vncserver@:1.service failed.

After browsing a bit I got to know there was a bug raised with Red Hat for a similar problem Bug 1410164 - tigervnc-server fails to remove /tmp files if not gracefully shut down where tigervnc fails to clear the files which it creates under /tmp and /tmp/.X11-unix/ if the tigervnc service is not stopped gracefully. Systemd cannot restart the service until those files are removed manually with the error "Job for vncserver@:2.service failed because a configured resource limit was exceeded. See "systemctl status vncserver@:2.service" and "journalctl -xe" for details."

Below list of files may appear in your /tmp

/tmp/.X11-unix:
total 8
drwxrwxrwt. 2 root root 4096 Sep 22 23:09 .
drwxrwxrwt. 18 root root 4096 Sep 22 23:11 ..
srwxrwxrwx 1 root root 0 Sep 22 21:41 X0
srwxrwxrwx 1 root root 0 Sep 22 18:59 X1
srwxrwxrwx 1 root root 0 Sep 22 23:08 X10
srwxrwxrwx 1 root root 0 Sep 22 23:08 X11
srwxrwxrwx 1 root root 0 Sep 22 23:09 X12
srwxrwxrwx 1 root root 0 Sep 22 22:13 X2
srwxrwxrwx 1 root root 0 Sep 22 21:34 X3
srwxrwxrwx 1 root root 0 Sep 22 21:35 X4
srwxrwxrwx 1 root root 0 Sep 22 21:35 X5
srwxrwxrwx 1 root root 0 Sep 22 21:36 X6
srwxrwxrwx 1 root root 0 Sep 22 21:36 X7
srwxrwxrwx 1 root root 0 Sep 22 21:36 X8
srwxrwxrwx 1 root root 0 Sep 22 21:41 X9

/tmp/.XIM-unix:
total 0
# ls -al /tmp/.*
-r--r--r-- 1 root root 11 Sep 22 23:08 /tmp/.X10-lock
-r--r--r-- 1 root root 11 Sep 22 23:08 /tmp/.X11-lock
-r--r--r-- 1 root root 11 Sep 22 23:09 /tmp/.X12-lock
-r--r--r-- 1 root root 11 Sep 22 22:13 /tmp/.X2-lock
-r--r--r-- 1 root root 11 Sep 22 21:41 /tmp/.X9-lock

# ls -al /tmp/.
./ ../ .esd-0/ .ICE-unix/ .X10-lock .X11-unix/ .X2-lock .XIM-unix/ .font-unix/ .Test-unix/ .X11-lock .X12-lock .X9-lock

To fix the issue, we must manually clear the temporary files created by tigervnc service as shown below

# rm -f /tmp/.X11-unix/X1

Next re-attempt to start the vncserver service

# systemctl start vncserver@:1.service

So looks like the solution worked.

# systemctl status vncserver@:1.service
● vncserver@:1.service - Remote desktop service (VNC)
Loaded: loaded (/etc/systemd/system/vncserver@:1.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2018-09-22 23:18:57 IST; 2min 17s ago
Process: 10897 ExecStart=/usr/sbin/runuser -l root -c /usr/bin/vncserver %i (code=exited, status=0/SUCCESS)
Process: 10892 ExecStartPre=/bin/sh -c /usr/bin/vncserver -kill %i > /dev/null 2>&1 || : (code=exited, status=0/SUCCESS)
Main PID: 10934 (Xvnc)
CGroup: /system.slice/system-vncserver.slice/vncserver@:1.service
‣ 10934 /usr/bin/Xvnc :1 -auth /root/.Xauthority -desktop openstack.example:1 (root) -fp catalogue:/etc/X11/fontpath.d -g...

Sep 22 23:18:54 openstack.example systemd[1]: Starting Remote desktop service (VNC)...
Sep 22 23:18:57 openstack.example systemd[1]: Started Remote desktop service (VNC).

Lastly I hope the steps from the article to troubleshoot vncserver service failed because a configured resource limit was exceeded was helpful. So, let me know your suggestions and feedback using the comment section.