Unable to start Control-M/EM Postgres database after Server reboot

Jul 23, 2020

Share this post
issues-after-qlikview-version-upgrade-on-passive-node-of-production-servers

Error Details:

CTMSERVER1% ./start_all

sh-1100 – sh-1100 module : 1100 – connection test to database server failed on timeout, check /local/mnt/apps/emuser_p/ctm_em/pgsql/data/pg_log/pgstart.log and /local/mnt/apps/emuser_p/ctm_em/bin/DBUData/log/dbu_start_err_29_06_20___21_56_32 for details

Enter the CONTROL-M/EM DBO password:

Unauthorized user. Login failed

CTMSERVER1% more /local/mnt/apps/emuser_p/ctm_em/bin/DBUData/log/dbu_start_err_29_06_20___21_56_32

psql: could not connect to server: Connection refused

Is the server running on host “CTMSERVER1” (172.23.68.108) and accepting

TCP/IP connections on port 5433?

CTMSERVER1% more /local/mnt/apps/emuser_p/ctm_em/pgsql/data/pg_log/pgstart.log

server starting

2020-06-29 21:56:33.027 PDTLOG:  could not bind IPv6 socket: Address already in use

2020-06-29 21:56:33.027 PDTHINT:  Is another postmaster already running on port 5433? If not, wait a few seconds and retry.

2020-06-29 21:56:33.028 PDTFATAL:  could not create shared memory segment: Invalid argument

2020-06-29 21:56:33.028 PDTDETAIL:  Failed system call was shmget(key=5433001, size=1109082112, 03600).

2020-06-29 21:56:33.028 PDTHINT:  This error usually means that PostgreSQL’s request for a shared memory segment exceeded your kernel’s SHMMAX parameter.  You can either reduce the request size or reconfigure the kernel with larger SHMMAX.  To reduce the request size (currently 1109082112 bytes), reduce PostgreSQL’s shared memory usage, perhaps by reducing shared_buffers or max_connections.

If the request size is already small, it’s possible that it is less than your kernel’s SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.

The PostgreSQL documentation contains more information about shared memory configuration.

 

Resolution Steps:

  1. Check for the shared memory current value as below.

[user1@CTMSERVER1 ~]$ cat /proc/sys/kernel/shmmax

536870912

  1. Logging in as root and perform the steps as below.

[root@CTMSERVER1 ~]# cat /proc/sys/kernel/shmmax

536870912

[root@CTMSERVER1 ~]# echo 2147483648 > /proc/sys/kernel/shmmax

  1. Verify the changes as below.

[root@CTMSERVER1 ~]# cat /proc/sys/kernel/shmmax

2147483648

  1. Restart the Control-M EM as below:

cd /local/mnt/apps/emuser_p/bin

CTMSERVER1% ./start_all

PostgreSQL server started

Enter the CONTROL-M/EM DBO password:

Checking port availability Communicating Naming Service: Running on CTMSERVER1:13075 Starting Naming Service… Naming Service was successfully started.

CONTROL-M Configuration Server started successfully!

CONTROL-M/EM Configuration Agent started successfully!

  1. Start the Control-M Server as below:

CTMSERVER1% cd /local/mnt/apps/controlm_p/ctm_server/scripts

CTMSERVER1% ./start_ca

Started CONTROL-M/Server Configuration Agent on Mon Jun 29 22:32:43 PDT 2020

CTMSERVER1% startdb

PostgreSQL server started

CTMSERVER1% start_ctm

Starting CONTROL-M/Server at Mon Jun 29 22:33:01 PDT 2020

CONTROL-M/Server Pause mode is N

  1. Start the Control-M Agent as below using “root” user

[root@CTMSERVER1 ~]# cd /local/mnt/apps/ctmagent_p/ctm/scripts

[root@CTMSERVER1 scripts]# ./start-ag

Enter Control-M/Agent UNIX username [ctmagent_p]:

Enter Control-M/Agent Process Name <AG|AT|AR|ALL> [ALL]:

The agent was previously running as ‘ctmagent_p’.

You have chosen to start the agent as ‘root’.

After the agent starts, you should execute the ‘/local/mnt/apps/ctmagent_p/ctm/scripts/set_agent_mode’ script

and disable non-root mode.

Do you want to continue ? [Y/N]: Y

Starting the agent as ‘root’ user

Control-M/Agent Listener started. pid: 13616

Control-M/Agent Tracker started. pid: 13680

Control-M/Agent started successfully.