Welcome to Telelogic Product Support
  Home Downloads Knowledgebase Case Tracking Licensing Help Telelogic Passport
Telelogic SYNERGY (steve huntington)
Decrease font size
Increase font size
Topic Title: informix failed with shared memory validation
Topic Summary:
Created On: 11-Jun-2004 16:39
Status: Read Only
Linear : Threading : Single : Branch
Search Topic Search Topic
Topic Tools Topic Tools
Subscribe to this topic Subscribe to this topic
E-mail this topic to someone. E-mail this topic
Bookmark this topic Bookmark this topic
View similar topics View similar topics
View topic in raw text format. Print this topic.
Answer This question was answered by Lutz Zollikofer, on Tuesday, August 3, 2004 7:56 AM

Answer:
Hello!

Thanks a lot to all for their help.

After a lot of testing we've found the reason why. Because Informix hits the max threat limit, it wasn't able to get mor resources. After increasing the NETTYPE parameter of the ccm server instance configuration ($CCM_HOME/informix/etc/<server name> all works fine. The value of this parameter is very importand for the number of parallel open connections.

old: NETTYPE ipcshm,3,20,CPU
new: NETTYPE ipcshm,3,50,CPU

I guess, the error message of too less shared memory is very mistakable.

Regards, Lutz
 11-Jun-2004 16:39
User is offline View Users Profile Print this message


Lutz Zollikofer

Posts: 44
Joined: 31-Jul-2003

Hello,

about more than 2 months we are in serious trouble with our CM/Synergy. On heavy workload (batch and online) the operations failed with different errors, which all results out of informix problems fetching shared memory.

Example online and batch:
Function: dbu_open
INFORMIX: -25588, ISAM: 22

Example batch ccmdb backup command:
Checking Table Space data ...

/ccm62/bin/dbutil/ccmdb_check_inf[132]: 23339 Memory fault
/ccm62/bin/dbutil/ccmdb_check_inf[132]: 23803 Memory fault
*
Corruption found in Informix database /ccmdb62/gha2/db.

So we are not able to complete our backups or doing work intensive jobs.
This trouble only occurred at a great number of parallel open sessions from cli or gui.
We have about 44 databases running on only 1 informix instance with about 170 users at all. Nobody can tell us, if this is too much or how else we should design it. Since 2 months or more the support tries to figure out, what is going on. But all modifications of shmem kernel parms or informix parms does not help.
We are using a HP-UX server running release 11.00 with patch level April 2004 and CM/Synergy 6.2 SP1.

Does anybody have some experiences on that or have had trouble in the same way?

Thanks and Regards,
Lutz
Report this to a Moderator Report this to a Moderator
 14-Jun-2004 13:51
User is offline View Users Profile Print this message


Dominik Jergus

Posts: 3
Joined: 30-Sep-2002


Dear Mr. Zollikofer,

I wonder what kind of server you are using (memory, cpus, network, etc.),
but from the few things that I heard about your server, it seems to be too
small to host Synergy with 170 users and 4 additional Oracle server instances
with additional 44 users on the same machine !!!

I heard that one of my colleagues is going to look into it at your site ...

Best regards,
Dominik Jergus

-------------------------
Dominik Jergus
Senior Presales Consultant
Telelogic Germany
Report this to a Moderator Report this to a Moderator
 14-Jun-2004 14:19
User is offline View Users Profile Print this message


Lutz Zollikofer

Posts: 44
Joined: 31-Jul-2003

Dear Mr. Jergus,

no one of your colleagues could told me, how to size a server (see ticket# 5005300). Therefor we should use the barely information of the manuals. If you calculate the server size, using the manual algorithm ;-), our server is big enough!
And why does it told me, that it is boring with 98% cpu and 90% mem idle, when informix fails fetching shared memory at the same time. By the way, at night, when we get batch aborts, oracle is sleeping. AND secondly, as you can see out of the mail copies I had send to you, we have also tested it with all oracle instances shutting down (like your colleague requested) and it also failed !

Best Regards,
Lutz


Report this to a Moderator Report this to a Moderator
 14-Jun-2004 16:39
User is offline View Users Profile Print this message


Vincent Gaudeul

Posts: 68
Joined: 10-Oct-2002

Hello all,

it is actually difficult to answer this kind of questions without all INFORMIX configuration items.

Could you please paste the result of this command (run as informix on the INFORMIX server) : ccmsrv status -a

Your problem may be caused by a too low number of Logical Logs for example.

Regards
Report this to a Moderator Report this to a Moderator
 15-Jun-2004 09:57
User is offline View Users Profile Print this message


Lutz Zollikofer

Posts: 44
Joined: 31-Jul-2003

Hello Vincent,

thank you very much for your answer. I append the ccmsrv-a output at the end of my reply.

As you can see, the chunks are fresh cleaned up. Onstat-a and ipcs-a tells me, that all is fine. But what makes me wondering is, the mashine is boring and informix can't fetch shared mem !? Other applications can put a headlock on this mashine, so it can't be that it blocks the resources.

Thanks and Regards,
Lutz


Informix 6.2>ccmsrv status -a -s ccm62

*** database server ccm62 ***

server number: 2
state: On-Line
maximum users allowed: 60
number of users: 10
archive device: /dev/null
network service: ccm62


--- logical logs ---

log size: 1024 Kbytes
maximum logs allowed: 23
total number of logs: 17
number of free logs: 16


--- dbspace rootdbs ---

total free space: 133862 Kbytes
chunk files:
/informixdb62/ccm62_root.dbs 72288 Kbytes total, 61588 Kbytes free
/informixdb62/ccm62_rootdbs_2.dbs 72288 Kbytes total, 72274 Kbytes free


--- dbspace temp ---

total free space: 567948 Kbytes
chunk files:
/informixdb62/ccm62_temp.dbs 72288 Kbytes total, 67954 Kbytes free
/informixdb62/ccm62_temp_2.dbs 500000 Kbytes total, 499994 Kbytes free


--- dbspace log ---

total free space: 29584 Kbytes
chunk files:
/informixdb62/ccm62_log.dbs 23552 Kbytes total, 6038 Kbytes free
/informixdb62/ccm62_log_2.dbs 23552 Kbytes total, 23546 Kbytes free


--- dbspace ccm ---

total free space: 3.93443e+06 Kbytes
chunk files:
/informixdb62/ccm62_ccm.dbs 1.9e+06 Kbytes total, 40720 Kbytes free
/informixdb62/ccm62_ccm_2.dbs 2e+06 Kbytes total, 1.89371e+06 Kbytes free
/informixdb62/ccm62_ccm_3.dbs 2e+06 Kbytes total, 1.99999e+06 Kbytes free


Database List:

Informix Dbname Version Database Path
==================== ========== =======================
aaidsl@ccm62 6.2 /ccmdb62/aaidsl/db
ad_pilot1@ccm62 6.2 /ccmdb62/ad_pilot1/db
ad_pilot_zum_wegschmeissen@ccm62 6.2 /ccmdb62/ad_pilot_zum_wegschmeissen/db
ad_stest@ccm62 6.2 /ccmdb62/ad_stest/db
ad_z@ccm62 6.2 /ccmdb62/ad_z/db
annett@ccm62 6.2 /ccmdb62/annett/db
anwendungen@ccm62 6.2 /ccmdb62/anwendungen/db
autoboerse_test@ccm62 6.2 /ccmdb62/autoboerse_test/db
base62@ccm62 6.2 /ccmdb62/base62/db
da@ccm62 6.2 /ccmdb62/da/db
dwh_cf10@ccm62 6.2 /ccmdb62/dwh_cf10/db
extranetde6@ccm62 6.2 /ccmdb62/extranetde6/db
extranetde@ccm62 6.2 /ccmdb62/extranetde/db
gha2@ccm62 6.2 /ccmdb62/gha2/db
internet@ccm62 6.2 /ccmdb62/internet/db
itms@ccm62 6.2 /ccmdb62/itms/db
ivms@ccm62 6.2 /ccmdb62/ivms/db
kredit@ccm62 6.2 /ccmdb62/kredit/db
leben@ccm62 6.2 /ccmdb62/leben/db
marco@ccm62 6.2 /ccmdb62/marco/db
model62@ccm62 6.2 /ccmdb62/model62/db
muv@ccm62 6.2 /ccmdb62/muv/db
oliver@ccm62 6.2 /ccmdb62/oliver/db
ottersbach@ccm62 6.2 /ccmdb62/ottersbach/db
pcv@ccm62 6.2 /ccmdb62/pcv/db
pcv_test@ccm62 6.2 /ccmdb62/pcv_test/db
phd@ccm62 6.2 /ccmdb62/phd/db
pms1@ccm62 6.2 /ccmdb62/pms1/db
pms@ccm62 6.2 /ccmdb62/pms/db
pos@ccm62 6.2 /ccmdb62/pos/db
ptest@ccm62 6.2 /ccmdb62/ptest/db
schulung@ccm62 6.2 /ccmdb62/schulung/db
scm@ccm62 6.2 /ccmdb62/scm/db
tutorial@ccm62 6.2 /ccmdb62/tutorial/db
vms1@ccm62 6.2 /ccmdb62/vms1/db
vmstest@ccm62 6.2 /ccmdb62/vmstest/db
vsl_itt@ccm62 6.2 /ccmdb62/vsl_itt/db
vsl_pilot@ccm62 6.2 /ccmdb62/vsl_pilot/db
vsl_wolfgang@ccm62 6.2 /ccmdb62/vsl_wolfgang/db
wolfgang@ccm62 6.2 /ccmdb62/wolfgang/db
zentral6@ccm62 6.2 /ccmdb62/zentral6/db
zentral@ccm62 6.2 /ccmdb62/zentral/db
zid@ccm62 6.2 /ccmdb62/zid/db
zolli@ccm62 6.2 /ccmdb62/zolli/db

There is a total of 44 databases.

Informix 6.2>
Report this to a Moderator Report this to a Moderator
 1-Jul-2004 07:36
User is offline View Users Profile Print this message


Mert Vuraldi

Posts: 22
Joined: 2-Feb-2004

Hi !

Perhaps You could try to check and increase OS stack size.

To get info...
ulimit -a

To increase
ulimit -s 12000 (?)

Sometimes stack size is a limiting thing, even if there is enough
processing power or general memory available.

Just a try....

Regards,

Mert
Report this to a Moderator Report this to a Moderator
 6-Jul-2004 08:25
User is offline View Users Profile Print this message


Lutz Zollikofer

Posts: 44
Joined: 31-Jul-2003

Hi Mert,

thank you very much for your answer.

As you can see, the stack's soft limit is already set to the hard limit:

OPER CCM 6.2>ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 466944
stack(kbytes) 15360
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 1024
OPER CCM 6.2>ulimit -Ha
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 466944
stack(kbytes) 15360
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) 1024

I guess, this isn't the reason. The resource limits are applied to each single process. A single process doesn't have trouble getting shared memory. The problem occurs forking a number of parallel processes, not threads.

Thanks and Regards,
Lutz
Report this to a Moderator Report this to a Moderator
 3-Aug-2004 07:56
User is offline View Users Profile Print this message


Lutz Zollikofer

Posts: 44
Joined: 31-Jul-2003

Answer Answer
Hello!

Thanks a lot to all for their help.

After a lot of testing we've found the reason why. Because Informix hits the max threat limit, it wasn't able to get mor resources. After increasing the NETTYPE parameter of the ccm server instance configuration ($CCM_HOME/informix/etc/<server name> all works fine. The value of this parameter is very importand for the number of parallel open connections.

old: NETTYPE ipcshm,3,20,CPU
new: NETTYPE ipcshm,3,50,CPU

I guess, the error message of too less shared memory is very mistakable.

Regards, Lutz
Report this to a Moderator Report this to a Moderator
Statistics
20925 users are registered to the Telelogic SYNERGY forum.
There are currently 1 users logged in.
The most users ever online was 15 on 15-Jan-2009 at 15:34.
There are currently 0 guests browsing this forum, which makes a total of 1 users using this forum.
You have posted 0 messages to this forum. 0 overall.

FuseTalk Standard Edition v3.2 - © 1999-2009 FuseTalk Inc. All rights reserved.