<div dir="ltr">Hello,<div>I'm monitoring these testes with "corex.shm_summary" provided by kamcmd.</div><div><br></div><div>I'm noticing that the core module - xavp.c is consuming lot of memory and it is increasing.</div><div><br></div><div>result of command corex.shm_summary is:</div><div><br></div><div><div><div> fm_status: summarizing all alloc'ed. fragments:</div><div> fm_status: count= 2 size= 1120 bytes from tm: t_reply.c: _reply_light(542)</div><div> fm_status: count= 2 size= 896 bytes from tm: t_msgbuilder.c: build_uac_req(1543)</div><div> fm_status: count= 2182 size= 231088 bytes from tm: t_fwd.c: prepare_new_uac(479)</div><div> fm_status: count= 12488 size= 12050808 bytes from tm: t_reply.c: relay_reply(1884)</div><div> fm_status: count= 12490 size= 14564656 bytes from core: msg_translator.c: build_req_buf_from_sip_req(2149)</div><div> fm_status: count= 12492 size= 71366536 bytes from tm: h_table.c: build_cell(317)</div><div> fm_status: count= 12490 size= 75086184 bytes from core: sip_msg_clone.c: sip_msg_shm_clone(494)</div><div> fm_status: count= 2 size= 96 bytes from tm: t_hooks.c: insert_tmcb(137)</div><div> fm_status: count= 2182 size= 17800 bytes from tm: t_fwd.c: prepare_new_uac(524)</div><div> fm_status: count= 3 size= 512 bytes from htable: ht_api.c: ht_cell_new(183)</div><div> fm_status: count= 12490 size= 10586712 bytes from core: sip_msg_clone.c: msg_lump_cloner(978)</div><div> fm_status: count= 56508 size= 3457904 bytes from core: usr_avp.c: create_avp(175)</div><div> fm_status: count= 4 size= 600 bytes from tmx: tmx_pretran.c: tmx_check_pretran(250)</div><div> fm_status: count= 4 size= 1008 bytes from usrloc: ucontact.c: new_ucontact(98)</div><div> fm_status: count= 4 size= 1320 bytes from tmx: tmx_pretran.c: tmx_check_pretran(271)</div><div> fm_status: count= 4 size= 144 bytes from usrloc: urecord.c: new_urecord(65)</div><div> fm_status: count= 4 size= 328 bytes from usrloc: urecord.c: new_urecord(58)</div><div> fm_status: count= 24 size= 952 bytes from usrloc: ../../ut.h: shm_str_dup(723)</div><div> fm_status: count= 2182 size= 53040 bytes from tm: t_fwd.c: prepare_new_uac(509)</div><div> fm_status: count=12545409 size=1420184592 bytes from core: xavp.c: xavp_new_value(94)</div><div> fm_status: count= 4 size= 128 bytes from dmq: worker.c: alloc_job_queue(229)</div><div> fm_status: count= 1 size= 13440 bytes from core: counters.c: counters_prefork_init(207)</div><div> fm_status: count= 1 size= 4032 bytes from sl: sl_stats.c: init_sl_stats_child(125)</div><div> fm_status: count= 1 size= 256 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90)</div><div> fm_status: count= 1 size= 5376 bytes from tm: t_stats.c: init_tm_stats_child(60)</div><div> fm_status: count= 1 size= 1008 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79)</div><div> fm_status: count= 3 size= 88 bytes from core: cfg/cfg_struct.c: cfg_clone_str(130)</div><div> fm_status: count= 1 size= 744 bytes from core: cfg/cfg_struct.c: cfg_shmize(217)</div><div> fm_status: count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51)</div><div> fm_status: count= 1 size= 40960 bytes from usrloc: udomain.c: new_udomain(93)</div><div> fm_status: count= 1 size= 48 bytes from usrloc: udomain.c: new_udomain(86)</div><div> fm_status: count= 1 size= 16 bytes from usrloc: dlist.c: new_dlist(573)</div><div> fm_status: count= 1 size= 32 bytes from usrloc: dlist.c: new_dlist(565)</div><div> fm_status: count= 1 size= 8 bytes from core: pt.c: init_pt(110)</div><div> fm_status: count= 1 size= 8 bytes from core: pt.c: init_pt(105)</div><div> fm_status: count= 1 size= 2944 bytes from core: pt.c: init_pt(104)</div><div> fm_status: count= 1 size= 32 bytes from usrloc: ul_callback.c: register_ulcb(94)</div><div> fm_status: count= 1 size= 24 bytes from dmq: ../../ut.h: shm_str_dup(723)</div><div> fm_status: count= 1 size= 448 bytes from dmq: dmqnode.c: build_dmq_node(156)</div><div> fm_status: count= 2 size= 168 bytes from dmq: peer.c: add_peer(67)</div><div> fm_status: count= 1 size= 8 bytes from dmq: dmq.c: mod_init(243)</div><div> fm_status: count= 1 size= 96 bytes from dmq: dmq.c: mod_init(237)</div><div> fm_status: count= 1 size= 24 bytes from dmq: dmqnode.c: init_dmq_node_list(66)</div><div> fm_status: count= 1 size= 24 bytes from dmq: peer.c: init_peer_list(33)</div><div> fm_status: count= 1 size= 16 bytes from usrloc: ul_callback.c: init_ulcb_list(45)</div><div> fm_status: count= 1 size= 4112 bytes from usrloc: ../../lock_alloc.h: lock_set_alloc(70)</div><div> fm_status: count= 1 size= 8 bytes from carrierroute: cr_data.c: rule_fixup_recursor(584)</div><div> fm_status: count= 5 size= 56 bytes from carrierroute: ../../ut.h: shm_str_dup(723)</div><div> fm_status: count= 1 size= 152 bytes from carrierroute: cr_rule.c: add_route_rule(74)</div><div> fm_status: count= 3 size= 240 bytes from core: dtrie.c: dtrie_insert(157)</div><div> fm_status: count= 3 size= 48 bytes from core: dtrie.c: dtrie_insert(148)</div><div> fm_status: count= 1 size= 48 bytes from carrierroute: cr_rule.c: add_route_flags(225)</div><div> fm_status: count= 2 size= 160 bytes from core: dtrie.c: dtrie_init(60)</div><div> fm_status: count= 2 size= 32 bytes from core: dtrie.c: dtrie_init(51)</div><div> fm_status: count= 1 size= 32 bytes from carrierroute: cr_domain.c: create_domain_data(83)</div><div> fm_status: count= 1 size= 8 bytes from carrierroute: cr_carrier.c: create_carrier_data(59)</div><div> fm_status: count= 1 size= 64 bytes from carrierroute: cr_carrier.c: create_carrier_data(50)</div><div> fm_status: count= 1 size= 8 bytes from carrierroute: cr_db.c: load_route_data_db(295)</div><div> fm_status: count= 5 size= 1008 bytes from dispatcher: dispatch.c: reindex_dests(600)</div><div> fm_status: count= 1 size= 48 bytes from carrierroute: cr_db.c: load_domain_map(182)</div><div> fm_status: count= 1 size= 24 bytes from carrierroute: cr_db.c: load_domain_map(171)</div><div> fm_status: count= 1 size= 48 bytes from carrierroute: cr_db.c: load_carrier_map(126)</div><div> fm_status: count= 1 size= 24 bytes from carrierroute: cr_db.c: load_carrier_map(115)</div><div> fm_status: count= 7 size= 168 bytes from dispatcher: dispatch.c: add_dest2list(350)</div><div> fm_status: count= 1 size= 64 bytes from carrierroute: cr_data.c: reload_route_data(172)</div><div> fm_status: count= 1 size= 8 bytes from carrierroute: cr_data.c: init_route_data(74)</div><div> fm_status: count= 5 size= 4200 bytes from dispatcher: dispatch.c: add_dest2list(324)</div><div> fm_status: count= 1 size= 16 bytes from dispatcher: dispatch.c: init_data(204)</div><div> fm_status: count= 1 size= 16 bytes from dispatcher: dispatch.c: init_data(195)</div><div> fm_status: count= 1 size= 8 bytes from dispatcher: dispatcher.c: mod_init(309)</div><div> fm_status: count= 1 size= 8 bytes from dispatcher: dispatcher.c: mod_init(307)</div><div> fm_status: count= 1 size= 8 bytes from dispatcher: dispatch.c: ds_ping_active_init(102)</div><div> fm_status: count= 1 size= 256 bytes from db_text: dbt_lib.c: dbt_init_cache(81)</div><div> fm_status: count= 1 size= 8 bytes from db_text: dbt_lib.c: dbt_init_cache(69)</div><div> fm_status: count= 1 size= 8 bytes from db_text: dbt_lib.c: dbt_init_cache(54)</div><div> fm_status: count= 3 size= 288 bytes from core: timer.c: register_timer(1011)</div><div> fm_status: count= 4 size= 8388608 bytes from htable: ht_api.c: ht_init_tables(381)</div><div> fm_status: count= 1 size= 8 bytes from sl: sl_funcs.c: sl_startup(83)</div><div> fm_status: count= 1 size= 8 bytes from sl: sl_stats.c: init_sl_stats(110)</div><div> fm_status: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(74)</div><div> fm_status: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(72)</div><div> fm_status: count= 1 size= 2097152 bytes from tm: h_table.c: init_hash_table(467)</div><div> fm_status: count= 2 size= 64 bytes from core: cfg/cfg_ctx.c: cfg_register_ctx(47)</div><div> fm_status: count= 1 size= 80 bytes from db_postgres: ../../lock_alloc.h: lock_set_alloc(70)</div><div> fm_status: count= 1 size= 8192 bytes from core: tcp_main.c: init_tcp(4635)</div><div> fm_status: count= 1 size= 32768 bytes from core: tcp_main.c: init_tcp(4629)</div><div> fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4621)</div><div> fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4614)</div><div> fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4607)</div><div> fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4601)</div><div> fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4589)</div><div> fm_status: count= 1 size= 8 bytes from core: usr_avp.c: init_avps(90)</div><div> fm_status: count= 1 size= 8 bytes from core: usr_avp.c: init_avps(89)</div><div> fm_status: count= 1 size= 16384 bytes from core: dst_blacklist.c: init_dst_blacklist(437)</div><div> fm_status: count= 1 size= 8 bytes from core: dst_blacklist.c: init_dst_blacklist(430)</div><div> fm_status: count= 2 size= 96 bytes from core: timer.c: timer_alloc(514)</div><div> fm_status: count= 1 size= 8 bytes from core: dns_cache.c: init_dns_cache(366)</div><div> fm_status: count= 1 size= 16384 bytes from core: dns_cache.c: init_dns_cache(358)</div><div> fm_status: count= 1 size= 16 bytes from core: dns_cache.c: init_dns_cache(351)</div><div> fm_status: count= 1 size= 8 bytes from core: dns_cache.c: init_dns_cache(345)</div><div> fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(283)</div><div> fm_status: count= 1 size= 16384 bytes from core: timer.c: init_timer(282)</div><div> fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(281)</div><div> fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(280)</div><div> fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(269)</div><div> fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(237)</div><div> fm_status: count= 1 size= 278544 bytes from core: timer.c: init_timer(220)</div><div> fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(219)</div><div> fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(206)</div><div> fm_status: count= 1 size= 64 bytes from core: cfg/cfg_struct.c: cfg_child_cb_new(830)</div><div> fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(361)</div><div> fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(354)</div><div> fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(347)</div><div> fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(335)</div><div> fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(323)</div><div> fm_status: count= 4 size= 928 bytes from htable: ht_api.c: ht_add_table(278)</div><div> fm_status: count= 1 size= 8 bytes from core: mem/shm.c: shm_core_lock_init(153)</div><div> fm_status: -----------------------------</div></div></div><div><br></div><div><div>If i stop these tests and after few minutes, run again the command "corex.shm_summary" the entry that contains the memory consume for xavp.c doesn't decrease.</div><div><br></div><div>It seems that there is a memory leak in this module xavp.c.</div></div><div><br></div><div>Thank you for your help.</div><div>Regards</div><div>José</div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-07-02 11:45 GMT+01:00 José Seabra <span dir="ltr"><<a href="mailto:joseseabra4@gmail.com" target="_blank">joseseabra4@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'm running burning tests against kamailio and after some time ~ 2 days I see kamailio out of mem:<br><br>ERROR: tm [t_lookup.c:1271]: new_t(): ERROR: new_t: out of mem:<br>ERROR: tm [t_lookup.c:1411]: t_newtran(): ERROR: t_newtran: new_t failed<br><br>The result of command kamcmd mod.stats all is:<div><br>Module: core<br>{<br> sip_msg_shm_clone(494): 10653392<br> build_req_buf_from_sip_req(2149): 2565480<br> msg_lump_cloner(978): 2943584<br> create_avp(175): 969736<br> xavp_new_value(94): 858469320<br> counters_prefork_init(207): 16896<br> cfg_clone_str(130): 88<br> cfg_shmize(217): 744<br> init_pt(110): 8<br> init_pt(105): 8<br> init_pt(104): 3080<br> dtrie_insert(157): 240<br> dtrie_insert(148): 48<br> dtrie_init(60): 160<br> dtrie_init(51): 32<br> register_timer(1011): 360<br> cfg_register_ctx(47): 32<br> init_tcp(4635): 8192<br> init_tcp(4629): 32768<br> init_tcp(4621): 8<br> init_tcp(4614): 8<br> init_tcp(4607): 8<br> init_tcp(4601): 8<br> init_tcp(4589): 8<br> init_avps(90): 8<br> init_avps(89): 8<br> init_dst_blacklist(437): 16384<br> init_dst_blacklist(430): 8<br> timer_alloc(514): 96<br> init_dns_cache(366): 8<br> init_dns_cache(358): 16384<br> init_dns_cache(351): 16<br> init_dns_cache(345): 8<br> init_timer(283): 8<br> init_timer(282): 16384<br> init_timer(281): 8<br> init_timer(280): 8<br> init_timer(269): 8<br> init_timer(237): 8<br> init_timer(220): 278544<br> init_timer(219): 8<br> init_timer(206): 8<br> cfg_child_cb_new(830): 64<br> sr_cfg_init(361): 8<br> sr_cfg_init(354): 8<br> sr_cfg_init(347): 8<br> sr_cfg_init(335): 8<br> sr_cfg_init(323): 8<br> shm_core_lock_init(153): 8<br> Total: 875992224<br>}<br><br>Module: dmq<br>{<br> alloc_job_queue(229): 128<br> shm_str_dup(723): 24<br> build_dmq_node(156): 448<br> add_peer(67): 168<br> mod_init(243): 8<br> mod_init(237): 96<br> init_dmq_node_list(66): 24<br> init_peer_list(33): 24<br> Total: 920<br>}<br><br>Module: auth_db<br>{<br> Total: 0<br>}<br><br>Module: dialog<br>{<br> init_dlg_table(284): 131088<br> init_dlg_table(276): 8<br> init_dlg_table(269): 8<br> init_dlg_table(264): 8<br> init_dlg_timer(57): 8<br> init_dlg_timer(48): 32<br> new_dlg_profile(187): 624<br> Total: 131776<br>}<br><br>Module: rr<br>{<br> Total: 0<br>}<br><br>Module: xlog<br>{<br> Total: 0<br>}<br><br>Module: kex<br>{<br> pkg_proc_stats_init(79): 1056<br> Total: 1056<br>}<br><br>Module: db_postgres<br>{<br> lock_set_alloc(70): 80<br> Total: 80<br>}<br><br>Module: pv<br>{<br> Total: 0<br>}<br><br>Module: textops<br>{<br> Total: 0<br>}<br><br>Module: htable<br>{<br> ht_cell_new(183): 1104<br> ht_init_tables(381): 8388608<br> ht_add_table(278): 928<br> Total: 8390640<br>}<br><br>Module: sqlops<br>{<br> Total: 0<br>}<br><br>Module: tm<br>{<br> build_uac_req(1543): 848<br> prepare_new_uac(479): 28880<br> _reply_light(542): 751504<br> build_local_reparse(341): 1168568<br> relay_reply(1884): 643088<br> build_cell(317): 9826624<br> prepare_new_uac(524): 8112<br> prepare_new_uac(509): 10192<br> init_tm_stats_child(60): 5632<br> insert_tmcb(137): 136<br> init_tmcb_lists(74): 16<br> init_tmcb_lists(72): 16<br> init_hash_table(467): 2097152<br> Total: 14540768<br>}<br><br>Module: usrloc<br>{<br> new_urecord(65): 136<br> new_ucontact(98): 1096<br> new_urecord(58): 288<br> shm_str_dup(723): 1200<br> build_stat_name(51): 64<br> new_udomain(93): 40960<br> new_udomain(86): 48<br> new_dlist(573): 16<br> new_dlist(565): 32<br> register_ulcb(94): 32<br> init_ulcb_list(45): 16<br> lock_set_alloc(70): 4112<br> Total: 48000<br>}<br><br>Module: tmx<br>{<br> tmx_check_pretran(250): 864<br> tmx_check_pretran(271): 1216<br> tmx_init_pretran_table(90): 256<br> Total: 2336<br>}<br><br>Module: sl<br>{<br> init_sl_stats_child(125): 4224<br> sl_startup(83): 8<br> init_sl_stats(110): 8<br> Total: 4240<br>}<br><br>Module: carrierroute<br>{<br> rule_fixup_recursor(584): 8<br> shm_str_dup(723): 56<br> add_route_rule(74): 152<br> add_route_flags(225): 48<br> create_domain_data(83): 32<br> create_carrier_data(59): 8<br> create_carrier_data(50): 64<br> load_route_data_db(295): 8<br> load_domain_map(182): 48<br> load_domain_map(171): 24<br> load_carrier_map(126): 48<br> load_carrier_map(115): 24<br> reload_route_data(172): 64<br> init_route_data(74): 8<br> Total: 592<br>}<br><br>Module: dispatcher<br>{<br> reindex_dests(600): 1008<br> add_dest2list(350): 168<br> add_dest2list(324): 4200<br> init_data(204): 16<br> init_data(195): 16<br> mod_init(309): 8<br> mod_init(307): 8<br> ds_ping_active_init(102): 8<br> Total: 5432<br>}<br><br>Module: db_text<br>{<br> dbt_init_cache(81): 256<br> dbt_init_cache(69): 8<br> dbt_init_cache(54): 8<br> Total: 272<br>}<br><br>The result of command kamctl shmem is:</div><div><br clear="all">shmem:fragments = 6353<br>shmem:free_size = 406208<br>shmem:max_used_size = 8464105472<br>shmem:real_used_size = 8463699264<br>shmem:total_size = 8464105472<br>shmem:used_size = 5190698208<br><br>The server has 8G of shared memory allocated, and the issue happens more or less after 2 days running tests.</div><div><br></div><div>I'm using sipp to generate traffic, it is sending 500CAPS.</div><div><br></div><div>There is any possibility of memory leak on Kamailio 4.4.0?</div><div><br></div><div>Best Regards </div><span class="HOEnZb"><font color="#888888"><div><br>-- <div data-smartmail="gmail_signature"><div>José Seabra</div></div>
</div></font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Cumprimentos<div>José Seabra</div></div>
</div>