CDR import stopped

[stephen]

Hi

I have TNG v1.0.7.0 running at a client and CDR importing seemed to have stopped. The call list view shows the last call on the 8th of Nov 2012.
If I look at /var/log/asterisk/cdr-custom/Master.csv I can see it is still logging fine since all the way.

I tried forcing TNG to load the cdr data, by extracting all the calls for December into a seperate csv file and pointing the CSV file location to it, nothing happened. I then pointed it back to the Master.csv and it is still stuck on the 8th.
I tried doing the "Synchronize data" too, but no luck.

Where can I start looking to get it working again?

[Belinda Frick]

Please upgrade to the version of TNG for Asterisk in forum post:
TNG for Asterisk version 1.0.7.7 upgrade

NOTE: your subscription needs to be up to date to run the upgrade(expire after 17 August 2011).

For release notes of version 1.0.7.6, please see forum post: Release notes: version 1.0.6.7 to 1.0.7.6
NOTE: there were some changes made between your version(1.0.7.0) and this release(1.0.7.6).

If you get the same error again,

1. please post the tng log entries at the time it stopped logging call details.
2. I will also need your expiry date (on the TNG License Manager page).

[stephen]

Hi Belinda

I upgraded it, but still no change. Below is the log for the last day I have call records in TNG:

Code: Select all

2012-11-08 01:00:00:001 [1.0.7.0] - [AMI LOG] - CDR File "/var/log/asterisk/cdr-custom/Master.csv" moved to processed.
2012-11-08 01:00:20:101 [1.0.7.0] - [AMI LOG] - CDR File "/var/log/asterisk/cdr-custom/Master.csv" added to list.
2012-11-08 10:00:39:314 [1.0.7.0] - [AMI LOG] - Requesting Dialplan...[Process Reload]
2012-11-08 10:00:40:923 [1.0.7.0] - [AMI LOG] - Dialplan Processed.
2012-11-08 10:01:13:985 [1.0.7.0] - [AMI LOG] - Waiting on Dial Plan and Peer List
2012-11-08 10:01:55:086 [1.0.7.0] - [AMI LOG] - Peerlist Processed.
2012-11-08 11:52:49:097 [1.0.7.0] - [AMI LOG] - Requesting Dialplan...[Process Reload]
2012-11-08 11:52:50:617 [1.0.7.0] - [AMI LOG] - Dialplan Processed.
2012-11-08 11:53:07:582 [1.0.7.0] - [AMI LOG] - Waiting on Dial Plan and Peer List
2012-11-08 11:53:46:315 [1.0.7.0] - [AMI LOG] - Peerlist Processed.
2012-11-08 11:55:08:134 [1.0.7.0] - [AMI LOG] - Requesting Dialplan...[Process Reload]
2012-11-08 11:55:09:813 [1.0.7.0] - [AMI LOG] - Dialplan Processed.
2012-11-08 11:55:32:675 [1.0.7.0] - [AMI LOG] - Waiting on Dial Plan and Peer List
2012-11-08 11:56:03:246 [1.0.7.0] - [AMI LOG] - Peerlist Processed.

Below the log for today when I upgraded:

Code: Select all

2013-01-08 07:55:03:083 [1.0.7.0] - admin disconnected after timeout
2013-01-08 07:57:08:007 [1.0.7.0] - User admin connected from 192.168.250.170
2013-01-08 07:57:23:388 [1.0.7.0] - User admin disconnected
2013-01-08 07:58:06:356 [1.0.7.0] - server close:EIdTerminateThreadTimeout:Terminate Thread Timeout
2013-01-08 07:58:06:356 [1.0.7.0] - server sessionstate exception:EIdHTTPCannotSwitchSessionStateWhenActive:Cannot change session state when the server is active.
2013-01-08 07:58:06:356 [1.0.7.0] - Import Thread : Shutting Down..
2013-01-08 07:58:06:624 [1.0.7.0] - AMI Disconnected:0 -
2013-01-08 07:58:06:665 [1.0.7.0] - Import Thread: Shutting Down.. Completed
2013-01-08 07:58:06:665 [1.0.7.0] - Reports Schedule Thread : Shutting Down..
2013-01-08 07:58:09:753 [1.0.7.0] - Reports Schedule Thread: Shutting Down.. Completed
2013-01-08 07:58:09:753 [1.0.7.0] - SerialThread: Shutting Down..
2013-01-08 07:58:10:104 [1.0.7.0] - SerialThread: Shutting Down.. Completed!
2013-01-08 07:58:10:304 [1.0.7.0] - PluginThread: Shutting Down..
2013-01-08 07:59:32:393 [1.0.7.7] - Memory Status Object Created
2013-01-08 07:59:32:394 [1.0.7.7] - Available Memory: NPR = 464 ATT = 1031669760 TOT = 3978612736 MCC = 2223426
2013-01-08 07:59:33:786 [1.0.7.7] - Last File Saved: 1data_1.0.7.0
2013-01-08 07:59:33:786 [1.0.7.7] - Last File Saved Checksum: Check:[1A1BD811001FA62204EEBFC302D9C2BD]
2013-01-08 07:59:33:787 [1.0.7.7] - Loading: /opt/datatex/tng/data/1data_1.0.7.0.dat
2013-01-08 07:59:33:787 [1.0.7.7] - Loading: Assigned File -- OK
2013-01-08 07:59:33:787 [1.0.7.7] - Loading: Opened File -- OK
2013-01-08 07:59:40:038 [1.0.7.7] - Loading: 203941 Record(s) Loaded -- OK -- Check:[1A1BD811001FA62204EEBFC302D9C2BD]
2013-01-08 07:59:40:038 [1.0.7.7] - Loaddata completed: TNGCalldata Count = 203941
2013-01-08 07:59:40:038 [1.0.7.7] - Reserving Unused Memory
2013-01-08 07:59:40:349 [1.0.7.7] - Reading Settings
2013-01-08 07:59:40:349 [1.0.7.7] - Settings Read
2013-01-08 07:59:40:350 [1.0.7.7] - SerialThread : 4135615344
2013-01-08 07:59:40:350 [1.0.7.7] - Integration: Server Created
2013-01-08 07:59:40:350 [1.0.7.7] - Integration Object Created
2013-01-08 07:59:40:350 [1.0.7.7] - Barring: Server Created
2013-01-08 07:59:40:350 [1.0.7.7] - Barring Object Created
2013-01-08 07:59:40:350 [1.0.7.7] - CAM: Created: Trunk Alarm: 0 minutes. Total Alarm: 0 minutes.
2013-01-08 07:59:40:351 [1.0.7.7] - Plugin:  Reading Settings.
2013-01-08 07:59:40:351 [1.0.7.7] - Raw Data Logging disabled
2013-01-08 07:59:40:352 [1.0.7.7] - Plugin: Loading Rules..
2013-01-08 07:59:40:352 [1.0.7.7] - Plugin: Loading Rules.. Completed.
2013-01-08 07:59:40:352 [1.0.7.7] - Loading Scheduled Reports..
2013-01-08 07:59:40:352 [1.0.7.7] - Loading Scheduled Reports.. Completed.
2013-01-08 07:59:40:353 [1.0.7.7] - User tngreportschedule connected from 127.0.0.1
2013-01-08 07:59:40:353 [1.0.7.7] - Setting AMI Host :127.0.0.1
2013-01-08 07:59:40:353 [1.0.7.7] - Setting AMI Port :5038
2013-01-08 07:59:40:353 [1.0.7.7] - Setting AMI Login Name :tng
2013-01-08 07:59:40:353 [1.0.7.7] - Setting AMI Interface Option to :CDR File
2013-01-08 07:59:40:353 [1.0.7.7] - [AMI LOG] - Clearing File List.
2013-01-08 07:59:40:353 [1.0.7.7] - Setting AMI CDR FileName to:/var/log/asterisk/cdr-custom/Master.csv
2013-01-08 07:59:40:353 [1.0.7.7] - AMI Strip Breakout Digit set to:True
2013-01-08 07:59:40:354 [1.0.7.7] - Startup for ***[/opt/datatex/tng/tng --Asterisk-- ]*** OK
2013-01-08 07:59:40:354 [1.0.7.7] - Comms thread ID                  4135615344
2013-01-08 07:59:40:354 [1.0.7.7] - Integration thread ID            4127222640
2013-01-08 07:59:40:354 [1.0.7.7] - Barring object thread ID         4110437232
2013-01-08 07:59:40:354 [1.0.7.7] - Call Activity Monitor thread ID  4093651824
2013-01-08 07:59:40:354 [1.0.7.7] - Plugin thread ID                 4084202352
2013-01-08 07:59:40:354 [1.0.7.7] - Flush thread ID                  4075809648
2013-01-08 07:59:40:354 [1.0.7.7] - Report Schedule thread ID        4067416944
2013-01-08 07:59:40:354 [1.0.7.7] - Import thread ID                 4059024240
2013-01-08 07:59:40:354 [1.0.7.7] - AMI Reader thread ID             4050631536
2013-01-08 07:59:40:354 [1.0.7.7] - AMI Process thread ID            4042238832
2013-01-08 07:59:40:354 [1.0.7.7] - Verifying Asterisk
2013-01-08 07:59:40:356 [1.0.7.7] - Verifying Asterisk - Completed
2013-01-08 07:59:40:356 [1.0.7.7] - Exchange Library      : Verifying Versions...
2013-01-08 07:59:40:421 [1.0.7.7] - Exchange Library      [ver 2009.08.01.01]: Version Ok!
2013-01-08 07:59:40:443 [1.0.7.7] - 23352 exchange entries loaded.
2013-01-08 07:59:40:479 [1.0.7.7] - AMI connected to port 5038 on 127.0.0.1
2013-01-08 07:59:40:484 [1.0.7.7] - [AMI LOG] - Requesting Dialplan...[CDRStatusResponse]
2013-01-08 07:59:40:950 [1.0.7.7] - [AMI LOG] - Requesting Dialplan...[CDRStatusResponse] [Repeated 1 times]
2013-01-08 07:59:40:950 [1.0.7.7] - Integration: Server [20003]: Started OK
2013-01-08 07:59:40:950 [1.0.7.7] - Barring: Server [20009]: Started OK
2013-01-08 07:59:41:776 [1.0.7.7] - [AMI LOG] - Dialplan Processed.
2013-01-08 08:00:03:921 [1.0.7.7] - [AMI LOG] - Dialplan Processed. [Repeated 1 times]
2013-01-08 08:00:03:921 [1.0.7.7] - User admin connected from 192.168.250.170
2013-01-08 08:00:35:362 [1.0.7.7] - [AMI LOG] - Waiting on Dial Plan and Peer List
2013-01-08 08:01:04:583 [1.0.7.7] - [AMI LOG] - Peerlist Processed.
2013-01-08 08:01:05:368 [1.0.7.7] - [AMI LOG] - CDR File "/var/log/asterisk/cdr-custom/Master.csv" added to list.
2013-01-08 08:01:05:473 [1.0.7.7] - Loading Ratebook: /opt/datatex/tng/rates/do_not_cost.so
2013-01-08 08:01:05:475 [1.0.7.7] - RateBook : do_not_cost Loaded [ver 2007.04.19.02]
2013-01-08 08:01:05:477 [1.0.7.7] - Loading Ratebook: /opt/datatex/tng/rates/za_telkom.so
2013-01-08 08:01:05:479 [1.0.7.7] - RateBook : za_telkom Loaded [ver 2010.03.01.01]

The expiry date for the license is on the 2013-07-02.

[stephen]

Some additional information...
I have just ran the tng process in foreground mode. I can see it lists the dialplan and then shows each peer one by one. It also looks like it is reading a few call statuses:

Code: Select all

TimeStamp: 08:34:31.216
Event: Hangup
Privilege: call,all
Channel: DAHDI/28-1
Uniqueid: 1357626769.42795
Cause: 16
Cause-txt: Normal Clearing


This does not update the live call list view though and output stops after it loads the Master.csv file:

Code: Select all

CDR File "/var/log/asterisk/cdr-custom/Master.csv" added to list.

I can see in another ssh session that there are lots of calls happening, but nothing happening in the tng process.

[Belinda Frick]

Please refer to settings in forum post - AMI Exception/Waiting on Dial Plan and Peer List

NOTE: The logs you listed, shows the TNG for Asterisk as 1.0.7.0 and not 1.0.7.7. Please check that your upgrade to 1.0.7.7 was successful. Ensure that you followed the steps to stop/start the deamon as per forum post TNG for Asterisk version 1.0.7.7 upgrade, else the executable would not be replaced.

[stephen]

Please refer to settings in forum post - AMI Exception/Waiting on Dial Plan and Peer List

NOTE: The logs you listed, shows the TNG for Asterisk as 1.0.7.0 and not 1.0.7.7. Please check that your upgrade to 1.0.7.7 was successful. Ensure that you followed the steps to stop/start the deamon as per forum post TNG for Asterisk version 1.0.7.7 upgrade, else the executable would not be replaced.

If you look at the second log, you will see it transition from 1.0.7.0 to 1.0.7.7.
I will enable the rawdata logging as suggested and feedback shortly.

[Belinda Frick]

You can also try the 'writetimeout' setting - see forum post
TNG4* Custom CSV is correct format but not logging calls

[stephen]

Hi Belinda

I sent you a PM with the logs attached after enabling rawdata logging. I already had the writetimeout set to 10000. It is still not going past the 8th of Nov.

Thanks

[Belinda Frick]

1. Please post me the TNG log files, when it stopped loggin calls. I need to check the log entries when trying to import.
   
2. Can you see call details for after 8 November 2012 in the master.csv file? Please forward me the master.csv file. I will need to setup a test TNG to see if we can import your CSV file.

[stephen]

This forum seems to fail when I try and upload the Master.csv, it is 9MB while compressed with bzip.
Is there another way I can send it to you?